Go port of OpenAI's tokenizer keeps dictionaries offline-ready

What it does

tiktoken-go is a straight Go port of OpenAI’s tiktoken BPE tokenizer. It encodes text into tokens using the same dictionaries and model mappings as the Python original, covering everything from gpt2 to gpt-4o and the newer o200k_base encodings. The API mirrors the familiar GetEncoding and EncodingForModel patterns, so token counts should match the reference implementation.

The interesting bit

The project treats network access as a bug you can fix. By default it caches downloaded dictionaries, but if you want to eliminate runtime downloads entirely you can inject an offline loader that reads embedded BPE files—handy for air-gapped binaries or containers with egress restrictions. The author also includes a ready-made helper for counting chat-message tokens, though they candidly warn that OpenAI’s math on this changes without notice.

Key highlights

• Supports the full range of OpenAI encodings (cl100k_base, o200k_base, p50k_base, r50k_base) and their associated models
• Pluggable BpeLoader interface lets you bring your own dictionary source, including fully offline embedding via a companion project
• Includes a reference token-counting implementation for Chat API message arrays based on the OpenAI cookbook
• Benchmarks on Apple Silicon show near-identical performance to the Python original, though the Go port lags on o200k_base and cl100k_base under Linux/AMD by roughly 50–100%

Caveats

• The chat-message token counter is explicitly labeled as fragile; OpenAI can change the formula at any time
• The offline BPE loader lives in a separate repository, so “batteries included” only applies if you are okay with runtime downloads or writing a custom loader

Verdict

Worth a look if you are shipping a Go service that needs to count or estimate OpenAI tokens without pulling in Python. Skip it if you are already running in a Python environment and have no egress constraints.