This library is a direct port of OpenAI’s tiktoken tokenizer to Go, providing fast BPE (Byte Pair Encoding) tokenization. It encodes text into integer token sequences compatible with OpenAI models including GPT-3.5 and GPT-4. The library supports caching token dictionaries and offers alternative loaders for offline or custom dictionary sources, making it a lightweight dependency for Go applications integrating with OpenAI’s APIs or processing LLM-formatted text.