niieani/gpt-tokenizer
A TypeScript BPE tokenizer for OpenAI's GPT models including GPT-5, GPT-4o, and o-series models.

Velocity · 7d
+0.7
★ / day
Trend
→steady
star history
This library provides byte-pair encoding (BPE) tokenization for all OpenAI models including GPT-4, GPT-4o, GPT-5, and o-series models. It is a direct port of OpenAI’s tiktoken written in TypeScript, supporting both synchronous and asynchronous operation. The package includes chat-specific encoding utilities, generator functions for streaming, and supports all major encoding schemes like cl100k_base and o200k_base used across different OpenAI model families.