← all repositories

niieani/gpt-tokenizer

A TypeScript BPE tokenizer for OpenAI's GPT models including GPT-5, GPT-4o, and o-series models.

807 stars TypeScript Language ModelsData Tooling
gpt-tokenizer
Velocity · 7d
+0.7
★ / day
Trend
steady
star history

This library provides byte-pair encoding (BPE) tokenization for all OpenAI models including GPT-4, GPT-4o, GPT-5, and o-series models. It is a direct port of OpenAI’s tiktoken written in TypeScript, supporting both synchronous and asynchronous operation. The package includes chat-specific encoding utilities, generator functions for streaming, and supports all major encoding schemes like cl100k_base and o200k_base used across different OpenAI model families.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.