OpenAI's tiktoken, rebuilt to load synchronously in JS

What it does

gpt-tokenizer is a TypeScript reimplementation of OpenAI’s tiktoken that encodes and decodes text using the same Byte Pair Encoding rules as GPT-3.5 through GPT-5, o1, o3, and the open-weight Harmony models. It runs in Node, the browser, or any JavaScript environment, and handles both plain text and chat-formatted conversations. The library also estimates API costs and checks token limits without fully encoding the input.

The interesting bit

While most tokenizers force an async import or stash state in a global cache, this one loads synchronously and keeps no global state. It exposes generator-based encode and decode functions, including an async generator that can consume a stream of tokens and yield text chunks on the fly.

Key highlights

• Synchronous loading and operation, eliminating async/await boilerplate in constrained contexts
• Generator and async generator variants for streaming encode and decode without buffering entire arrays
• isWithinTokenLimit short-circuits limit checks, and estimateCost references built-in OpenAI pricing tables
• No global cache, avoiding the memory leaks the README attributes to earlier GPT-3-Encoder implementations
• Used in production by Microsoft Teams, Elastic Kibana, and Effect TS

Verdict

Grab it if you need client-side token counting, offline cost estimation, or streaming decode without a Python dependency. If you already proxy all token math through a backend service, you probably don’t need another npm package for it.