Inside the GPT-4 tokenizer, one clean Python file at a time

What it does

minbpe provides minimal Python classes for byte-level Byte Pair Encoding, the algorithm used by GPT, Llama, and Mistral to turn text into tokens. It includes a BasicTokenizer that runs BPE directly on text, a RegexTokenizer that pre-splits text by category—letters, numbers, punctuation—to prevent cross-boundary merges, and a GPT4Tokenizer wrapper that reproduces tiktoken’s output exactly. You can train a vocabulary from scratch on your own data, encode text to token IDs, and decode back.

The interesting bit

The GPT4Tokenizer is not merely GPT-4-like; it matches tiktoken token-for-token, including special tokens and what the README calls “unfortunate” historical 1-byte permutations. The entire repository is deliberately short and heavily commented, built to be read and modified rather than imported and ignored.

Key highlights

• Reproduces GPT-4 tokenization from the tiktoken library with exact merge recovery and special-token handling
• Supports training custom BPE vocabularies from scratch, unlike tiktoken
• RegexTokenizer uses the same regex pre-splitting strategy introduced in the GPT-2 paper and still used through GPT-4
• All files are short, heavily commented, and include inline usage examples
• Includes a step-by-step exercise guide and a full lecture transcript for self-study

Caveats

• The author notes it is not yet optimized for large files or big vocabularies; a production-scale training run would likely need a rewrite in a faster language
• GPT4Tokenizer is currently limited to GPT-4; GPT-2/3/3.5 and Llama-style SentencePiece equivalents are listed as future work

Verdict

Grab this if you want to understand or customize LLM tokenization without wading through opaque production code. Skip it if you need a battle-hardened, high-performance tokenizer for training on massive datasets.