Your tokenizer shouldn't be slower than your SSD

What it does Gigatoken is a Rust tokenizer that reads text directly from disk and parallelizes across cores to hit throughput measured in gigabytes per second. It offers a native API for raw speed, plus compatibility shims that mimic HuggingFace and Tiktoken outputs exactly. The target is the data-preprocessing bottleneck in LLM training pipelines, where existing tools often stall on regex and Python overhead.

The interesting bit Rather than outsourcing pretokenization to a regex engine, Gigatoken reimplements it with SIMD and heavy branching reduction, then caches pretoken-to-ID mappings to skip re-encoding repeated strings. The README admits caching is a hard problem here because pretoken distributions are long-tailed, so naive growth would eat memory alive.

Key highlights

• Benchmarked at 24 GB/s on a dual-socket AMD EPYC and 8.7 GB/s on an Apple M4 Max for many BPE-style tokenizers, with speedups of 100×–1000× over HuggingFace.
• Compatibility mode wraps existing HF or Tiktoken objects and promises bit-for-bit identical outputs, though the README warns this carries a “non-negligible cost to performance.”
• The native TextFileSource API lets the Rust core handle file I/O and splitting on byte separators, avoiding Python object overhead entirely.
• Covers most major modern tokenizer families: GPT-2, Llama 3/4, Qwen, DeepSeek, Phi-4, Nemotron, OLMo, and others.
• Gains are consistent across x86 and ARM, not a single-platform trick.

Caveats

• SentencePiece-based tokenizers—including Gemma, Mistral, CodeLlama, and Llama 2/Phi-3 variants—are explicitly “not well optimized” and show far smaller speedups, often in the 10×–20× range.
• The headline ~1000× figure applies to the native API; exact HuggingFace compatibility costs performance, though it is still “way faster” according to the README.

Verdict A clear win for anyone preprocessing massive text corpora for model training and tired of watching tokenization become the bottleneck. If your pipeline is already GPU-bound or you rely heavily on SentencePiece tokenizers, the gains are real but less dramatic.