NLPOptimize/flash-tokenizer
A high-performance C++ tokenizer implementation for LLM inference serving.
Not currently ranked — collecting fresh signals.
star history
FlashTokenizer is a CPU-optimized tokenizer library written in C++ that implements BertTokenizer for LLM inference. It uses pybind11 for Python bindings and achieves reportedly 10x faster tokenization than HuggingFace’s BertTokenizerFast. The implementation uses trie-based data structures for the WordPiece algorithm and is designed as a drop-in replacement for HuggingFace tokenizers in production inference scenarios.
Frequently asked
- What is NLPOptimize/flash-tokenizer?
- A high-performance C++ tokenizer implementation for LLM inference serving.
- Is flash-tokenizer open source?
- Yes — NLPOptimize/flash-tokenizer is an open-source project tracked on heatdrop.
- What language is flash-tokenizer written in?
- NLPOptimize/flash-tokenizer is primarily written in C++.
- How popular is flash-tokenizer?
- NLPOptimize/flash-tokenizer has 461 stars on GitHub.
- Where can I find flash-tokenizer?
- NLPOptimize/flash-tokenizer is on GitHub at https://github.com/NLPOptimize/flash-tokenizer.