VKCOM/YouTokenToMe
A fast C++ implementation of Byte Pair Encoding (BPE) tokenization for text preprocessing.
Not currently ranked — collecting fresh signals.
star history
YouTokenToMe is an unsupervised text tokenizer implementing Byte Pair Encoding (BPE), a standard algorithm used to prepare text data for language models. It is implemented in C++ with Python bindings for computational efficiency, claiming up to 60x speedup over comparable tokenizers like Hugging Face tokenizers, fastBPE, and SentencePiece. The tool supports multithreading, BPE-dropout regularization, and provides both Python and command-line interfaces for training tokenizers and encoding text into subword sequences.
Frequently asked
- What is VKCOM/YouTokenToMe?
- A fast C++ implementation of Byte Pair Encoding (BPE) tokenization for text preprocessing.
- Is YouTokenToMe open source?
- Yes — VKCOM/YouTokenToMe is open source, released under the MIT license.
- What language is YouTokenToMe written in?
- VKCOM/YouTokenToMe is primarily written in C++.
- How popular is YouTokenToMe?
- VKCOM/YouTokenToMe has 979 stars on GitHub.
- Where can I find YouTokenToMe?
- VKCOM/YouTokenToMe is on GitHub at https://github.com/VKCOM/YouTokenToMe.