quickwit-oss/whichlang
A Rust library that uses multiclass logistic regression over n-grams to detect language in text at high throughput.
Not currently ranked — collecting fresh signals.
star history
This library provides fast, lightweight language detection supporting 16 languages including Arabic, Chinese, Hindi, Japanese, Korean, and European languages. It uses a hashing trick to project n-gram features (2, 3, 4-grams of letters and codepoint features) into a space of size 4,096, with the logistic regression model trained in Python and the weights generated as Rust code. The project was created to serve high-throughput text processing needs for the Quickwit search engine.
Frequently asked
- What is quickwit-oss/whichlang?
- A Rust library that uses multiclass logistic regression over n-grams to detect language in text at high throughput.
- Is whichlang open source?
- Yes — quickwit-oss/whichlang is open source, released under the MIT license.
- What language is whichlang written in?
- quickwit-oss/whichlang is primarily written in Rust.
- How popular is whichlang?
- quickwit-oss/whichlang has 454 stars on GitHub.
- Where can I find whichlang?
- quickwit-oss/whichlang is on GitHub at https://github.com/quickwit-oss/whichlang.