Pure-Python search that outruns Elasticsearch

What it does

bm25s is a BM25 text-retrieval library written in pure Python. It tokenizes a corpus, builds an index using sparse matrices, and returns ranked document IDs and scores for queries. It runs on NumPy alone—no Java, no PyTorch—and can optionally use Numba for a further speed boost on large datasets.

The interesting bit

The speed comes from eager sparse scoring: instead of calculating BM25 weights at query time, it front-loads the work at index time and stores every document-token score in a sparse matrix. At query time it just slices and aggregates, which is why the README shows it outpacing both Elasticsearch and the popular rank-bm25 library by orders of magnitude in single-threaded BEIR benchmarks.

Key highlights

• Keeps the dependency stack minimal: core functionality needs only NumPy.
• Supports custom tokenization pipelines, stopword lists, and stemmer functions.
• Corpus entries can be plain strings or arbitrary dictionaries; retrieval returns the original objects by position.
• Indices and corpora can be saved to disk and reloaded later.
• Ships with a high-level API and a terminal CLI for indexing CSV, JSON, JSONL, or plain text without writing Python scripts.

Caveats

• The authors now recommend the separate BM25 PyPI package for beginners, suggesting bm25s is deliberately lean and requires choosing the right extras (e.g., bm25s[core]) for stemming and JIT compilation.
• Retrieved documents are mapped strictly by index position, so the corpus you pass for retrieval must stay in the same order and length as the indexed set.
• Saved corpora must be JSON-serializable strings, dicts, lists, or tuples.

Verdict

Worth a look if you need fast lexical search inside a Python stack without pulling in a search server or ML framework. Skip it if you need semantic/vector search or a fully managed, distributed index.