A vector index that shrinks 31 GB to 4 GB and outruns FAISS

What it does

turbovec is a Rust vector index with Python bindings built on Google Research’s TurboQuant algorithm. It compresses high-dimensional embeddings down to 2-bit or 4-bit representations using a data-oblivious scalar quantizer—no k-means codebook training, no offline build step. You add vectors, they are indexed immediately, and you search with hand-written SIMD kernels on ARM and x86.

The interesting bit

The trick is a random rotation that forces every coordinate into a predictable Beta distribution regardless of the input data. Because the distribution is known up front, the Lloyd-Max codebook can be computed from the math rather than fitted to the corpus, and a per-coordinate calibration frozen after the very first batch removes the need for any later retraining. Length-renormalized scoring fixes the bias introduced by squashing vectors into tiny buckets, recovering accuracy at zero search-time cost.

Key highlights

• No training phase. Vectors are quantized online during ingest; the index never requires a separate build or rebuild as data grows.
• Faster than FAISS. On ARM, hand-written NEON kernels beat FAISS IndexPQFastScan by 12–20%. On x86 with AVX-512BW, it wins on 4-bit configs and stays within ~1% on 2-bit single-threaded search.
• Search-time filtering. Pass an id allowlist and the SIMD kernel short-circuits disallowed 32-vector blocks before scoring, avoiding the usual over-fetch and recall penalty of post-filtering.
• Drop-in framework integrations. Swappable backends for LangChain, LlamaIndex, Haystack, and Agno that reuse the same persistence and retriever wiring.
• Pure local operation. Runs offline with no managed service; pair it with an open-source embedding model for an air-gapped RAG stack.

Caveats

• On x86 multi-threaded search, 2-bit configs at d=1536 and d=3072 lag FAISS by 2–4% because the inner accumulate loop is too short to amortize unrolling against FAISS’s AVX-512 VBMI path.
• Low-dimensional embeddings like GloVe d=200 are a harder regime for the asymptotic Beta assumption; at 2-bit and R@1, TurboQuant trails FAISS by 1.2 points, though it closes the gap by k≈16.

Verdict

If you are building RAG on memory-constrained or air-gapped hardware and want to stop babysitting vector index training jobs, this is worth a hard look. If you are already happy with a managed cloud vector store and don’t pay for RAM, it is probably overkill.