Local embeddings and rerankers, sans Tokio

What it does

fastembed-rs wraps ONNX inference and Hugging Face tokenizers to run open-source embedding, sparse embedding, image embedding, and reranking models locally. It exposes a deliberately synchronous API with no Tokio dependency, turning text and images into vectors without ongoing API calls.

The interesting bit

Rather than forcing you to wire up separate crates for each retrieval primitive, it bundles dense text embeddings, sparse SPLADE vectors, CLIP-style image embeddings, and rerankers behind one interface. BGE-M3 even emits dense, sparse, and ColBERT representations in a single forward pass, which cuts the model-loading overhead for hybrid search.

Key highlights

• Synchronous API with zero Tokio dependency
• Dense, sparse, and ColBERT embeddings from BGE-M3 in one forward pass
• Quantized CPU-friendly variants for several models
• Image embeddings and cross-encoder rerankers alongside text support
• Inference runs locally without API keys after the initial model download

Caveats

• The default quantized BGE-M3 model is CPU-only; GPU execution providers will fail
• Qwen3 and Nomic Embed v2 MoE require feature flags and use the Candle backend instead of ONNX
• Model weights are fetched from Hugging Face on first use, so the initial run needs network access

Verdict

A solid fit for Rust RAG pipelines that need model variety without async complexity. Less compelling if you already have a custom ONNX workflow or need heavily optimized GPU inference.