Spotting RAG fibs with an encoder, not an LLM

What it does

LettuceDetect flags unsupported tokens in RAG-generated answers by comparing them against the retrieved context and question. It is trained on the RAGTruth dataset and exposes both span-level and token-level predictions with confidence scores.

The interesting bit

The framework swaps the usual LLM-as-judge pattern for compact encoder models—ModernBERT for English and EuroBERT for multilingual—claiming to beat other encoder and prompt-based baselines on RAGTruth while running faster and leaner. It even reportedly tops LLAMA-2-13B on the same benchmark, though it trails the fine-tuned LLM from the RAG-HAT paper.

Key highlights

• Pinpoints exact hallucinated character spans rather than binary answer-level scores
• Supports seven languages via EuroBERT with 8K context windows; English models use ModernBERT at 4K
• Offers TinyLettuce variants as small as 17M parameters for constrained environments
• Ships with a web API, async client, and Streamlit demo alongside the core library
• Optional OpenAI-based LLM baselines are included if you need a reference point

Caveats

• All performance claims are tied to the RAGTruth dataset; transfer to other domains or datasets is not discussed
• The web API is a thin FastAPI wrapper with dev/prod startup scripts; the README does not claim enterprise-grade operational features

Verdict

Try it if you want a lightweight, offline hallucination guardrail for RAG that points to exact fibs. Look elsewhere if you need a battle-tested cross-domain factuality layer or a managed service.