Autopsy your RAG pipeline at the claim level

What it does

RAGChecker evaluates Retrieval-Augmented Generation systems by breaking responses into atomic claims and checking them against retrieved context and ground-truth answers. It returns overall pipeline scores alongside separate diagnostic metrics for the retriever and generator, so you know which component to blame. The framework also includes a benchmark dataset and a meta-evaluation set validated against human preferences.

The interesting bit

Instead of treating an answer as a monolithic string, it uses claim-level entailment to measure precisely how much of the ground truth made it into the context, how much the generator actually used, and how much is hallucination or self-knowledge. It also integrates with LlamaIndex if you are already in that ecosystem.

Key highlights

• Splits evaluation into retriever_metrics (claim recall, context precision) and generator_metrics (faithfulness, hallucination, noise sensitivity, self-knowledge)
• Requires an external LLM for claim extraction and entailment checking; the README points to RefChecker for model selection guidance
• Includes a 4k-question benchmark across 10 domains and a human-annotated meta-evaluation dataset
• Joint-check mode runs faster, though the README notes that disabling it yields slightly more accurate results

Caveats

• Evaluation hinges on an external LLM for claim extraction and entailment checking, so your mileage will vary with the model you select
• The default joint-check mode favors speed over a slight accuracy gain; one-by-one checking is noted as “slower but slightly more accurate”
• You will need to consult RefChecker’s documentation to configure the extractor and checker models

Verdict

Worth a look if you are shipping a RAG application and need to prove whether the retriever or the LLM is the bottleneck. Skip it if coarse precision and recall numbers are enough and you do not need component-level forensics.