Before you blame the LLM, benchmark the rest of the pipeline
XRAG dissects retrieval-augmented generation into its component modules — splitters, embedders, retrievers, orchestrators — and measures each one, so you can see what is actually dragging your answers down.

What it does XRAG treats a RAG pipeline as a stack of swappable parts rather than a black box: text splitters, embedding models, retrievers (vector, BM25, hybrid, tree-based, keyword, document-summary), post-processors, and orchestrators. You compose a pipeline, run it against benchmark datasets like HotpotQA, and get scored on everything from retrieval quality (MRR, Hit@K, NDCG) to answer faithfulness and hallucination. It ships as a CLI, a web UI, and a backend API service, and it can generate QA pairs from your own document folders.
The interesting bit The orchestrator layer is the unusual part: agentic RAG strategies — self-RAG, adaptive-RAG, SIM-RAG — are expressed as five orchestration patterns (sequential, conditional, iterative, parallel, hybrid), which turns the question of which fancy RAG paper to copy into a controlled experiment. The accompanying paper was accepted at ICDE 2026, so the tool is the artifact of a real research question: which components actually move the needle?
Key highlights
- Pluggable modules across the whole pipeline, with LLM support for OpenAI, Hugging Face, and Ollama
- Three tiers of metrics: traditional (F1, EM, MRR, MAP, NDCG), LLM-judged (faithfulness, relevancy, correctness), and deep evaluation (contextual precision/recall, hallucination, bias)
- Agentic RAG via five orchestrator patterns, including self-RAG and adaptive-RAG
- Web UI covers the full workflow: dataset upload, index building, strategy configuration, interactive testing, evaluation
- Doubles as a QA-pair generator for your own documents
Caveats
- Dependency hygiene is shaky — the README itself instructs installing
jurystripped of its dependencies and pinning a 2020-eraurllib3, so expect some environment wrangling - Requires Python 3.11 or later
- The roadmap still lists several metrics and modules (semantic entropy, late chunking) as unchecked, so coverage is a work in progress
Verdict Worth a look if you are building or tuning a RAG system and want evidence instead of vibes when picking components. Skip it if you just want chat-with-your-docs out of the box — this is a measurement instrument, not a product.
Frequently asked
- What is DocAILab/XRAG?
- XRAG dissects retrieval-augmented generation into its component modules — splitters, embedders, retrievers, orchestrators — and measures each one, so you can see what is actually dragging your answers down.
- Is XRAG open source?
- Yes — DocAILab/XRAG is open source, released under the Apache-2.0 license.
- What language is XRAG written in?
- DocAILab/XRAG is primarily written in Python.
- How popular is XRAG?
- DocAILab/XRAG has 612 stars on GitHub.
- Where can I find XRAG?
- DocAILab/XRAG is on GitHub at https://github.com/DocAILab/XRAG.