Half a million fake documents to stress-test enterprise RAG

What it does EnterpriseRAG-Bench simulates a fictional AI-inference company called “Redwood Inference” with roughly 500,000 documents spread across nine sources you’d actually find in a workplace. Slack dominates the corpus at 275,000 messages, followed by Gmail, Linear tickets, Google Drive files, HubSpot records, meeting transcripts, GitHub pull requests, Jira tickets, and Confluence wikis. The accompanying 500 questions span ten categories ranging from simple lookups to adversarial headaches like conflicting information, intra-document reasoning, and queries where the answer simply does not exist.

The interesting bit The dataset is deliberately engineered to be annoying in the specific ways real company data is annoying: documents use internal jargon and codenames, information contradicts across systems, near-duplicates accumulate, and volume follows a realistic power-law where Slack drowns out the wiki. The authors also provide scaffolding code to generate new fake companies for different industries or scales, so you are not stuck benchmarking against one synthetic startup.

Key highlights

• ~500,000 documents across nine enterprise sources, with Slack and Gmail making up the bulk
• 500 questions in 10 categories, from basic retrieval to conflicting-info and “answer not found” scenarios
• 100 additional metadata-dependent questions for teams testing metadata-aware RAG (excluded from the main leaderboard)
• Human-in-the-loop scaffolding ensures cross-document coherence before high-volume synthetic generation
• Includes a HuggingFace leaderboard; the creators (Onyx) exclude themselves to avoid conflict of interest

Caveats

• The dataset is synthetic, so while the noise and volume distributions are designed to feel realistic, it is still generated rather than scraped from a real company.
• The arXiv citation lists a 2026 date and identifier 2605.05253, which appears to be either a typo or a very forward-looking preprint.

Verdict Teams building enterprise search or internal RAG should grab this to move beyond public-web benchmarks. If your retrieval problem is already limited to clean, well-structured documentation, it is probably overkill.