LLMs Love to Ramble. This Grades the Facts.

What it does

LongFact is a collection of 2,280 prompts engineered to make models generate lengthy, fact-dense responses. SAFE (Search-Augmented Factuality Evaluator) automatically verifies the factual claims in those responses. The repo also ships F1@K, a metric that stretches standard F1 scoring into long-form territory by baking human-preferred answer length into recall.

The interesting bit

Most benchmarks test short answers; this one tackles the messier problem of verifying accuracy across sprawling multi-sentence output without hiring a room of human fact-checkers. The main pipeline is pre-wired to benchmark OpenAI and Anthropic models.

Key highlights

• 2,280 prompts in longfact/ designed to elicit long, information-heavy answers
• SAFE uses search-augmented verification to judge factual accuracy automatically
• F1@K adapts precision/recall metrics for long-form text using human-preferred length
• Includes a full experimentation pipeline for OpenAI and Anthropic model APIs
• Most files have corresponding unit tests

Caveats

• The benchmarking pipeline requires your own API keys for OpenAI and Anthropic, dropped into common/shared_config.py
• Top-level documentation repeatedly points to subdirectory READMEs for real detail, so expect to click around

Verdict

Worth a look if you need to evaluate long-form generation and want to replace manual fact-checking with an automated pipeline. Not for you if your models only emit short, single-sentence answers.