Systematic reviews with a committee of cheap LLMs

What it does

MetaScreener ingests bibliographic exports from PubMed, Scopus, or similar sources and runs title/abstract screening through an ensemble of 4+ open-source LLMs. It returns include/exclude decisions with confidence scores and per-criteria breakdowns across PICO/PEO/SPIDER frameworks, automatically escalating uncertain papers to human review. Beyond abstract screening, it also handles full-text PDF review, structured data extraction, and risk-of-bias assessment.

The interesting bit

Rather than trusting a single model, the tool pipes every paper through a four-layer Hierarchical Consensus Network: parallel inference, a semantic rule engine, calibrated confidence aggregation using Platt or isotonic regression, and a tiered decision router. The Element Consensus Score pinpoints exactly which criteria elements—say, Population versus Outcome—the models actually agree on, so you know where the uncertainty lives.

Key highlights

• 15 open-source models via OpenRouter, from DeepSeek V3 and Qwen 3 down to Gemma 3 and Phi 4, with preset blends costing roughly $0.003–$0.009 per paper
• Deterministic settings (temperature=0.0, seed=42) and a full audit trail for reproducibility
• Active learning loop that recalibrates model weights from human overrides
• Evaluation suite covering WSS@95, AUROC, Brier score, and calibration curves
• Vue 3 frontend over a FastAPI backend; test suite runs fully offline

Caveats

• Live screening requires an OpenRouter API key and external API calls; the ensemble does not appear to support local self-hosted execution for production use
• The provided README is truncated mid-sentence in the development section, leaving the complete local setup and type-checking workflow unclear

Verdict

Researchers conducting systematic reviews who want transparent, probabilistic automation with a human fallback should try this; teams needing fully offline, air-gapped screening should look elsewhere.