Beating GPT-4o at search with a 3B model and zero labels

What it does

DeepRetrieval is a reinforcement-learning pipeline that teaches a 3B-parameter LLM to rewrite search queries to maximize retrieval recall. It skips supervised fine-tuning on human-annotated query pairs and instead lets the model learn by trial and error, using live retrieval metrics as scalar rewards. The agent outputs chain-of-thought reasoning before committing to a final augmented query, and the system plugs into real search engines, SQL databases, and both dense and sparse retrievers.

The interesting bit

The reward function is almost comically direct: the README spells out a lookup table where recall ≥ 0.7 nets a +5.0 reward and recall below 0.05 slaps the model with -3.5. That single number is the entire training signal, yet it is enough to push a Qwen2.5-3B-Instruct-based agent past GPT-4o, Claude-3.5-Sonnet, and the previous 7B supervised best on PubMed publication and clinical-trials search.

Key highlights

• Hits 65.07% recall on publication search and 63.18% on clinical-trials search, more than doubling the previous best (24.68% and 32.11%).
• Runs on a 3B model, outperforming GPT-4o, Claude-3.5-Sonnet, and the supervised 7B LEADS baseline.
• Needs no labeled query-augmentation data; training is purely reward-driven from retrieval outcomes.
• Supports real-world engines (e.g., PubMed), SQL search, BM25 sparse retrieval, and dense passage retrieval.
• Built atop the verl RL framework and PySerini, with trained checkpoints and datasets on Hugging Face.

Caveats

• The README includes a lengthy “Commonly Encountered Issues” section for a reason: expect NumPy 2.x incompatibilities, missing Python packages, and CUDA/SLURM environment mismatches that need manual fixes.
• Training is GPU-bound and tied to a specific environment stack; this is not a lightweight CPU library you can install and forget.
• Adapting the pipeline to a new search API or custom reward logic requires editing framework source files such as pubmed.py and ray_trainer.py.

Verdict

Worth a look if you are building retrieval pipelines and suspect your bottleneck is query quality, not the index itself. Skip it if you need a polished, drop-in query-rewriting SDK without the RL training baggage.