RL training that teaches LLMs when to search and when to think

What it does

Search-R1 is a reinforcement learning framework—built on veRL—that trains LLMs to interleave chain-of-thought reasoning with search engine calls. It treats retrieval as a tool the model learns to invoke via API during rollouts, using algorithms such as PPO, GRPO, or REINFORCE with rule-based outcome rewards. The search backend runs as a decoupled server, which can be a local sparse or dense retriever, an ANN index, or an online API like Google or Bing.

The interesting bit

The project shows that small base models—specifically 3B-parameter Llama and Qwen—can learn to search and reason from scratch without supervised tool-use data, simply by being rewarded for correct final answers. A 7B model goes further, spontaneously developing multi-turn search trajectories where it issues queries, reads results, and refines its strategy.

Key highlights

• Supports PPO, GRPO, and REINFORCE across models from 3B to 30B+ (multinode)
• Plugs into local retrievers (BM25, FAISS) or commercial search APIs
• Includes neural reranker support and fully open experiment logs
• Two published papers with detailed empirical studies
• Already integrated into veRL upstream and adopted by projects like DeepResearcher

Caveats

• The authors still characterize results as “preliminary,” and the setup requires juggling separate conda environments for training and retrieval.

Verdict

A strong fit for researchers building autonomous reasoning agents who need a trainable, transparent alternative to OpenAI DeepResearch. Skip it if you want a polished end-user product rather than a training pipeline.