Open-source RL for agentic tool use without hand-holding

What it does

ReCall is a training framework that uses reinforcement learning to teach LLMs how and when to call external tools. It evolved from the earlier ReSearch project—now archived in a branch—expanding support from search alone to arbitrary user-defined tools implemented as Python code. The goal is to match the agentic tool chaining seen in closed models like OpenAI o3, but without relying on human demonstrations of reasoning steps or tool trajectories.

The interesting bit

Instead of imitating curated tool-use traces, the model learns by generating tool calls, executing them in a sandbox, and receiving feedback through RL. The authors bootstrap this with a synthetic dataset called SynTool alongside MuSiQue, creating diverse multi-step tasks where the model must discover useful tool combinations on its own.

Key highlights

• Trains on a mixture of synthetic SynTool data and MuSiQue without supervised tool-use labels
• Supports arbitrary user-defined Python tools, not just Wikipedia search
• Customizes the verl RL framework and uses vLLM for training rollouts
• Provides trained Qwen2.5-based models and a Python inference wrapper
• Evaluates multi-hop QA via FlashRAG; broader tool-use benchmarks are pending

Caveats

• Explicitly marked as a work in progress; APIs and implementations are still shifting
• The sandbox for executing tool code is currently basic and potentially unsafe for local use
• Full reproduction requires multi-node training; the provided single-node script is only a quick-start example

Verdict

Researchers exploring RL-based agent training outside of imitation learning should keep an eye on this. If you need a hardened, production-ready tool-calling system today, the basic sandbox and unfinished benchmarks make this premature.