Post-Training for Agents That Need 300 Turns to Get a Reward

What it does AxisRL is a post-training framework for agentic reinforcement learning. It wraps SGLang inference and Megatron distributed training in a single system that handles multi-turn rollouts, tool calls, reward collection, weight synchronization, and training sample construction. The aim is to stop the rollout and trainer paths from silently diverging when agents interact with environments over long trajectories.

The interesting bit Instead of funneling everything through a bloated central driver, AxisRL keeps the control plane thin and moves heavy payloads via handles that trainer workers read on demand. It also supports partial rollout so a single slow trajectory cannot stall an entire batch, and it replays MoE routing decisions to keep expert choices consistent across rollout and training.

Key highlights

• Supports white-box agent loops (math, search) and black-box harness capture through an OpenAI-compatible proxy.
• Ships with configurable policy optimizers including PPO, GRPO/GRPO2, GSPO, TOPR, and TIS.
• Uses partial rollout and handle-based data movement to cut idle time from long-tail trajectories.
• Provides mismatch analysis, routing replay, and spike replay to debug drift between rollout and training workers.
• Includes recipes for GSM8K, Search-R1, and a black-box OpenHands/E2B integration.

Caveats

• The black-box RL recipe using OpenHands and E2B is explicitly a work in progress; its configs and proxy interfaces may shift.

Verdict Worth exploring if you are building distributed RL infrastructure for tool-using LLMs and care about rollout-trainer consistency. If you just need a quick single-turn fine-tuning script, this is overkill.