Reinforcement Learning for Agents That Edit Their Memory

What it does

verl-agent is a research framework built on top of veRL for training LLM and vision-language agents with reinforcement learning. It wraps a suite of environments—ALFWorld, WebShop, Sokoban, and others—in a gym-style interface and bundles implementations of algorithms including GRPO, PPO, and the paper’s GiGPO. The target is multi-turn agent training that remains practical across dozens of steps.

The interesting bit

Rather than concatenating full interaction histories into a context window that balloons with every turn, verl-agent treats each step as independent. You freely define what history, summaries, or external knowledge matter for the current step, keeping context length nearly flat over time. It is deliberate forgetting as a scalability strategy.

Key highlights

• Official NeurIPS 2025 GiGPO implementation, alongside GRPO, PPO, DAPO, REINFORCE++, and more.
• Supports text-only and vision-language models such as Qwen3-VL, Qwen2.5-VL, and LLaMA3.2.
• Parallelized gym environments and “group environments” that share initial states for algorithms requiring multiple rollouts from the same starting point.
• LoRA fine-tuning supported; the README notes 7B models can train on two H100 GPUs.
• Modular memory manager allows per-step customization of what observations or summaries the agent sees.

Caveats

• GiGPO’s performance changed slightly after the June 2025 major update; reproducing the original paper results requires the pre-update release.
• AppWorld support is currently marked experimental.

Verdict A solid choice for RL researchers working on long-horizon tool-use, web navigation, or embodied agents. If you are looking for a plug-and-play agent API rather than a training framework, this is not it.