RL framework makes VLMs build mental maps

What it does VAGEN is a reinforcement-learning framework for training vision-language model agents across multiple turns. It treats agentic tasks as a Partially Observable Markov Decision Process and explicitly rewards the agent for internal world-model reasoning—estimating current state and predicting transitions—using an LLM-as-Judge signal alongside task success. The architecture decouples environments, harnesses, models, and rollouts from the underlying training algorithm so you can plug in different RL methods without rewriting the agent loop.

The interesting bit Instead of relying solely on sparse end-of-episode rewards, VAGEN supervises the agent’s “mental model” by grading its perception-reasoning-prediction chain, which the authors frame as World Modeling RL. This is unusual: most RL pipelines for VLMs optimize for final accuracy; VAGEN optimizes for whether the agent can articulate what changed and what will change next.

Key highlights

• Modular axes for algorithms, environments, harnesses, models, and evaluation, all importing through stable facades.
• Supports Sokoban, FrozenLake, Navigation, ManiSkill, and SpatialGym (SVG) environments out of the box.
• Compatible with Qwen2.5-VL, Qwen3-VL, Qwen3.5, InternVL3.5, and GLM-4.6V model families.
• Built on the VERL agent-loop; recently refactored into a lightweight VAGEN-Lite reimplementation.
• Structured output formats separate perception, reasoning, prediction, and answer (or a compact think/answer variant).

Caveats

• Several environments (spatial_gym, navigation, primitive_skill) require additional manual setup and dependencies such as AI2-THOR or ManiSkill.
• The codebase underwent a major refactor to decouple the environment and harness layer for Verl 0.9.0, so older branches exist for prior versions.

Verdict Worth a look if you are researching multi-turn VLM agents or need a modular RL harness for embodied-AI tasks. Probably overkill if you just want to fine-tune a vision model on static image-caption pairs.