Teaching cars to think in language but steer with diffusion

What it does ReCogDrive is a research framework for end-to-end autonomous driving that uses a vision-language model to interpret driving scenarios before delegating actual trajectory planning to a diffusion model. The authors first instill human driving cognition into the VLM through a hierarchical data pipeline built on twelve re-annotated public datasets and 752k synthetic Navsim QA pairs. Once the VLM understands the scene, its learned priors are injected into a diffusion planner that outputs continuous trajectories, with a final reinforcement stage—DiffGRPO—nudging the planner toward safer, more comfortable paths.

The interesting bit The central bet here is that trajectory planning should not be a language modeling task; instead of forcing a VLM to emit waypoints as text strings, ReCogDrive uses the VLM as a cognitive backbone and lets a diffusion policy handle the physics. This separation of concerns is what the authors credit for their reported state-of-the-art numbers on NAVSIM and Bench2Drive while avoiding the slow, format-prone inference of pure language agents.

Key highlights

• Achieves a 90.8 PDMS score on NAVSIM with the 2B-parameter base model after full reinforcement training, and 90.4 with the 8B variant.
• Releases pretraining datasets covering twelve public driving corpora—re-annotated with Qwen2.5VL-72B and filtered—plus 752k proprietary Navsim QA pairs.
• Publishes model weights, training code, and evaluation scripts for NAVSIM, with Bench2Drive and DriveLM evaluation frameworks still on the roadmap.
• Introduces DiffGRPO, a reinforcement-learning stage tailored for the diffusion planner to optimize for safety and comfort metrics rather than just imitation.

Caveats

• The README notes that evaluation frameworks for Bench2Drive, DriveLM, NAVSIM2.0, and DriveBench are not yet released, so reproducing those specific leaderboard numbers requires extra legwork.
• A NumPy bug previously corrupted the training metric cache; the authors fixed it in a late-2025 update, but early adopters may need to verify their local results if they pulled code before then.

Verdict Researchers building end-to-end driving stacks should look here for a concrete, open-weight alternative to text-only VLM planners; if you need a fully turnkey evaluation suite for every benchmark mentioned, the repo is still a work in progress.