An RL framework that treats diffusion models as first-class citizens

What it does

VeRL-Omni is an RL alignment and fine-tuning framework for multimodal generative models—think image diffusion, video generators like Wan2.2, and omni-modality models like Qwen3-Omni that juggle text, image, audio, and video simultaneously. Built on top of verl, it handles the distinct I/O patterns, compute profiles, and runtime bottlenecks that come with generating pixels and waveforms rather than tokens. It offers validated training recipes for algorithms including FlowGRPO, Flow-DPPO, DiffusionNFT, DanceGRPO, and GSPO across a growing roster of models.

The interesting bit

The framework treats rollout as a first-class problem for multimodal generation, using vLLM-Omni as a backend and overlapping reward computation asynchronously so the GPUs aren’t sitting idle while a GenRM-OCR scorer ponders the aesthetic quality of a generated frame. The authors claim roughly 25% higher end-to-end throughput than a diffusers-based flow_grpo baseline on their reference Qwen-Image setup, driven by FSDP2 training, async rewards, and faster rollouts.

Key highlights

• Supports three model families: diffusion generators (Qwen-Image, SD3.5, Wan2.2), unified understanding/generation models (BAGEL), and full omni-modality models (Qwen3-Omni-Thinker).
• Modular training backends: pick between VeOmni and FSDP2, with combinable parallelism (USP, TP, DP).
• Async multi-reward serving with HTTP scorers and overlap with the rollout phase.
• Stability tooling specifically for diffusion RL, including rollout correction and deterministic rollout/reward/trainer pipelines.
• Runs on Ascend NPU in addition to the usual GPU stack.

Caveats

• Several entries on the model support table are still “WIP” or “Planned” (LTX2.3, HunyuanImage-3.0), so the roster is promising but not fully delivered yet.
• It is explicitly built on top of verl, so if you’re already deep in that ecosystem, this is a focused spin-off rather than a from-scratch alternative.

Verdict

Worth a look if you’re doing RL post-training on diffusion, video, or omni-modality models and need more than a text-LLM trainer with a diffusion model duct-taped to it. If you’re only fine-tuning LLMs, stick with the base verl and save yourself the complexity.