On-policy self-distillation fixes diffusion's supervision gap

What it does DiffusionOPSD post-trains text-to-image diffusion models to optimize reward functions without waiting for an entire multi-step denoising trajectory to finish. A frozen behavior policy generates low-noise query states and clean-output anchors; differentiable reward gradients build bounded positive and negative targets around each anchor, and the trainable policy fits those detached targets under a finite update budget. An EMA refreshes the behavior policy before the next round, so the supervision stays current.

The interesting bit The method separates target construction from model realization: reward and decoder computation graphs are discarded before policy fitting, which isolates target quality from how well the model currently approximates them. The README reports best held-out scores in nineteen of twenty reward-matched settings across SD3.5-M and Z-Image-Turbo, with training GPU-hours dropping by 40% and 63% relative to DiffusionNFT.

Key highlights

• Supports SD3.5-M at 512² and Z-Image-Turbo at 1024².
• Claims top held-out performance on 19 of 20 reward-matched benchmarks against prior post-training methods.
• Handles both single-reward and mixed-reward optimization across seven open-weight evaluators.
• Detaches reward-gradient targets before the policy loss step, keeping supervision explicitly bounded and on-policy.
• Training efficiency tables show roughly 0.60× and 0.37× the GPU-hours per 100 updates compared to DiffusionNFT on the respective backbones.

Caveats

• This is an external implementation of the paper, not the authors’ official release.
• Several reward models require manual checkpoint downloads, and ImageReward’s package metadata conflicts with the validated dependency stack.
• SD3.5-M is gated on Hugging Face, and Z-Image-Turbo needs a specific Diffusers build containing ZImagePipeline.

Verdict A solid bet if you’re researching reward-guided diffusion post-training and want a faster alternative to DiffusionNFT. Less appealing if you need a fully automated pipeline without gated model access or manual checkpoint management.