When the teacher has nothing left to teach: a field guide to failed distillation

What it does This is the code release for a systematic study of on-policy distillation (OPD) in LLM post-training, built on verl and LLaMA-Factory. The authors identify two conditions that decide whether OPD succeeds: student and teacher must share compatible thinking patterns, and the teacher must actually offer capabilities the student hasn’t already absorbed. They also propose two recovery strategies — off-policy cold start and teacher-aligned prompt selection — plus a GRPO baseline for comparison.

The interesting bit The token-level finding is the headline: successful OPD concentrates 97–99% of probability mass on a small shared token set at student-visited states. Meanwhile, same-family 1.5B and 7B teachers turn out to be distributionally indistinguishable from the student’s perspective — a bigger teacher from the same lineage may simply have nothing new to say. The paper also questions whether OPD’s “free” dense token-level reward scales to long-horizon distillation.

Key highlights

• Two failure conditions for OPD, validated via weak-to-strong reverse distillation experiments
• Token-level mechanism analysis showing progressive alignment on high-probability tokens
• Two practical recovery strategies: off-policy cold start and teacher-aligned prompt selection
• Top-k overlap diagnostics (distillation/overlap_ratio, distillation/overlap_token_advantage) merged upstream into verl
• Insights already adopted in MiniCPM5-1B’s RL+OPD recipe; released SFT and GRPO checkpoints on Hugging Face

Caveats

• Built on verl v0.7.0, whose built-in validation under-reports performance by 5–7 points; the README tells you to disable it and evaluate separately (fixed in verl v0.8.0)
• Experiments ran on 8× A800 80GB GPUs — this is a research reproduction repo, not a turnkey tool

Verdict Worth a close read if you’re distilling models and seeing suspiciously flat results — the failure diagnosis alone may save you a training run. Skip it if you just want a drop-in distillation library; this is science with scripts attached.