Teaching LLMs to learn from their own mistakes

What it does

SDPO fine-tunes large language models with reinforcement learning in verifiable domains such as math and code. It augments standard on-policy optimization by distilling the model’s own high-reward trajectories—and any textual feedback from failures—back into the policy. The result is a denser training signal that uses error messages and judge evaluations without requiring an external teacher or explicit reward model.

The interesting bit

The policy acts as its own teacher: it conditions on feedback from failed attempts, predicts what it should have done, and distills those feedback-informed next-token probabilities back into itself. The same trick works at test time, where the model iteratively refines hard problems by treating its own best generations as in-context demonstrations.

Key highlights

• Converts rich textual feedback—runtime errors, judge evaluations, stack traces—into dense supervision.
• Outperforms GRPO on chemistry reasoning and LiveCodeBenchv6 when rich feedback is available, according to the README benchmarks.
• Falls back to implicit self-distillation from high-reward rollouts when environment feedback is sparse.
• Supports test-time self-distillation to bootstrap harder coding problems without additional training.
• Built as an extension to the verl RL framework, with Docker images pre-configured for NVIDIA GH200 clusters.

Caveats

• Blackwell GPU support (RTX 50 series, B100/B200) is explicitly flagged as not fully tested.
• The README is dominated by cluster deployment, Docker builds, and reproduction scripts; this is an infrastructure-heavy research codebase, not a lightweight library.
• Results shown focus on specific models and benchmarks (Olmo3-7B-Instruct, chemistry, LiveCodeBench); broader generalization is not demonstrated.

Verdict

A solid bet if you are already running GPU clusters for LLM reinforcement learning and want to exploit detailed environment feedback instead of collapsing it to a scalar reward. Everyone else should admire the idea from a distance.