Fixing SFT's hidden reward bias with a single detached gradient

What it does Dynamic Fine-Tuning (DFT) is a drop-in replacement for standard supervised fine-tuning that tweaks how each token contributes to the loss. Instead of treating every token equally, it rescales the objective by the model’s own detached predicted probability for that token. The authors argue this corrects a problematic reward structure baked into vanilla SFT gradients, which they blame for the generalization gap between SFT and reinforcement learning.

The interesting bit The entire intervention is literally one line of code—multiply the loss by the detached softmax probability—yet the authors report gains over standard SFT on math benchmarks and competitive results against offline RL. Major training frameworks including Hugging Face TRL, LLaMA-Factory, and ms-swift have already merged support, suggesting the community finds the tradeoff worthwhile.

Key highlights

• One-line loss modification: each token’s loss is scaled by its predicted probability, detached to stop gradient flow back through the rescaling term.
• Framed as a reward-rectification fix: the authors claim vanilla SFT gradients implicitly encode a skewed reward that restricts generalization.
• Released scripts and checkpoints are built on the verl framework and target mathematical chain-of-thought reasoning.
• Already integrated into popular training stacks (TRL, LLaMA-Factory, ms-swift).
• The authors acknowledge the current evaluation is limited to math-focused datasets and models up to 7 billion parameters.

Caveats

• DFT appears weaker on low-entropy tasks with a single, constrained ground-truth answer; the authors explicitly caution that it does not replace SFT in all domains.
• Community feedback includes reported failures on literary and financial tasks.
• Code-focused benchmarks and larger-scale validation are noted as pending future releases.

Verdict Try it if you are fine-tuning reasoning models and suspect SFT has hit a generalization wall, but look elsewhere if your task is deterministic extraction with one correct path. The one-line diff makes it a low-cost experiment.