DKD and DOT live here, alongside a dozen distillation baselines

What it does

MDistiller is a PyTorch library that implements classical knowledge distillation algorithms—such as KD, FitNet, AT, and CRD—for standard computer vision benchmarks including CIFAR-100, ImageNet, and MS-COCO. It also serves as the official code release for two research papers: Decoupled Knowledge Distillation (CVPR 2022) and DOT: A Distillation-Oriented Trainer (ICCV 2023). The repo provides training configurations, pre-trained checkpoints, and an extensible base class so you can plug in your own distiller without rewriting the training loop.

The interesting bit

Rather than dumping a single paper script, the authors built a small ecosystem. DOT is designed to pair with existing distillers like vanilla KD, consistently lifting accuracy across CIFAR-100, Tiny-ImageNet, and ImageNet without replacing the underlying loss.

Key highlights

• Bundles 12+ classical distillation methods with a clear support matrix for CIFAR-100, ImageNet, and MS-COCO.
• Provides official DKD and DOT implementations, plus pre-trained checkpoints and reproducible benchmark tables.
• DOT boosts vanilla KD by up to several percentage points across CIFAR-100, Tiny-ImageNet, and ImageNet.
• DKD posts strong gains over standard KD on CIFAR-100; for instance, ResNet32x4 → ResNet8x4 jumps from 73.33 % to 76.32 %.
• Exposes a Distiller base class and registration hooks for dropping in custom methods without touching the training loop.

Caveats

• README commands contain minor typos (eval.p instead of eval.py), so copy-paste with caution.
• MS-COCO support is thin: only ReviewKD and DKD are listed as compatible.
• Weights & Biases logging is on by default and must be disabled via a config file edit.

Verdict

Grab it if you need a benchmark harness for standard vision distillation or want to extend DKD and DOT. Skip it if you are looking for generative-model or NLP distillation—this is strictly a computer vision shop.