Self-supervised Vision Transformers, the original DINO

What it does

DINO trains Vision Transformers—and the occasional ResNet—on raw images without labels. It learns visual representations by forcing a student network to match the output of a momentum teacher. The repo ships pretrained weights for several architectures, evaluation logs, and ONNX exports.

The interesting bit

The teacher is simply an exponential moving average of the student weights, which means the method generates its own supervision without negative pairs or external labels. The README makes clear this is the original implementation; the team now directs users to the newer DINOv2.

Key highlights

• Pretrained backbones—ViT-S, ViT-B, ResNet-50, and XCiT—are loadable via PyTorch Hub or direct download, with full checkpoints containing both student and teacher projection heads.
• Linear evaluation reaches 80.1% with ViT-B/8 and 77.0% with ViT-S/16; k-NN scores are provided for all releases.
• A vanilla 100-epoch run on ViT-Small takes 1.75 days on a single node with eight GPUs, and the code supports multi-node Slurm clusters via submitit.
• Each release includes training arguments, raw logs, and ONNX exports for reproducibility.

Caveats

• The authors flag DINOv2 as the current effort, making this repo largely a reference implementation of the original paper.
• Training requires the full ImageNet dataset and a multi-GPU setup; the documented environment uses Python 3.6 and PyTorch 1.7.1.

Verdict

Grab these weights if you are reproducing the original “Emerging Properties” paper or studying the lineage of self-supervised ViTs. If you need a modern backbone for downstream work, the authors suggest using DINOv2 instead.