Your contrastive loss is secretly just cross-entropy

What it does The repo is a reference implementation of Supervised Contrastive Learning and, incidentally, SimCLR. At its core is a single SupConLoss in losses.py that takes L2-normalized features and optional labels. Feed it labels and it performs supervised contrastive learning; omit them and it degenerates to SimCLR. The code uses CIFAR as the primary example and includes a pretrained ImageNet model that tops 79.1% top-1 accuracy using a momentum encoder trick.

The interesting bit The authors later point out—citing the StableRep paper—that the supervised contrastive loss is mathematically just cross-entropy in disguise. That meta-admission makes the whole exercise less about inventing a new loss and more about reframing how we think about pulling same-class embeddings together.

Key highlights

• One loss function toggles between supervised and unsupervised modes by checking for labels.
• CIFAR-10 accuracy climbs from 95.0% (cross-entropy) to 96.0%; CIFAR-100 goes from 75.3% to 76.5%.
• Includes an ImageNet checkpoint hitting 79.1% top-1 with a MoCo-style momentum encoder.
• t-SNE visualizations compare embedding clusters across standard cross-entropy, SupContrast, and SimCLR.

Caveats

• The --syncBN flag currently does nothing because the code relies on DataParallel rather than DistributedDataParallel.
• The README itself points to a cleaner reimplementation of the loss in the StableRep repository if you find this one hard to parse.

Verdict Worth a look if you are researching contrastive learning or need a baseline that bridges supervised and self-supervised training. Skip it if you want production-ready infrastructure—this is reference code, not a framework.