Dropout alone trains sentence embeddings, no labels required

What it does SimCSE learns to compress sentences into dense vectors using contrastive learning. In its unsupervised mode, a sentence is trained to agree with itself across two separate forward passes; the supervised mode adds natural-language-inference pairs, treating entailments as positives and contradictions as hard negatives. The repository provides pre-trained BERT and RoBERTa checkpoints, a small Python embedding utility, and evaluation code for semantic textual similarity tasks.

The interesting bit The unsupervised approach is almost insultingly simple: standard dropout is the only noise source, so the model learns by matching a sentence to its own stochastically masked twin. No paraphrasing, no synonym substitution, no extra unlabeled corpus—just the same text fed through twice with different random masks.

Key highlights

• Unsupervised models are trained on Wikipedia; supervised variants add NLI datasets and use contradiction pairs as hard negatives.
• Released checkpoints span BERT and RoBERTa (base and large), with the supervised RoBERTa-large averaging 83.76 on STS benchmarks.
• The simcse package exposes encoding, cosine-similarity, and Faiss-backed search, and all checkpoints load natively in HuggingFace Transformers.
• Evaluation relies on a modified SentEval suite that reports Spearman correlation across seven STS tasks.
• A small demo website demonstrates sentence retrieval, borrowing its frontend from the DensePhrases project.

Caveats

• Faiss search has known compatibility problems on Nvidia Ampere GPUs (RTX 3090 and A100), forcing a CPU fallback on newer hardware.
• Pooling strategy is tightly coupled to model flavor: supervised models want cls, unsupervised checkpoints need cls_before_pooler, and vanilla BERT/RoBERTa prefers averaging the first and last layers.
• The original published scores were obtained with PyTorch 1.7.1, so reproduction on newer releases may drift slightly.

Verdict A solid choice if you need sentence embeddings off the shelf or want to test how little augmentation contrastive learning can get away with. Pass if you are already locked into another embedding pipeline and have no patience for pooling-method bookkeeping.