Pretrained Transformers that read amino acids, not English

What it does ProtTrans hosts a catalog of pretrained Transformer models—T5, BERT, Albert, XLNet, and Electra variants—trained on protein sequences instead of natural language. You pass in an amino-acid string and get back dense embeddings, either per-residue or averaged per-protein, ready for downstream tasks like secondary structure prediction, subcellular localization, or fine-tuning with LoRA.

The interesting bit The project consumed thousands of GPUs on Summit and hundreds of Google TPUs to pretrain these models so individual labs don’t have to. The resulting embeddings have already escaped the repository: UniProt now serves pre-computed ProtT5 embeddings for selected organisms, and third-party services like LambdaPP build on top of them.

Key highlights

• Multiple model families on HuggingFace, from ProtT5-XL up to XXL, plus BERT-style and Electra options
• Colab notebooks for feature extraction and common prediction tasks
• Recent additions include LoRA fine-tuning notebooks and “evo-tuning” on homologous sequences
• Pre-computed embeddings available via UniProt; no GPU required for lookup
• Standard benchmark datasets (CASP12, CB513, TS115, DeepLoc) provided for evaluation

Caveats

• The Generate, Visualization, and Benchmark sections still carry “More information coming soon” notices, suggesting those capabilities remain unfinished
• The README flags a tokenizer compatibility issue with recent transformers releases that requires manual workarounds

Verdict Computational biologists who need protein embeddings yesterday should bookmark this. If you’re an NLP architect hunting for novel attention mechanisms, look elsewhere—the architectures are borrowed wholesale; the contribution is the scale and the domain shift.