Sber’s Russian GPT-3 family: five models, one language, 256 GPUs

What it does

ru-gpts is a collection of autoregressive transformer language models—ruGPT3XL, Large, Medium, Small, and ruGPT2Large—trained by SberDevices on massive Russian text corpora. The project publishes the pretrained weights, HuggingFace integration for smaller variants, and the exact DeepSpeed/Megatron scripts used to train the flagship XL model on the Christofari cluster. You can use them for text generation, fine-tuning, or downstream tasks like simplification and detoxification.

The interesting bit

The XL model mixes sparse and dense attention blocks and was trained in two stages: first on 80 billion tokens at 512 length for four epochs, then a final epoch at 2048 context. That two-stage curriculum—plus the raw compute bill of ten days on 256 Nvidia V100s—is the kind of expensive detail usually locked behind corporate walls. The README even documents the final test perplexity for each model, which is unusually transparent for a big-tech release.

Key highlights

• Five model sizes, from Small up to XL, with published test perplexities (12.05 for XL down to 17.4 for Medium).
• XL uses DeepSpeed and Megatron with sparse attention; smaller models drop into standard HuggingFace transformers.
• Trained on the Christofari cluster of Nvidia Tesla V100-SXM3 GPUs—up to 256 cards for the XL run.
• Active downstream ecosystem: papers and repos cover text simplification, detoxification, paraphrasing, and even generating Russian standardized-test questions.
• Colab notebooks for generation and fine-tuning are provided for both XL and smaller models.

Caveats

• The XL setup is brittle: it requires a custom apex fork, specific LLVM/Clang versions, and manually copying wrapper files into your Python environment.
• Smaller models are locked to transformers==4.24.0, which may conflict with newer dependencies.
• The README’s own Pushkin generation sample contains a glaring factual error, so don’t trust the output blindly.

Verdict

Worth a look if you need Russian text generation or fine-tuning without training from scratch. Skip it if you are only working with English or if wrestling with DeepSpeed sparse-attention builds sounds like a weekend ruined.