This repository provides tools for training transformer language models including BERT and GPT-2 at scale. It integrates NVIDIA’s Megatron-LM with Microsoft’s DeepSpeed for distributed training optimizations. The project is the primary codebase for the BigScience collaborative research initiative focused on large language model development.