NVIDIA's kitchen-sink transformer trainer, now with a library inside

What it does

Megatron-LM is NVIDIA’s research codebase for training very large transformers across thousands of GPUs. It recently reorganized into two pieces: Megatron-LM (reference scripts for researchers and quick experiments) and Megatron Core (a composable library of GPU-optimized building blocks for framework builders). The Core handles the usual parallelism alphabet soup—TP, PP, DP, EP, CP—plus mixed precision down to FP4, and now exports to TensorRT-LLM.

The interesting bit

The project is refreshingly candid about its own benchmarking: it reports end-to-end MFU including data loading, logging, and checkpointing, not just the happy kernel path. It also notes that performance results were “measured without training to convergence”—a small honesty that saves you from chasing phantom numbers.

Key highlights

• Benchmarked up to 462B parameters on 6,144 H100 GPUs, hitting 47% MFU
• Superlinear weak scaling (41% → 47-48% MFU) as larger GEMMs get more efficient
• Strong scaling a ~175B model from 96 to 4,608 GPUs drops MFU from 47% to 42%—communication overhead exposed
• Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion
• Now developed openly on GitHub with community contributions accepted

Caveats

• Python 3.10 support is being dropped in the upcoming 0.17.0 release; minimum will be 3.12
• Building from source can exhaust memory without limiting parallel compilation jobs
• The README’s “model table” image is referenced but not described in text—what it actually compares is unclear

Verdict

Worth studying if you’re building a training framework or running large-scale LLM experiments on NVIDIA hardware. If you’re fine-tuning 7B models on a single A100, this is overkill with a side of configuration headache.