The checkpoint Rosetta Stone for Megatron-scale training
NVIDIA's Megatron Bridge turns Hugging Face checkpoints into Megatron training runs — and back — for a long list of frontier open models.

What it does
Megatron Bridge is a Python training library for Megatron-based models, with bidirectional Hugging Face ↔ Megatron checkpoint conversion as its signature trick. It ships recipes for pretraining, full SFT, long-context SFT with context parallelism, and LoRA/PEFT across a wide roster of models — Qwen, DeepSeek, GLM, Gemma, Nemotron, ERNIE, and many more, including MoE and multimodal (vision, audio, text) variants. It also carries performance recipes, such as the DeepSeek-V3 training configuration NVIDIA highlighted at 1,648 TFLOPS/GPU on GB300 in MLPerf Training v6.0.
The interesting bit
The boring part is the value: checkpoint conversion between HF and Megatron formats is the tax everyone pays when moving between “download a model” and “actually train it at scale,” and this library exists to make that round-trip routine. It also tracks new frontier releases with day-0 support — sometimes on a dedicated branch before merging to main — and publishes “verification cards” documenting which configurations were actually run and what the current limitations are, which is a refreshingly honest touch.
Key highlights
- Bidirectional HF ↔ Megatron checkpoint conversion, including quantized formats (FP8, MXFP4, NVFP4) with regenerated scale tensors.
- Recipes for pretraining, SFT, long-context SFT (32K–128K with context parallelism), and LoRA across dozens of models.
- Multimodal coverage: vision-language, audio-language, and omni models, plus an autoregressive-to-diffusion conversion workflow.
- Unified training and conversion launchers, plus performance recipes for GB200/GB300-class hardware.
- Verification cards per model listing verified configurations and known limitations.
Caveats
- The README is dominated by a news feed; the actual architecture and API documentation live elsewhere, so scope beyond “it converts and trains many models” requires digging into the docs.
- Some model support lives on temporary branches rather than main, so feature availability depends on which branch you track.
- Clearly aimed at multi-GPU/GB200-class setups — single-GPU hobbyists are not the audience.
Verdict
If you fine-tune large open models on NVIDIA clusters and have ever hand-written a checkpoint converter, this is squarely for you. If you train on one GPU with plain PyTorch, it’s a fascinating artifact of how the other half lives.
Frequently asked
- What is NVIDIA-NeMo/Megatron-Bridge?
- NVIDIA's Megatron Bridge turns Hugging Face checkpoints into Megatron training runs — and back — for a long list of frontier open models.
- Is Megatron-Bridge open source?
- Yes — NVIDIA-NeMo/Megatron-Bridge is open source, released under the Apache-2.0 license.
- What language is Megatron-Bridge written in?
- NVIDIA-NeMo/Megatron-Bridge is primarily written in Python.
- How popular is Megatron-Bridge?
- NVIDIA-NeMo/Megatron-Bridge has 1k stars on GitHub.
- Where can I find Megatron-Bridge?
- NVIDIA-NeMo/Megatron-Bridge is on GitHub at https://github.com/NVIDIA-NeMo/Megatron-Bridge.