bigscience-workshop/Megatron-DeepSpeed
A fork of Megatron-LM and Megatron-DeepSpeed for training large transformer language models at scale, used by the BigScience project.
Not currently ranked — collecting fresh signals.
star history
This repository provides tools for training transformer language models including BERT and GPT-2 at scale. It integrates NVIDIA’s Megatron-LM with Microsoft’s DeepSpeed for distributed training optimizations. The project is the primary codebase for the BigScience collaborative research initiative focused on large language model development.
Frequently asked
- What is bigscience-workshop/Megatron-DeepSpeed?
- A fork of Megatron-LM and Megatron-DeepSpeed for training large transformer language models at scale, used by the BigScience project.
- Is Megatron-DeepSpeed open source?
- Yes — bigscience-workshop/Megatron-DeepSpeed is an open-source project tracked on heatdrop.
- What language is Megatron-DeepSpeed written in?
- bigscience-workshop/Megatron-DeepSpeed is primarily written in Python.
- How popular is Megatron-DeepSpeed?
- bigscience-workshop/Megatron-DeepSpeed has 1.4k stars on GitHub.
- Where can I find Megatron-DeepSpeed?
- bigscience-workshop/Megatron-DeepSpeed is on GitHub at https://github.com/bigscience-workshop/Megatron-DeepSpeed.