microsoft/torchscale
Microsoft's PyTorch library for scaling Transformer architectures and developing foundation models including BitNet, RetNet, and LongNet.
Not currently ranked — collecting fresh signals.
star history
TorchScale provides foundational architectures for large language models and multimodal systems. It implements research innovations such as DeepNet for stable deep scaling, Magneto for general-purpose modeling across language/vision/speech, and BitNet/RetNet as potential Transformer successors. The library focuses on training stability, modeling capability, and efficiency through sparse Mixture-of-Experts and long-context length extrapolation techniques.
Frequently asked
- What is microsoft/torchscale?
- Microsoft's PyTorch library for scaling Transformer architectures and developing foundation models including BitNet, RetNet, and LongNet.
- Is torchscale open source?
- Yes — microsoft/torchscale is open source, released under the MIT license.
- What language is torchscale written in?
- microsoft/torchscale is primarily written in Python.
- How popular is torchscale?
- microsoft/torchscale has 3.1k stars on GitHub.
- Where can I find torchscale?
- microsoft/torchscale is on GitHub at https://github.com/microsoft/torchscale.