davidmrau/mixture-of-experts
PyTorch implementation of the sparsely-gated Mixture-of-Experts layer for scaling neural networks.
Not currently ranked — collecting fresh signals.
star history
This repository provides a PyTorch re-implementation of the sparsely-gated MoE layer described in the influential paper ‘Outrageously Large Neural Networks’. The implementation enables sparse activation of expert subnetworks within a larger model, allowing efficient scaling to billions of parameters. It was used as a reference implementation by FastMoE and includes working examples with CIFAR-10 and dummy data for training and evaluation.
Frequently asked
- What is davidmrau/mixture-of-experts?
- PyTorch implementation of the sparsely-gated Mixture-of-Experts layer for scaling neural networks.
- Is mixture-of-experts open source?
- Yes — davidmrau/mixture-of-experts is open source, released under the GPL-3.0 license.
- What language is mixture-of-experts written in?
- davidmrau/mixture-of-experts is primarily written in Python.
- How popular is mixture-of-experts?
- davidmrau/mixture-of-experts has 1.2k stars on GitHub.
- Where can I find mixture-of-experts?
- davidmrau/mixture-of-experts is on GitHub at https://github.com/davidmrau/mixture-of-experts.