AviSoori1x/makeMoE
A PyTorch implementation of a sparse mixture-of-experts autoregressive character-level language model, inspired by Andrej Karpathy's makemore.
Not currently ranked — collecting fresh signals.
star history
This repository implements a sparse mixture-of-experts (MoE) language model from scratch using only PyTorch. The model replaces the standard feed-forward layer with a router and multiple expert networks, using top-k and noisy top-k gating mechanisms. The character-level autoregressive model trains on Shakespeare-like text, similar to the original makemore project. It includes both a standalone Python module and an explanatory Jupyter notebook walking through the architecture.
Frequently asked
- What is AviSoori1x/makeMoE?
- A PyTorch implementation of a sparse mixture-of-experts autoregressive character-level language model, inspired by Andrej Karpathy's makemore.
- Is makeMoE open source?
- Yes — AviSoori1x/makeMoE is open source, released under the MIT license.
- What language is makeMoE written in?
- AviSoori1x/makeMoE is primarily written in Jupyter Notebook.
- How popular is makeMoE?
- AviSoori1x/makeMoE has 811 stars on GitHub.
- Where can I find makeMoE?
- AviSoori1x/makeMoE is on GitHub at https://github.com/AviSoori1x/makeMoE.