pjlab-sys4nlp/llama-moe
A project that constructs sparse Mixture-of-Experts models by partitioning LLaMA's feed-forward networks into experts with top-K routing gates.
Not currently ranked — collecting fresh signals.
star history
LLaMA-MoE builds sparse Mixture-of-Experts models from LLaMA by partitioning feed-forward networks into experts and inserting top-K routing gates at each layer. The initialized MoE models are then continually pre-trained on optimized data sampling from SlimPajama and filtered datasets. This approach achieves reduced activated parameter counts (3.0-3.5B) compared to dense LLaMA models while maintaining language modeling capabilities.
Frequently asked
- What is pjlab-sys4nlp/llama-moe?
- A project that constructs sparse Mixture-of-Experts models by partitioning LLaMA's feed-forward networks into experts with top-K routing gates.
- Is llama-moe open source?
- Yes — pjlab-sys4nlp/llama-moe is open source, released under the Apache-2.0 license.
- What language is llama-moe written in?
- pjlab-sys4nlp/llama-moe is primarily written in Python.
- How popular is llama-moe?
- pjlab-sys4nlp/llama-moe has 1k stars on GitHub.
- Where can I find llama-moe?
- pjlab-sys4nlp/llama-moe is on GitHub at https://github.com/pjlab-sys4nlp/llama-moe.