MoonshotAI/MoBA
A novel mixture-of-block attention mechanism for training efficient long-context LLMs using parameter-less top-k gating to select relevant KV blocks.
Not currently ranked — collecting fresh signals.
star history
MoBA is a trainable block sparse attention mechanism for transformer-based LLMs where query tokens learn to attend to the most relevant key-value blocks, reducing quadratic complexity of traditional full attention. It introduces a parameter-less top-k gating mechanism that dynamically selects the most informative blocks per query, enabling efficient long-context processing. The architecture allows seamless transitions between full and sparse attention modes during training and inference.
Frequently asked
- What is MoonshotAI/MoBA?
- A novel mixture-of-block attention mechanism for training efficient long-context LLMs using parameter-less top-k gating to select relevant KV blocks.
- Is MoBA open source?
- Yes — MoonshotAI/MoBA is open source, released under the MIT license.
- What language is MoBA written in?
- MoonshotAI/MoBA is primarily written in Python.
- How popular is MoBA?
- MoonshotAI/MoBA has 2.2k stars on GitHub.
- Where can I find MoBA?
- MoonshotAI/MoBA is on GitHub at https://github.com/MoonshotAI/MoBA.