microsoft/MInference
MInference is a sparse attention kernel that accelerates long-context LLM inference by up to 10x on A100 GPUs.
Not currently ranked — collecting fresh signals.
star history
MInference implements dynamic sparse attention to speed up the pre-fill phase of long-context language model inference. It uses approximate calculation techniques to reduce memory bandwidth bottlenecks while maintaining accuracy. The project provides optimized kernels compatible with major inference frameworks like SGLang and vLLM, achieving significant speedups especially for million-token context lengths.
Frequently asked
- What is microsoft/MInference?
- MInference is a sparse attention kernel that accelerates long-context LLM inference by up to 10x on A100 GPUs.
- Is MInference open source?
- Yes — microsoft/MInference is open source, released under the MIT license.
- What language is MInference written in?
- microsoft/MInference is primarily written in Python.
- How popular is MInference?
- microsoft/MInference has 1.2k stars on GitHub.
- Where can I find MInference?
- microsoft/MInference is on GitHub at https://github.com/microsoft/MInference.