jy-yuan/KIVI
A tuning-free asymmetric 2-bit quantization method for KV cache to accelerate and compress LLM inference.
Not currently ranked — collecting fresh signals.
star history
KIVI is a quantization technique specifically for KV cache in large language models to reduce memory footprint and accelerate inference. It achieves compression by quantizing the KV cache to 2-bit asymmetric representation without tuning. The method is designed to work with popular LLM architectures including Llama, Mistral, and supports group-based attention for efficiency. It has been adopted into HuggingFace Transformers and integrated with flash attention for faster prefilling.
Frequently asked
- What is jy-yuan/KIVI?
- A tuning-free asymmetric 2-bit quantization method for KV cache to accelerate and compress LLM inference.
- Is KIVI open source?
- Yes — jy-yuan/KIVI is open source, released under the MIT license.
- What language is KIVI written in?
- jy-yuan/KIVI is primarily written in Python.
- How popular is KIVI?
- jy-yuan/KIVI has 413 stars on GitHub.
- Where can I find KIVI?
- jy-yuan/KIVI is on GitHub at https://github.com/jy-yuan/KIVI.