Zefan-Cai/KVCache-Factory
A unified framework implementing multiple KV cache compression methods to accelerate inference for auto-regressive language models.
Not currently ranked — collecting fresh signals.
star history
KVCache-Factory provides implementations of various KV cache compression techniques including PyramidKV, SnapKV, H2O, and StreamingLLM. The project targets inference optimization for large language models by reducing memory footprint and computational overhead during autoregressive generation. It supports popular models like LLaMA and Mistral with multi-GPU inference capabilities and integrates with Flash Attention v2 for performance optimization.
Frequently asked
- What is Zefan-Cai/KVCache-Factory?
- A unified framework implementing multiple KV cache compression methods to accelerate inference for auto-regressive language models.
- Is KVCache-Factory open source?
- Yes — Zefan-Cai/KVCache-Factory is open source, released under the MIT license.
- What language is KVCache-Factory written in?
- Zefan-Cai/KVCache-Factory is primarily written in Python.
- How popular is KVCache-Factory?
- Zefan-Cai/KVCache-Factory has 1.4k stars on GitHub.
- Where can I find KVCache-Factory?
- Zefan-Cai/KVCache-Factory is on GitHub at https://github.com/Zefan-Cai/KVCache-Factory.