SqueezeAILab/SqueezeLLM
A post-training quantization framework for LLMs that uses dense-and-sparse weight decomposition to enable serving large models with reduced memory footprint.
Not currently ranked — collecting fresh signals.
star history
SqueezeLLM implements Dense-and-Sparse Quantization, a technique that splits weight matrices into a heavily compressible dense component and a sparse component preserving sensitive outlier values. This approach allows serving large language models like Vicuna within 6 GB of memory while achieving higher accuracy than full-precision baselines. The framework is designed for efficient LLM serving and has been integrated into the vLLM inference engine.
Frequently asked
- What is SqueezeAILab/SqueezeLLM?
- A post-training quantization framework for LLMs that uses dense-and-sparse weight decomposition to enable serving large models with reduced memory footprint.
- Is SqueezeLLM open source?
- Yes — SqueezeAILab/SqueezeLLM is open source, released under the MIT license.
- What language is SqueezeLLM written in?
- SqueezeAILab/SqueezeLLM is primarily written in Python.
- How popular is SqueezeLLM?
- SqueezeAILab/SqueezeLLM has 722 stars on GitHub.
- Where can I find SqueezeLLM?
- SqueezeAILab/SqueezeLLM is on GitHub at https://github.com/SqueezeAILab/SqueezeLLM.