OpenGVLab/OmniQuant
LLM quantization framework that compresses model weights (W4/W3/W2) and activations for efficient inference on GPUs and mobile devices.
Not currently ranked — collecting fresh signals.
star history
OmniQuant provides quantization algorithms for large language models, supporting weight-only quantization (W4A16/W3A16/W2A16) and weight-activation quantization (W6A6/W4A4). The repository includes a model zoo with pre-quantized checkpoints for LLaMA, LLaMA-2-Chat, OPT, Falcon, and Mixtral-7Bx8. It also integrates with MLC-LLM to deploy quantized models on GPUs and mobile hardware, enabling efficient inference for resource-constrained environments.
Frequently asked
- What is OpenGVLab/OmniQuant?
- LLM quantization framework that compresses model weights (W4/W3/W2) and activations for efficient inference on GPUs and mobile devices.
- Is OmniQuant open source?
- Yes — OpenGVLab/OmniQuant is open source, released under the MIT license.
- What language is OmniQuant written in?
- OpenGVLab/OmniQuant is primarily written in Python.
- How popular is OmniQuant?
- OpenGVLab/OmniQuant has 902 stars on GitHub.
- Where can I find OmniQuant?
- OpenGVLab/OmniQuant is on GitHub at https://github.com/OpenGVLab/OmniQuant.