mit-han-lab/llm-awq
AWQ provides low-bit weight quantization (INT3/4) for efficient LLM compression and acceleration with optimized CUDA kernels.
Not currently ranked — collecting fresh signals.
star history
AWQ implements activation-aware weight quantization to compress large language models into 3-4 bit precision while maintaining accuracy. The library includes pre-computed quantization parameters for popular LLMs (LLaMA, OPT, CodeLlama, Vicuna, VILA, LLaVA) and provides memory-efficient 4-bit linear layers in PyTorch with custom CUDA kernels for fast context and decoding inference. TinyChat 2.0 extends this work to deliver state-of-the-art prefilling speeds for LLMs and VLMs on edge devices including RTX 4090 and Jetson Orin.
Frequently asked
- What is mit-han-lab/llm-awq?
- AWQ provides low-bit weight quantization (INT3/4) for efficient LLM compression and acceleration with optimized CUDA kernels.
- Is llm-awq open source?
- Yes — mit-han-lab/llm-awq is open source, released under the MIT license.
- What language is llm-awq written in?
- mit-han-lab/llm-awq is primarily written in Python.
- How popular is llm-awq?
- mit-han-lab/llm-awq has 3.6k stars on GitHub.
- Where can I find llm-awq?
- mit-han-lab/llm-awq is on GitHub at https://github.com/mit-han-lab/llm-awq.