mit-han-lab/TinyChatEngine
On-device LLM/VLM inference engine written in C++ with SmoothQuant and AWQ quantization support for x86, ARM, and CUDA.
Not currently ranked — collecting fresh signals.
star history
TinyChatEngine is a from-scratch C++ implementation for running compressed LLMs and VLMs on edge devices like laptops, cars, and robots. It uses SmoothQuant and AWQ quantization techniques for model compression and supports Intel/AMD x86, Apple M1/M2 ARM, and Nvidia CUDA platforms. The library enables real-time inference with better privacy since data stays local.
Frequently asked
- What is mit-han-lab/TinyChatEngine?
- On-device LLM/VLM inference engine written in C++ with SmoothQuant and AWQ quantization support for x86, ARM, and CUDA.
- Is TinyChatEngine open source?
- Yes — mit-han-lab/TinyChatEngine is open source, released under the MIT license.
- What language is TinyChatEngine written in?
- mit-han-lab/TinyChatEngine is primarily written in C++.
- How popular is TinyChatEngine?
- mit-han-lab/TinyChatEngine has 958 stars on GitHub.
- Where can I find TinyChatEngine?
- mit-han-lab/TinyChatEngine is on GitHub at https://github.com/mit-han-lab/TinyChatEngine.