zhihu/ZhiLight
A high-performance LLM inference engine developed by Zhihu and ModelBest for accelerated model serving on GPUs.
Not currently ranked — collecting fresh signals.
star history
ZhiLight is a CUDA-based inference engine targeting PCIe GPUs that accelerates transformer-based LLMs including Llama, DeepSeek-V3/R1, and LLaMA3 variants. It implements tensor parallelism, pipeline parallelism, and dynamic batching alongside fused kernels for attention, layer norm, and quantization operations. The engine supports INT8, FP8, AWQ, and GPTQ quantization with custom memory management and async OpenAI-compatible APIs adapted from vllm.
Frequently asked
- What is zhihu/ZhiLight?
- A high-performance LLM inference engine developed by Zhihu and ModelBest for accelerated model serving on GPUs.
- Is ZhiLight open source?
- Yes — zhihu/ZhiLight is open source, released under the Apache-2.0 license.
- What language is ZhiLight written in?
- zhihu/ZhiLight is primarily written in C++.
- How popular is ZhiLight?
- zhihu/ZhiLight has 905 stars on GitHub.
- Where can I find ZhiLight?
- zhihu/ZhiLight is on GitHub at https://github.com/zhihu/ZhiLight.