ztxz16/fastllm
A backend-independent, C++-based high-performance large language model inference library supporting dense and MoE architectures with tensor parallelism and FP8/INT4 quantization.
Not currently ranked — collecting fresh signals.
star history
fastllm replaces PyTorch with custom C++ operators to deliver high-throughput LLM inference. It supports dense models (Qwen, Llama, Phi) and MoE models (DeepSeek, Qwen-moe), with tensor parallelism across multiple GPUs and mixed CPU/GPU inference for extremely large models. The library achieves 20+ tokens per second for full-precision DeepSeek R1 671B and 30+ tokens per second for INT4-quantized variants on a single GPU, targeting production deployment scenarios.
Frequently asked
- What is ztxz16/fastllm?
- A backend-independent, C++-based high-performance large language model inference library supporting dense and MoE architectures with tensor parallelism and FP8/INT4 quantization.
- Is fastllm open source?
- Yes — ztxz16/fastllm is open source, released under the Apache-2.0 license.
- What language is fastllm written in?
- ztxz16/fastllm is primarily written in C++.
- How popular is fastllm?
- ztxz16/fastllm has 4.9k stars on GitHub.
- Where can I find fastllm?
- ztxz16/fastllm is on GitHub at https://github.com/ztxz16/fastllm.