Run DeepSeek 671B on a single 10 GB GPU, no PyTorch required

What it does

fastllm is a C++ inference library built from custom operators, designed to run dense and mixture-of-experts LLMs without a PyTorch dependency. It handles model loading, quantization, and serving for architectures like Qwen, Llama, Phi, DeepSeek, and Qwen-MoE, and can distribute work across GPUs, CPUs, and NUMA nodes.

The interesting bit

Instead of assuming you own a rack of identical A100s, the library will happily tensor-parallel across odd numbers of mismatched cards, pin MOE expert layers to CPU or NUMA memory while keeping the dense path on GPU, and run FP8 math on elderly cards like the K80 or M40.

Key highlights

• Claims single-card DeepSeek R1 671B inference on any GPU with >10 GB VRAM via CPU offloading of MOE layers.
• Reports ~20 tps for FP8 original and ~30 tps for INT4 on a dual-socket server with one GPU.
• Supports multi-GPU tensor parallelism across odd counts (3, 5, 7) and mixed GPU-CPU splits.
• Runs on NVIDIA (K80 through 5090), AMD (MI50, 7900, ROCm), Huawei Ascend, and other domestic accelerators.
• FP8 inference works on CPU and GPU; front-end/back-end separation allows Android builds.

Caveats

• GGUF support is currently partial and requires an extra --ori parameter pointing to the original model config.
• Conda environments occasionally trigger installation errors; the authors suggest using venv instead.
• Windows support is limited to NVIDIA GPUs only, and AMD installs require manual ROCM setup because PyPI size limits prevent bundling those dependencies.

Verdict

Worth a look if you are sitting on heterogeneous or older hardware and want to run modern dense or MOE models without buying a cluster. Skip it if you are already invested in a mature PyTorch-based serving stack and only deploy on top-tier uniform hardware.