Frontier MoE models on gaming GPUs, via hand-written SASS

What it does It ships kernel-side patches on top of official vLLM images that make frontier MoE checkpoints run on Blackwell consumer/workstation cards (sm_120), where the official images either won’t start or the model doesn’t fit. The flagship line runs DeepSeek-V4.1-Flash on 4× RTX PRO 6000 with a rebuilt decode step and prefill. Other lines cover a GPU+CPU hybrid on 4× RTX 5090, 2-bit experts for GLM-5.2, DeepSeek-V4-Flash and Kimi-K2.7-Code, and Qwen3.8-Flash-Next-FP8.

The interesting bit Every patch is claimed bit-identical to the vLLM path it replaces unless documented otherwise, and each sits behind a runtime switch — so you can verify the patches rather than trust them. The 2-bit line’s SASS kernels are produced by the author’s own assembler (cubit) built on a machine-readable SM120 ISA database, and every kernel in the DeepSeek image round-trips through it bit for bit. The hybrid line keeps 120 of 384 experts per layer on the GPUs and computes the rest on the CPU from the checkpoint’s own bytes, still bit-identical to the all-resident path.

Key highlights

• DeepSeek-V4.1-Flash on 4× RTX PRO 6000: ~293 tok/s prose / ~802 tok/s code single-stream, 163K-token prefill in 7.2 s (vs ~15.5 s on the official image).
• KV cache of 9.62M tokens on the GPUs plus host RAM and a disk tier — 18× concurrency at 512K context, vs 1.49M with official KV formats.
• Quality checks: GSM8K-200 194/200, needle tests pass at 29K/106K/400K, and identical requests return identical logits at any batch size.
• Measurement records are published per line under docs/benchmarks/, not just headline numbers.
• Apache 2.0, with per-line docs, op-level kernel tests, and a changelog of served images.

Caveats

• Hardware requirements are steep: the flagship line needs four RTX PRO 6000s, ~330 GB of host shared memory, and PCIe peer-to-peer; an RTX 5090 doesn’t fit the model at TP4.
• The 2-bit line is frozen (builds, fixes accepted, no development) and the MoET hybrid is still on a branch, not merged.
• First startup takes ~22 min while FlashInfer autotunes.

Verdict Worth a look if you have Blackwell cards and want frontier MoE inference that the official vLLM images can’t deliver — the bit-identical discipline and published benchmarks are unusually rigorous. If you’re on datacenter GPUs or already served by upstream vLLM, this isn’t for you.