A 552B-parameter model, served from a single RTX 4090

What it does

Xolotl is a C++ inference server that runs DeepSeek-V4.1-Flash (552B MoE), GLM-5.3-Flash, Qwen3.8 variants and LFM3-Omni behind one OpenAI- and Anthropic-compatible API. The trick is expert tiering: whatever fits in VRAM stays there, the rest lives in pinned RAM or on the NVMe drive, with a “Xolotl Direct” path that streams experts from NVMe straight into VRAM at 3.1 ms per expert. On a 4090 with 110 GB of DDR4, DeepSeek-V4.1 reportedly hits 32.6 tok/s on real agent tasks (41.1 in fast mode) with 512K context.

The interesting bit

Fast mode is a compress-then-correct scheme: a ~40 GB sidecar of 2-bit copies of the biggest expert formats replaces about half the bytes crossing the bus, buying +26% decode for roughly one changed token in a hundred — same Gauntlet score. The benchmarks are also refreshingly honest: their own quantized builds are scored against each model’s full-precision API on the same six agentic tasks, and the local DeepSeek build (4.99) beats its own API (4.49).

Key highlights

• 552B DeepSeek-V4.1-Flash at ~33 tok/s agent decode, 512K context, on one RTX 4090 + 110 GB RAM
• Xolotl Direct: NVMe-to-VRAM expert writes at 3.1 ms vs 11 ms
• Fast mode: +26% decode, opt-in, lossy (~1 token in 100 differs)
• Community runs on 12 GB cards: 8–10 tok/s on DeepSeek, 39–48 on Qwen3.8-Flash-Next
• Built on the Strata engine, extended by 70,000+ lines; Linux, NVIDIA, MIT

Caveats

• Linux and NVIDIA only (Windows 11 works for DeepSeek-V4.1 alone); RAM matters as much as VRAM
• GLM-5.3-Flash has no Gauntlet score yet (“to come”), so its quality claims rest on the API comparison
• Xolotl Direct is labeled experimental; fast mode is lossy by design

Verdict

Worth a look if you have a big-RAM gaming PC and want frontier MoE models as local agents without a datacenter. If you’re on Apple silicon or modest hardware, the docs’ own community numbers suggest tempering expectations.