A 744B-parameter model, streamed off your NVMe drive

What it does

colibri is a pure-C inference engine for very large mixture-of-experts models. The insight it leans on: a 744B-parameter MoE like GLM-5.2 only touches ~40B parameters per token, and only ~11 GB of those actually change token to token. So instead of fitting the model in VRAM, colibri places it — dense layers and shared experts in RAM, routed experts on disk, pulled in through a cache that learns which experts your workload actually uses. A GPU, if you have one, holds the hottest experts.

The interesting bit

The tiering is the whole trick, and the README is refreshingly honest about the trade: where a weight lives changes speed, never the math. The default policy never silently alters precision or router semantics — less fast memory may cost tokens per second, but it won’t quietly redefine the model. Thirteen engines exist today, covering GLM, DeepSeek, Kimi, Qwen (including an image generator), plus decision/classification models, and it reads already-published GGUF files as-is.

Key highlights

• Runs a 744B MoE in ~10 GB of RAM with int4, streaming the rest from NVMe
• CPU, Vulkan (any GPU), and CUDA backends; GPU strictly optional
• Setup inspects your machine and recommends a model that actually fits
• Dashboard shows every expert firing in real time — a research platform, not just a runner
• OpenAI- and Anthropic-compatible local endpoints, plus an MCP server for AI assistants

Caveats

• Disk speed sets the pace for the big models: measured decode for DeepSeek V4 Flash ranges from 0.93 tok/s on CPU to ~1.6 with CUDA and two NVMe drives — usable, not snappy
• Several model entries in the tables have no measurements yet, so expect some unmarked territory
• Qwen-Image-2.1 carries a non-commercial licence

Verdict

If you’ve wanted to poke at frontier-scale MoE models without renting an H100, this is the most credible DIY route going. If you need fast interactive chat on huge models, check the measured tok/s tables first — streaming from disk is physics, not magic.