Your SSD is now a VRAM tier

What it does It’s a llama.cpp fork that runs mixture-of-experts models too big for your RAM or VRAM by streaming the expert weights from SSD, using ordinary GGUF files. Tested models include DeepSeek-V4.1-Flash-1017 (763B total, 8–16B active) and Muse-Spark-Open (816B total, 42B active) — the kind of checkpoints that normally laugh at consumer hardware. It ships a terminal chat, a browser UI, and an OpenAI-compatible API, on macOS/Metal and Linux/CUDA, with Windows in progress.

The interesting bit MoE is what makes this viable at all: only a few experts fire per token, so the hot working set is small and the SSD becomes a slow-but-adequate tier for the rest. The benchmarks show it paying off — on an RTX 5060 Ti, decode goes from 0.24 token/s in llama.cpp to 2.55 token/s (10.5×) on DeepSeek; on a DGX Spark, prefill hits 127.5 token/s versus llama.cpp’s 2.7.

Key highlights

• Runs models far larger than physical memory; the launcher checks the memory budget and rejects what genuinely won’t fit
• Big speedups over stock llama.cpp on the same hardware — up to ~48× prefill and ~18× decode in the published tables
• Standard GGUF files from the usual community converters (Unsloth, bartowski, mrfakename); no new format
• First run measures your machine; there is no memory tuning knob — it uses what’s available
• Apache-2.0, with llama.cpp/ggml bundled under their original licenses

Caveats

• Decode rates on big models are still modest — 1–7 token/s depending on hardware; this is “it runs at all” territory, not local-speed territory
• The serve API has no authentication; anyone who can reach the port can use it
• Model downloads are large (about 470 GiB for DeepSeek or Muse)

Verdict Worth a look if you’ve ever wanted to poke at a 700B+ MoE model on a MacBook or a 16 GB GPU box. If your models already fit in memory, plain llama.cpp does that job without the SSD round-trips.