Your 12 GB phone can stream a 91 GB MoE model, lossless

What it does

BigMoeOnEdge is a C++ inference engine that runs Mixture-of-Experts LLMs on devices whose RAM is a fraction of the model size. It sits atop llama.cpp’s public API and streams individual expert weights from flash storage only when a token routes to them, keeping the shared dense weights resident. The result is byte-identical to a fully loaded model, whether you are running a 17 GB Gemma variant or a 91 GB DeepSeek V4 Flash model on a 12 GB phone.

The interesting bit

The project does not fork llama.cpp; it registers new MoE architectures in a single registry row and lets llama.cpp handle quantization, tokenization, and math. That means MXFP4 and Q4_K_M stream through the same path, and a new architecture like Qwen3.8-Flash-Next can work the day weights drop. It also reads multi-shard GGUFs natively, so there is no merge step for models split by Hugging Face’s file limits.

Key highlights

• Lossless CPU-only inference for models several times larger than available RAM.
• Built as a layer on llama.cpp, not a fork; inherits quantizers, chat templates, and tokenizers automatically.
• Supports multi-shard GGUFs natively—no merge step for 50 GB+ model files.
• Tunable expert cache, direct I/O bypassing the OS page cache, and overlapping compute with flash reads.
• Optional lossy speed-ups—like dropping cold experts or drafting with MTP—when the alternative is no output at all.

Caveats

• Several tuning knobs—predictive prefetch, cache-aware substitution, route-ahead—are marked experimental, and the README notes that predictive prefetch lost its on-device A/B test.
• The lossless path is reliable, but the lossy shortcuts are not interchangeable; reducing active experts cuts compute differently than dropping cache-missed experts.
• Speed is flash-bound: the 91 GB DeepSeek V4 model runs at roughly 0.94 tok/s on the demo 12 GB phone.

Verdict

Reach for this if you are trying to run frontier-scale MoE models on memory-constrained phones, laptops, or edge boxes without a GPU. Skip it if you have enough RAM to keep the whole model resident and just want standard llama.cpp performance.