A 284B model on a 64 GB desktop, no GPU in sight

What it does It’s a bundle of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tooling for running sparse MoE models purely on CPU (x86_64 AVX-512/AVX2, ARM64 NEON/SVE, Apple Metal). The headline trick: shrinking DeepSeek-V4-Flash (284.33B total, 13.17B active) from a 76.87 GiB UD-IQ1_S build down to 56.56 GiB at 1.71 bpw, so the model plus a 256K KV cache fits in 64 GB of RAM with zero swap. Smaller 30B MoEs run on 16–32 GB laptops at 25–60 tok/s.

The interesting bit The quantization is surgical rather than uniform: shared experts, MLA projections, and embeddings stay in Q4_K/Q5_K/F32, while the 256 routed experts get crushed to IQ1_S with a second-order “SwiGLU-H2” scale-matching scheme, and boundary layers get gentler treatment. The drafter is a 1.64 MB Zobrist n-gram table borrowed from Stockfish’s chess engine — a genuinely odd lineage for speculative decoding.

Key highlights

• 44.4 tok/s prefill and 9.1–14.7 tok/s decode on a 16-core Zen 5 CPU, 0 GPUs
• SIMD cache-tiling patches claim +77% prefill / +53% decode vs stock kernels
• /dev/shm KV snapshots restore 40-tool Claude Code agent context in 13–33 ms
• Default mode evaluates all Top-6 experts with zero dropping (1.0 logit cosine similarity vs unpatched runtime)
• Hybrid mode puts the hot trunk on a 16 GB GPU with cold experts in RAM (~80.8 tok/s)

Caveats

• The custom Fractal-64 tensors (type ID 40) are rejected by stock llama.cpp, Ollama, and LM Studio — you’re locked into their patched runtime or prebuilt binary.
• 1.71 bpw is aggressive; the README reports a 100/100 quality gate for the 30B build, but quality claims for the 284B rest on their own eval scripts.
• The 284B path is Linux x86_64 territory; Windows is routed through a hybrid GPU+CPU path instead.

Verdict Worth a look if you have a big-RAM desktop and want frontier-ish MoE models without a GPU, or if you enjoy reading quantization schemes as spectator sport. If you’re on a Mac with 16 GB or need guaranteed upstream-llama.cpp compatibility, move along.