A 105 GB model on a 16 GB Mac, via SSD streaming

What it does

Slotstream is a fully native Swift inference engine built on Apple’s MLX and Metal. It runs Qwen3.8-Flash-Next 4-bit — a 125-billion-parameter mixture-of-experts model weighing about 105 GB — on Macs with as little as 16 GB of RAM. The trick: only the active experts live in a memory cache; the rest stream from SSD on demand. It exposes Ollama-, OpenAI- and Anthropic-compatible APIs plus a Swift library, and can launch Claude Code, Codex, Pi, opencode, Hermes and fx directly against the local model.

The interesting bit

The engine predicts which experts the next layers will need and prefetches them from SSD before they’re requested — which changes speed but never the output. Auto mode also picks the memory target, cache size, speculative decoding and context window for your specific Mac, and refuses to start on 8 GB machines rather than thrash into swap.

Key highlights

• Fully offline after the one-time model download; no Python runtime, no cloud account.
• Measured 20.91–23.69 tok/s on a 48 GB M5 Pro at a 33 GB memory budget.
• Planning ranges from ~1–6 tok/s on 16–24 GB Macs up to ~15–27 tok/s on 48–96 GB.
• Prefix caching survives restarts via serve --prefix-cache-dir; follow-up turns reuse unchanged history.
• One maintained model at a time — all tuning effort goes into making this checkpoint work on constrained hardware.

Caveats

• Prompt processing is slow: a long prompt can take minutes to prefill, and first reads of a 2K prompt ranged roughly 14–28 s in their own study.
• The speed numbers are short controlled tests on one machine with an uncontrolled filesystem cache — not a sustained-session guarantee.
• On 96 GB+ Macs, fully-resident engines are faster; this is explicitly outside the optimization target.

Verdict

Worth a look if you have a 16–64 GB Apple Silicon Mac and want a genuinely large local model for coding agents without buying new hardware. If your Mac can hold the model in RAM, or you’re not on macOS, look elsewhere.