Run 126 GB models on a 64 GB MacBook

What it does SwiftLM is a native Swift/Metal inference server for MLX models on Apple Silicon. It exposes a strict OpenAI-compatible API and ships as a single self-contained binary—no Python runtime or GIL in sight. It loads HuggingFace Safetensors directly and handles text, vision, and audio models.

The interesting bit The project’s real trick is treating NVMe SSD as a memory tier for Mixture-of-Experts layers. For genuinely enormous models—like a 126 GB DeepSeek-V4-Flash—it streams expert weights from disk while keeping peak physical RAM under 17 GB. Pair that with TurboQuant KV-cache compression and you can apparently squeeze 100K-token contexts into a 24 GB MacBook Pro that previously needed a 64 GB Mac Studio.

Key highlights

• SSD expert streaming with concurrent pread and async pipeline evaluation; README claims a 10× speedup on 122B+ models with only ~10 GB resident memory.
• TurboQuant + MTP speculative decoding cuts long-context memory dramatically: at 40K tokens GPU allocation drops from 54.8 GB to 23.9 GB on a Gemma 4-26B benchmark, and time-to-first token falls from 63 s to 34 s at 100K context.
• Native multimodal support: VLM via --vision and ALM via --audio, both processed through Metal.
• A needsMoeFlush gate fixed a bug where the SSD-streaming sync barrier was unconditionally flushing the GPU pipeline on full-RAM paths, boosting Qwen3.6-35B throughput from ~19 tok/s to ~62 tok/s on an M1 Ultra.

Caveats

• The DFlash speculative-decoding path is currently unsuitable for production on some models: the draft model uses pure greedy decoding that can lock into low-entropy repetition loops, and the README notes earlier 70 tok/s numbers were degenerate output.
• SSD streaming trades speed for capacity: at 40K context the SSD+TurboQuant config drops to 2.5 tok/s on Gemma 4-26B, so this is a capacity play, not a latency one.

Verdict Apple Silicon developers who want to self-host frontier-class models without renting a cloud GPU should look here—especially if you’re comfortable trading throughput for the ability to load models that simply do not fit in RAM. If you need consistently fast token generation above all else, stick to smaller in-RAM models.