Apple Silicon LLM Inference That Uses the Model’s Own Drafting Heads

What it does

MTPLX is a native macOS app and CLI that runs local LLMs on Apple Silicon through MLX. It finally puts the multi-token prediction heads in recent models—Qwen 3.5/3.6, Gemma 4—to work: the model drafts several tokens ahead, verifies them in one batched forward pass, and keeps only what survives exact rejection sampling. The project cites a 1.6× speedup on a 16 GB M4 Mac mini and up to 2.24× on an M5 Max running Qwen 3.6 27B at temperature 0.6, with the same output distribution as standard decoding.

The interesting bit

Most speculative-decoding schemes either burn RAM on a separate draft model or cheat with greedy argmax that silently warps sampling. MTPLX does neither. It uses the target model’s own spare heads, then applies the Leviathan and Chen rejection-sampling theorem with residual correction so temperature=0.6 and top_p=0.95 behave exactly as they would normally. It also auto-tunes draft depth per machine—accounting for chip, memory bandwidth, and thermals—rather than pretending one size fits all.

Key highlights

• Live dashboard tracking tokens per second, per-depth acceptance rates, cache state, and system pressure while the model runs.
• OpenAI- and Anthropic-compatible local API with streaming, tool calls, and an optional SSD cache that restores sessions across restarts.
• Forge converts Hugging Face repos into MTPLX-ready MTP models, trains the adapter, and verifies the speedup on your hardware before you publish.
• Explicit compatibility tiers: mtplx inspect labels models verified, unverified, incompatible, or missing MTP heads, and refuses to run unverified ones unless forced.
• Thermal modes with a crash-safe watchdog that restores automatic fan control even after a kill -9.

Caveats

• Apple Silicon only; the README bluntly tells Linux users to use vLLM instead.
• Requires macOS 14+ and an M1 or newer; the 27B model wants 32 GB of RAM or more.
• Tightly bound to the MLX ecosystem and to model families that actually ship MTP heads.

Verdict

Grab it if you run local LLMs on a modern Mac and want exact sampling without a second model eating your RAM. If you are on CUDA or non-Apple silicon, there is nothing to see here.