Rebuilding vLLM From Scratch, One MLX Array at a Time

What it does

This course is a three-week sprint through LLM inference engineering on Apple Silicon. You write pure Python against MLX’s low-level array APIs to construct a Qwen3-based server from the transformer block up: attention, RoPE, grouped query attention, and sampling in week one, then KV caches, continuous batching, and flash attention in week two. The aim is not a polished product but a working mental model of why serving is hard.

The interesting bit

By stripping out every high-level neural-network framework, the author forces you to sit with the actual matrix operations. It is a deliberately contrarian bet that understanding matmul beats understanding model.generate(), and that a MacBook is enough to learn the trade.

Key highlights

• Implements core transformer components—attention, RoPE, RMSNorm, MLP—from scratch in Python.
• Builds a simplified vLLM-like serving stack including KV cache, continuous batching, chunked prefill, and flash attention for both CPU and GPU.
• Targets Qwen3 with official 4-bit MLX weights, keeping hardware requirements modest and predictable.
• Pairs a public repository with a free online book and a Discord community.
• Week 1 and 2 are fully documented and tested; Week 3 (paged attention, speculative decoding, MoE, agents, RAG) is partially underway.

Caveats

• Week 3 topics—MoE, RAG pipelines, AI agents, and long-context handling—are still largely works in progress.
• Several production techniques, such as quantized KV caches and prefix caching, are explicitly out of scope.
• The entire stack is Apple Silicon–only; there is no NVIDIA GPU path.

Verdict

Ideal for systems engineers who want to feel inference bottlenecks in their own code rather than read about them in a paper. Not for anyone who needs a drop-in production server or a Windows/Linux workstation.