DeepMind's guide to scaling LLMs that was almost a dragon pun

What it does

This repository hosts How To Scale Your Model, a blog-style textbook that explains how TPUs work, how large language models run across them at scale, and how to choose parallelism strategies for training and inference that sidestep communication bottlenecks. It is published as a Jekyll-powered website, and the repo includes scripts to merge all chapters into a single Markdown or Word document for offline reading.

The interesting bit

The book was originally titled How To Scale Your Dragon after the Dreamworks film, which tells you the authors are serious about the hardware but not about marketing. It is written by working Google DeepMind engineers and focuses on the practical, often undocumented mechanics of TPU pod behavior rather than generic distributed systems theory.

Key highlights

• Written by Jacob Austin, Sholto Douglas, Roy Frostig, and others at Google DeepMind
• TPU-specific focus: covers hardware behavior, training and inference parallelism, and communication bottlenecks
• Published as a readable, Distill-style web book at jax-ml.github.io/scaling-book
• Includes a Python script to merge all chapters into a single Markdown file, plus Pandoc support for Word export
• Open for contributions via GitHub, though you’ll need to sign a Google CLA

Caveats

• This is a textbook, not a code library: do not expect importable JAX utilities or reference implementations
• Contributing requires signing a Google Contributor License Agreement

Verdict

Worth your time if you are designing or debugging distributed LLM training or inference on TPUs and want practical guidance from people who ship at scale. Skip it if you are looking for copy-paste JAX code or GPU-centric scaling advice.