Rebuilding GPT-2 from an empty file, one clean commit at a time

What it does

This is a pedagogical repo that reconstructs the nanoGPT training pipeline starting from a completely empty file. It walks all the way to a working GPT-2 (124M) model, with the git history deliberately preserved as clean, sequential commits so you can follow the exact order of construction. A companion YouTube lecture narrates each commit, adding context to the transformer pieces as they appear. The same code can scale toward GPT-3 sizes if you bring more time and money.

The interesting bit

The repo treats version control as a narrative device: the commits are not squashed or polished, but left as a linear record of assembly. The README even hosts an active errata section for bugs caught after filming—like a forgotten register_buffer cleanup after switching to flash attention—so the project reads more like a living textbook than a sanitized release.

Key highlights

• Reproduces GPT-2 (124M) in roughly one hour and ~$10 of cloud GPU time.
• Every commit is a deliberate, explained step in the video lecture.
• Publishes errata openly—e.g., torch.autocast device_type quirks and a leftover register_buffer after switching to flash attention.
• Explicitly not a chatbot: the trained model is a raw internet-text “dreamer” with no supervised fine-tuning.
• Recommends litGPT or TinyLlama if you need production-grade code.

Caveats

• Apple Silicon users may hit friction: the author notes that mps gets mapped to a CPU device_type for torch.autocast, and he is not sure that is the intended PyTorch behavior.
• Chat-style interaction is out of scope; the model only generates internet-document continuations.

Verdict

Worth your time if you want to understand transformers by watching one being assembled bolt-by-bolt; skip it if you need a production training framework or a conversational model out of the box.