karpathy/build-nanogpt
A step-by-step video lecture with accompanying code that builds a GPT-2 (124M) language model from an empty file.

This repository provides a from-scratch reproduction of nanoGPT where git commits are organized incrementally to teach how a GPT model is constructed. Accompanied by a YouTube video lecture, the project starts with an empty file and progressively implements all components—transformer architecture, attention mechanisms, training loop, and tokenizer—until reaching a functional GPT-2 (124M) model that trains on internet documents. The learner can follow the commit history to understand each piece as it is built.