← all repositories

karpathy/build-nanogpt

A step-by-step video lecture with accompanying code that builds a GPT-2 (124M) language model from an empty file.

5.2k stars Python LearningLanguage Models
build-nanogpt
Velocity · 7d
+7.2
★ / day
Trend
steady
star history

This repository provides a from-scratch reproduction of nanoGPT where git commits are organized incrementally to teach how a GPT model is constructed. Accompanied by a YouTube video lecture, the project starts with an empty file and progressively implements all components—transformer architecture, attention mechanisms, training loop, and tokenizer—until reaching a functional GPT-2 (124M) model that trains on internet documents. The learner can follow the commit history to understand each piece as it is built.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.