The clearest 600-line GPT trainer, now deprecated

What it does

nanoGPT is a minimal PyTorch implementation for training and finetuning GPT-style transformers. Its two core files — train.py and model.py — are each roughly 300 lines of plain code that can reproduce GPT-2 (124M) on OpenWebText or finetune existing OpenAI checkpoints up to the 1.3B XL model. It scales from CPU/MacBook experiments up to multi-node A100 clusters via PyTorch Distributed Data Parallel.

The interesting bit

It is a rewrite of minGPT that deliberately prioritizes “teeth over education” — meaning it drops tutorial scaffolding in favor of working, hackable training code. The author notes that this simplicity makes it very easy to modify for custom needs.

Key highlights

• Reproduces GPT-2 (124M) in about 4 days on a single 8×A100 40GB node
• Supports finetuning from OpenAI GPT-2 checkpoints (124M through 1.3B)
• Runs on CPU, single GPU, or Apple Silicon MPS; scales to multinode training
• Each core file is roughly 300 lines of readable PyTorch
• Author deprecated the project in November 2025 in favor of successor nanochat, but keeps this repository available for posterity

Caveats

• Explicitly described by the author as “very old and deprecated”; new projects should likely use nanochat instead
• Multinode training without Infiniband will “most likely crawl,” per the README

Verdict

A must-read if you want to understand GPT mechanics through concise, hackable PyTorch. Avoid if you need an actively maintained trainer — the author has already moved on.