GPT-2 squeezed into 40 lines of pure NumPy

What it does

picoGPT is a minimal implementation of GPT-2 inference written entirely in NumPy. It downloads OpenAI’s pretrained weights and runs text generation through a forward pass that spans roughly 40 lines of code in gpt2_pico.py. There is no training code, no batching, and no sampling strategy beyond greedy decoding—just the bare mathematical operations needed to push tokens through the model.

The interesting bit

The project treats code golf as pedagogy. By refusing PyTorch, GPUs, and even sensible formatting, it exposes the underlying tensor operations that usually hide inside a framework. The accompanying blog post walks through the same logic in 60 lines, making this a deliberate exercise in stripping away abstraction until only the transformer math remains.

Key highlights

• Forward pass in ~40 lines of pure NumPy (gpt2_pico.py), with a more readable but still minimal version in gpt2.py
• Loads official OpenAI GPT-2 weights and hyperparameters directly via utils.py
• Supports all standard GPT-2 model sizes: 124M, 355M, 774M, and 1558M
• Uses OpenAI’s original BPE tokenizer (encoder.py) rather than a reimplementation
• Greedy decoding only: no temperature, top-k, top-p, or batch inference

Caveats

• Inference is explicitly described as “megaSLOW” and limited to greedy sampling
• gpt2_pico.py sacrifices readability for line count; the README admits it is not readable
• No training code or batch inference, so it is strictly a single-token-at-a-time curiosity

Verdict

Grab this if you want to see transformer inference decomposed to raw NumPy operations, or if you are teaching someone what a forward pass actually does under the hood. Skip it if you need speed, training, or anything resembling production inference.