What it does

This repo implements all 30 papers from Ilya Sutskever’s reading list — the one he told John Carmack teaches you “90% of what matters” in deep learning. Every notebook runs on pure NumPy, with synthetic data and visualizations built in. No PyTorch, no TensorFlow, no CUDA headaches. Just pip install numpy matplotlib scipy and start breaking things.

The coverage is genuinely broad: RNNs and LSTMs, ResNet and transformers, VAEs and scaling laws, plus curveballs like Kolmogorov complexity, AIXI, and a notebook on the thermodynamics of coffee mixing (really). Each implementation is designed for immediate execution and interactive tinkering.

The interesting bit

The author isn’t afraid of tedious work. The relational RNN notebook includes ~1,100 lines of manual backpropagation with numerical gradient checking. The GPipe notebook walks through pipeline bubble time analysis. These aren’t API demos; they’re anatomy lessons where you dissect the organism yourself.

Key highlights

• 30/30 papers complete, from “The Unreasonable Effectiveness of RNNs” to “Lost in the Middle”
• Pure NumPy implementations with synthetic data — no dataset downloads, no GPU required
• Notable depth on attention mechanisms (Bahdanau, transformers, pointer networks)
• Theory papers get serious treatment: MDL principle, Solomonoff induction, AIXI approximation
• Includes modern practical topics: RAG, dense passage retrieval, multi-token prediction, long-context position bias
• Author sells Colab-ready versions on Gumroad; the GitHub repo itself appears fully open

Caveats

• “Toy implementations” is the author’s own phrase; these are educational sketches, not production baselines
• Some notebooks are architecture demonstrations with forward passes only; training to convergence isn’t the goal
• The README has minor numbering glitches (two entries labeled “28,” truncated text on the second pass)

Verdict

Ideal if you’re tired of framework magic and want to see the gears mesh. Skip it if you need pretrained weights or SOTA benchmarks — this is a museum workshop, not a factory.