Neural networks, spelled out: from micrograd to GPT

What it does

This repository holds the Jupyter notebooks built during Karpathy’s “Neural Networks: Zero to Hero” video course. It walks from a tiny autograd engine (micrograd) through bigram and MLP language models, a manual backprop exercise, a WaveNet-style CNN, and finally a from-scratch GPT and its tokenizer. The materials assume only basic Python and a faint memory of calculus; everything else is derived in the open.

The interesting bit

The pedagogy treats abstraction as a leaky roof you must open to inspect the plumbing. You build a working GPT while deliberately avoiding PyTorch autograd in one exercise, hand-cranking gradients through BatchNorm and embedding tables so tensor-level optimization stops feeling like witchcraft.

Key highlights

• Starts with micrograd, a scalar-level autograd engine, and ends with a functioning GPT-2/GPT-3 style transformer
• Includes a manual backpropagation exercise through a full MLP + BatchNorm stack without calling loss.backward()
• Covers the full autoregressive language modeling lifecycle: training, sampling, loss evaluation, and tokenization with Byte Pair Encoding
• Materials are explicitly ongoing; BatchNorm is in, but residual connections and Adam optimizer are still queued for future videos
• MIT licensed

Caveats

• The repository is a capture of live-coding notebooks, not a standalone library or polished course platform
• Several advanced topics (causal dilated convolutions, residual connections, Adam optimizer) are noted as not yet covered
• Most exercises and context live in YouTube video descriptions and external Colab links rather than the repo itself

Verdict

Grab this if you want to understand transformers by unbolting every layer, and if you learn best by typing alongside a video. Skip it if you need a self-contained reference text or a drop-in training framework.