A Deep Learning Course That Builds a Toy PyTorch First

What it does

zero_to_gpt is a sequential course of Jupyter notebooks that takes you from gradient descent and dense networks up through transformers, GPU kernels, and distributed training. Each chapter pairs explanatory material with optional videos and implementation notebooks, asking you to recreate the code yourself rather than just reading. The stated goal is to get you from basic Python fluency to training your own GPT model.

The interesting bit

Instead of importing PyTorch on day one, the course makes you build a miniature version of it from scratch to understand backpropagation and computation graphs. That deliberate friction—suffering through your own autograd before touching the real framework—turns the later magic into actual mechanics.

Key highlights

• Covers the full stack from linear regression and RNNs to attention, efficient transformers, and (eventually) distributed training
• Applies theory to concrete problems like weather prediction and language translation
• Most early chapters include optional video companions alongside the notebooks
• Licensed CC BY-NC 4.0, so you can adapt the material for your own non-commercial teaching
• Includes optional side quests on CNNs, GRUs, and encoder-decoder architectures

Caveats

• The back half of the course is largely unfinished: lessons from transformers through GPT-2, GPU kernels, and distributed training are marked “coming soon,” and several promised implementations are missing
• Because it is built as a sequential curriculum, jumping straight to the transformer or GPT sections is not really supported

Verdict

Worth bookmarking if you are a Python-literate developer who wants to understand why GPT works, not just how to call it. Look elsewhere if you need a complete, end-to-end course today; this is very much a construction zone past chapter ten.