Neural nets from scratch, in 150 lines of Python

What it does

micrograd is a minimal reverse-mode automatic differentiation engine and a tiny neural-network library built on top. The engine tracks scalar operations—every add, multiply, and ReLU—building a dynamic computation graph, then backpropagates through it. A companion nn module adds layers and neurons with an API that apes PyTorch just enough to feel familiar.

The interesting bit

The perverse, instructive choice to stay strictly scalar. Each neuron is exploded into dozens of primitive operations, which is comically inefficient but makes the gradient flow visible and the code readable. The README notes this is “enough to build up entire deep neural nets doing binary classification”—a claim the demo.ipynb notebook backs up with a working 2-layer MLP on the moons dataset.

Key highlights

• ~100 lines for the autograd engine, ~50 for the neural net layer
• PyTorch-like Value objects with overloaded operators (+, *, **, relu())
• backward() computes gradients via reverse-mode autodiff over the dynamically built DAG
• trace_graph.ipynb generates GraphViz visualizations showing data and gradients per node
• Unit tests verify correctness against PyTorch itself
• MIT licensed, installable via pip

Caveats

• Scalar-only operations mean this is dog-slow for any real workload; the README admits it is “potentially useful for educational purposes” and nothing more
• The demo.ipynb and trace_graph.ipynb notebooks are the real documentation—there is no API reference beyond the README examples

Verdict

Anyone who has used PyTorch’s .backward() but never traced the gradient flow by hand should spend an hour here. If you need to train models on actual data, keep walking—this is a dissection kit, not a production framework.