RL from Scratch: 18 Ways to Teach an Agent Without a Library Safety Net

What it does This repository is an interactive educational collection of 18 reinforcement learning algorithms implemented from scratch in Jupyter notebooks. It covers the full arc from tabular methods like Q-Learning and SARSA through policy gradients such as REINFORCE and PPO, and on to actor-critic variants, multi-agent approaches, and planners like MCTS and PlaNet. Each notebook pairs code with explanations and visualizations—learning curves, Q-table heatmaps, and policy grids—using only basic tools like NumPy, Matplotlib, and PyTorch.

The interesting bit The author explicitly rejects the idea of a performance-optimized library, treating the project instead as an interactive textbook where readability mirrors mathematical descriptions. A bundled cheatsheet.md file condenses core concepts, pseudocode, and hyperparameters into a quick reference, making the repo useful both for linear study and sporadic lookup.

Key highlights

• 18 self-contained notebooks spanning value-based, policy-based, actor-critic, hierarchical, and multi-agent methods
• Heavy emphasis on clarity over speed; code is written to resemble the underlying math rather than production frameworks
• Includes visualizations such as Q-value heatmaps and policy grids to make abstract update rules concrete
• A separate cheatsheet.md provides algorithm summaries, formulas, and tuning tips for quick reference

Caveats

• Some notebooks—particularly complex ones like Hierarchical Actor-Critic—may contain bugs or incomplete implementations, as the author notes
• The A3C implementation requires running a standalone a3c_training.py script rather than the notebook due to multiprocessing limitations in Jupyter
• This is explicitly not a library; do not expect the speed or robustness of optimized RL frameworks

Verdict Worth bookmarking if you are learning RL and want to see how Bellman equations and policy gradients translate into actual loops and tensors. Skip it if you need battle-tested, production-ready agents.