Deep RL algorithms in 340 lines, no assembly required

What it does CleanRL implements seven deep reinforcement learning algorithms—PPO, DQN, C51, SAC, DDPG, TD3, and PPG—as standalone Python scripts. Each variant (e.g., ppo_atari.py, ppo_atari_lstm.py) is deliberately self-contained, duplicating code across files so nothing is hidden behind abstractions. It targets researchers who need to understand or tweak implementation details without spelunking through a modular framework.

The interesting bit The project openly admits it is not a library—don’t try to import it. This anti-pattern is the point: by sacrificing DRY principles, CleanRL makes every hyperparameter, preprocessing step, and environment interaction visible in one scroll. The authors even published a JMLR paper defending this heresy. There’s a certain honesty to code that refuses to be clever.

Key highlights

• ppo_atari.py clocks in at ~340 lines including all Atari-specific details
• Benchmarked across 34+ games with results published at benchmark.cleanrl.dev
• Supports both PyTorch and JAX implementations for several algorithms
• Integrates with Weights & Biases, TensorBoard, and AWS Batch for experiment scaling
• Includes variants for Procgen, PettingZoo multi-agent, IsaacGym, and LSTM/Transformer architectures

Caveats

• Migration to Gymnasium (from OpenAI Gym) is still in progress per issue #277
• The envpool speedups are Linux-only and may reduce sample efficiency
• Python version is pinned to <3.11, which is increasingly dated

Verdict Grab this if you’re learning RL, reproducing papers, or prototyping weird architectural changes. Skip it if you need a production-grade, composable RL framework—Stable-Baselines3 or RLlib will serve you better.