A battle-tested RL library that keeps the GPU busy

What it does

rl_games implements PPO and related algorithms with a focus on keeping the entire training pipeline on the GPU. It supports asymmetric actor-critic setups, masked actions, multi-agent training with centralized or decentralized critics, and self-play. The library wraps a wide range of environments—Isaac Gym, Brax, EnvPool, Atari, StarCraft II, DeepMind Control Suite—behind a unified config-driven runner.

The interesting bit

The project has real research mileage: it backs published work on dexterous manipulation (DeXtreme, DexPBT) and quadruped locomotion, and the authors actively maintain compatibility with the shifting ecosystem—most recently migrating from gym to gymnasium and restoring EnvPool support for NumPy 2.x and MuJoCo 3.x. That is the unglamorous work that keeps a library usable.

Key highlights

• End-to-end GPU training with Isaac Gym and Brax; observations and actions never leave the device
• Multi-GPU via torchrun, experiment tracking via Weights & Biases
• ONNX export for trained models, including LSTM variants
• Optional extras for atari, mujoco, envpool, brax, pufferlib
• Colab notebooks for MuJoCo, Brax, and ONNX export quickstarts

Caveats

• TensorFlow 1.x implementations (Rainbow DQN, A2C, PPO) were removed; only PyTorch remains
• Version 2.0.0 is unreleased, so the gymnasium migration is not yet in a stable tag

Verdict

Worth a look if you are doing GPU-heavy RL research in robotics or need a single framework that spans Isaac Gym to Atari. Probably overkill if you just want to train a CartPole baseline.