An RL workbench treating tensor libraries as interchangeable batteries

What it does

MushroomRL is a Python reinforcement learning library that provides classical algorithms like Q-Learning and SARSA alongside deep RL methods such as DQN, SAC, and PPO. It abstracts experiment setup so you can use PyTorch or TensorFlow backends and environments ranging from Gymnasium to PyBullet, MuJoCo, or realistic simulators like Habitat and iGibson. You write an experiment script to wire components together; there is no single entry point that hides the plumbing.

The interesting bit

The library treats tensor computation and environment interfaces as modular dependencies rather than core assumptions, so you can point the same agent at PyTorch or TensorFlow, then swap Gymnasium for Habitat without touching the algorithm code.

Key highlights

• Supports both classical RL (Q-Learning, SARSA, FQI) and deep RL (DQN, DDPG, SAC, TD3, TRPO, PPO)
• Backend-agnostic design accepts PyTorch, TensorFlow, or other tensor libraries
• Environment wrappers span Gymnasium, PyBullet, MuJoCo, DeepMind Control Suite, Habitat, and iGibson
• Published in JMLR (2021), suggesting academic credibility
• Minimal base install; heavy dependencies like Atari or physics simulators are opt-in

Caveats

• The README is heavy on installation steps and light on architectural detail or performance claims
• Running experiments requires writing custom scripts; there is no high-level training API
• Habitat and iGibson integration involves significant manual YAML and dataset wrangling that the library does not abstract away

Verdict

Researchers who juggle multiple backends or need to benchmark agents across both simple Gymnasium tasks and realistic simulators will find MushroomRL a pragmatic foundation. If you want a fully packaged, one-command training framework, this is not it.