Tianshou v2 splits algorithms from policies and breaks backward compatibility

What it does

Tianshou is a PyTorch reinforcement learning framework built on Gymnasium that bundles dozens of algorithms—from DQN variants and PPO to offline methods like CQL and BCQ—into a single training stack. It offers a high-level API for training agents out of the box and a lower-level procedural API for researchers who prefer to dissect and reassemble algorithm internals.

The interesting bit

Version 2 is a ground-up rewrite that draws hard lines between Algorithm and Policy abstractions, and it uses the type hierarchy to separate on-policy, off-policy, and offline methods at the class level. In a field where RL frameworks often collapse these distinctions into tangled inheritance, that kind of taxonomy-first design feels almost botanical.

Key highlights

• Targets state-of-the-art MuJoCo performance for REINFORCE, A2C, TRPO, PPO, DDPG, TD3, and SAC.
• Supports vectorized environments (including EnvPool integration), multi-GPU training, and RNN-style actors for partially observable tasks.
• Accepts arbitrary state and action types—dicts, custom classes—without flattening everything into plain tensors.
• Experimental support for multi-agent RL and model-based RL.
• Extensive documentation, type checking, and test coverage, though the PyPI release lags significantly behind the master branch.

Caveats

• Version 2 is explicitly not backwards compatible with earlier Tianshou releases.
• Multi-agent and model-based RL support is flagged as experimental.
• The PyPI build is described as “far behind” master, so source installs are effectively required for current work.

Verdict

RL researchers who want a hackable, type-safe PyTorch base for prototyping will appreciate the low-level API. If you need a stable, drop-in trainer and prefer installing from PyPI without migration headaches, the v2 breaking changes and release lag may frustrate you.