Vector rewards for the chronically indecisive agent

What it does

MO-Gymnasium is a collection of multi-objective reinforcement learning environments built on top of the standard Gymnasium API. Instead of returning a single scalar reward, each env.step() yields a numpy array of rewards, letting an agent optimize for multiple—often conflicting—objectives at once. The library includes classic MORL problems as well as multi-objective versions of existing domains like MuJoco, all version-locked for reproducibility.

The interesting bit

The clever part isn’t the environments themselves, but the standardization: by keeping the exact same Gymnasium interface and only changing the reward shape, existing tools mostly just work, and the ecosystem gets a shared benchmark rather than a dozen incompatible custom loops. A LinearReward wrapper lets you collapse the vector back into a scalar if you want to pretend your agent only cares about one thing.

Key highlights

• Extends the standard Gymnasium API; env.step() returns a vectorized numpy reward array instead of a float
• Bundles classic MORL literature environments plus multi-objective variants of MuJoco and other classical domains
• Includes reward wrappers like LinearReward for scalarization when you need a single objective
• Strict environment versioning (e.g., -v0) bumped on any change that could affect learning results
• Backed by the Farama Foundation and published at NeurIPS 2023

Caveats

• The README notes that installing all environment families at once ([all]) can be problematic on some systems, so expect dependency friction depending on your OS
• Beyond the API change and bundled envs, the library is intentionally narrow; you’ll likely want to pair it with something like MORL-Baselines for actual algorithms

Verdict

Worth a look if you’re researching or building multi-objective RL agents and need a common interface. Skip it if you’re happy with single-objective rewards and standard Gymnasium—this is a specialization, not an upgrade.