PyTorch baselines when one reward isn't enough

What it does

MORL-Baselines is a library of Multi-Objective Reinforcement Learning (MORL) algorithms implemented in PyTorch. It wraps a broad catalog of single- and multi-policy methods behind the MO-Gymnasium API, which differs from standard Gymnasium only by returning a vector of rewards instead of a scalar. The project emphasizes reproducibility, with automatic testing, pre-commit enforcement, and tracked experiments fed into Weights and Biases dashboards.

The interesting bit

The repository tries to keep each algorithm in a single file, making methods self-contained and easier to audit. It also explicitly distinguishes between algorithms optimized for the Scalarized Expected Return (SER) and Expected Scalarized Return (ESR) criteria, a subtle but important taxonomic split that many RL libraries gloss over.

Key highlights

• Implements over a dozen algorithms spanning discrete and continuous observation and action spaces.
• Strictly follows the MO-Gymnasium API, so environments only need to return a numpy array reward vector.
• Automatically reports results to Weights and Biases and participates in the Open RL Benchmark for cross-library comparison.
• Includes utility functions for Pareto pruning, experience buffers, and hyperparameter optimization.
• All algorithms are automatically tested and documented.

Caveats

• The README warns that some algorithms have limited features.
• PCN assumes environments with deterministic transitions.
• IPRO-2D is strictly a two-objective specialization and cannot handle higher-dimensional reward vectors.

Verdict

Researchers and practitioners working on problems with inherently conflicting objectives—like robotics or resource allocation—should find this a solid starting point. If your reward function is a single scalar, standard libraries like Stable Baselines 3 already have you covered.