Offline RL algorithms, unzipped and benchmarked

What it does

CORL is a research library that implements roughly a dozen state-of-the-art offline and offline-to-online reinforcement learning algorithms—including CQL, IQL, ReBRAC, and Decision Transformer—as standalone, single-file scripts. It targets the D4RL benchmark suite and reports reproduced final and best scores across Gym-MuJoCo, Maze2d, and Antmaze tasks, with Weights & Biases integration for experiment tracking.

The interesting bit

The project borrows the “clean” philosophy from CleanRL and applies it to offline RL, where you cannot simply let an agent explore to debug your code. By isolating each algorithm to its own file, CORL makes the logic inspectable while still supporting large-scale hyperparameter sweeps.

Key highlights

• Each algorithm lives in a single file (e.g., offline/cql.py, finetune/awac.py)
• Covers both pure offline and offline-to-online fine-tuning variants where the underlying paper supports it
• Reproduced D4RL scores are reported for all algorithms, though the README notes that “final” and “best” metrics can differ significantly
• Built to scale from one-off reproductions to thousands of experiments
• Explicitly skips discrete control; the authors point those users to their sibling project, Katakomba

Caveats

• Benchmark reporting is inherently messy: the authors flag that reproduced final and best scores differ by a wide margin, and original papers don’t always clarify which they report
• Discrete control environments are out of scope

Verdict

Researchers who need to reproduce, compare, or tweak offline RL algorithms on D4RL will find this a rare combination of readability and rigor. If you want a drop-in black-box framework or are working on discrete action spaces, look elsewhere.