When the function approximator is an optimizer

What it does

mpcrl is a Python library for model-based reinforcement learning that swaps the usual neural network policy for a Model Predictive Control solver. You formulate an MPC problem in CasADi via csnlp, then let standard RL algorithms—Q-learning and Deterministic Policy Gradient—tune the controller’s parameters from Gymnasium rollouts. The library handles the tedious part by automatically computing the MPC’s sensitivities with respect to its learnable parameters so the RL updates have gradients to follow.

The interesting bit

The MPC itself acts as the function approximator for the state and action value functions. Because the optimization problem enforces your dynamics and constraints at every step, the resulting policy stays interpretable and physically feasible even while it is learning.

Key highlights

• Learnable parameters are MPC internals such as dynamics matrices, cost weights, and constraint bounds.
• Gradient-based updates rely on sensitivities computed through csnlp and CasADi.
• Supports algorithms such as Q-learning and Deterministic Policy Gradient.
• Built around Gymnasium environments and requires Python >=3.11.
• Includes documented examples for classic linear time-invariant control problems.

Verdict

Grab it if you want adaptive control with RL but need to keep constraints and model structure in the loop. Skip it if you are looking for a conventional deep-RL stack for high-dimensional discrete tasks.