What it does

CppRl implements A2C and PPO using PyTorch’s C++ frontend, with support for recurrent policies, continuous and discrete control. It ships with a ZeroMQ-based OpenAI Gym server so you can still test against standard environments. The author needed RL in a desktop app without asking users to install Python.

The interesting bit

This is essentially a faithful C++ translation of ikostrikov’s well-known PyTorch RL suite — same API, same algorithms, different ABI. The honesty is refreshing: the README calls it “very heavily based” and admits the Gym bridge involves “a few extra copies” that hurt performance on image observations.

Key highlights

• A2C and PPO implemented in libtorch, no Python runtime required
• Recurrent GRU policies and both continuous/discrete action spaces
• Cross-platform: tested on Windows 10, Ubuntu 16.04/18.04
• ~60 seconds to 200 reward on LunarLander-v2 (i7-8550U), per the author’s laptop
• CMake build, most deps included as submodules; libtorch is the only external install

Caveats

• Windows performance runs about 75% of Linux; the author is investigating
• The Gym server/client path is explicitly not well optimized for image observations
• You are trading the Python ecosystem for deployability — know which you need

Verdict

Grab this if you’re shipping RL inside a C++ application and can’t bundle Python. Skip it if you’re doing research in a Jupyter notebook; you’re losing the Python tooling for no gain.