A thread pool that plays Atari 1M times per second

What it does

EnvPool is a C++ execution engine wrapped in Python via pybind11 that runs batches of reinforcement-learning environments in parallel. It speaks both Gymnasium and DeepMind dm_env APIs, supports sync and async execution, and covers a long list of domains: Atari, MuJoCo, Procgen, DM Control Suite, ViZDoom, Box2D, and roughly a dozen others. The pitch is simple: drop it in as a faster replacement for gym.vector_env without changing your training code.

The interesting bit

The speedups are large enough to be slightly absurd. On a DGX-A100 with 256 CPU cores, EnvPool hits ~1M Atari frames/sec and ~3M MuJoCo steps/sec—roughly 15–20× over Python subprocess baselines. Even on a 12-core laptop it claims ~3×. The trick is a lock-free queue and NUMA-aware scheduling, not GPU simulation, so it works for environments that haven’t been (and maybe can’t be) ported to CUDA.

Key highlights

• Ships prebuilt wheels for Linux, macOS, and Windows; Python 3.11–3.14
• Single-env mode still gets a claimed ~2× speedup over standard Python execution
• Built-in batched rendering (rgb_array and human modes)
• XLA interface for JAX jit compatibility
• C++ API for adding custom environments without forking the project
• Already integrated with Stable-Baselines3, Tianshou, CleanRL, ACME, and rl_games

Caveats

• Procgen on Linux requires a system Qt5 install; wheels intentionally don’t vendor it
• Windows MuJoCo rendering relies on Mesa software OpenGL in CI; local reproduction needs manual DLL setup
• Benchmark tables mix historical (PongNoFrameskip-v4) and current (ALE/Pong-v5) env versions, so read the labels before comparing numbers

Verdict

Worth a look if you’re burning CPU cycles waiting on Python’s GIL while training RL agents. Less relevant if you’ve already gone all-in on GPU-native simulators like Brax or Isaac Gym, or if you only ever run single-env loops on a laptop.