A robot simulator that finally takes the GPU seriously

What it does

ManiSkill is a robot simulation and training framework with a narrow focus on manipulation skills. It wraps the SAPIEN physics engine in a task-building API that handles the gnarly GPU memory management for you, then runs thousands of parallel environments—each potentially with completely different scenes and objects—to generate synthetic training data at throughput that would make CPU-bound simulators weep.

The interesting bit

The “heterogeneous simulation” angle is the quiet killer feature: every parallel environment can hold a different scene and object set, which means you’re not just batch-processing clones of the same world. That, plus real2sim environments for evaluating real-world policies 100× faster in GPU simulation, suggests the authors actually want this used for research that leaves the lab.

Key highlights

• GPU-parallelized visual data collection: RGBD + segmentation at 30,000+ FPS on a 4090 (per their own benchmark)
• Heterogeneous parallel environments—different scenes, different objects, same batch
• Pre-tuned baselines spanning RL (PPO, SAC, TD-MPC2), imitation learning (Diffusion Policy), and VLA models (Octo, RDT-1B, RT-x)
• Sim2real examples and real2sim evaluation environments included
• Object-oriented task API that abstracts GPU memory management
• Runs in Colab free tier if you want to kick the tires without hardware

Caveats

• GPU simulation and rendering are Linux/NVIDIA only; Windows, WSL, and macOS users get CPU simulation at best, and WSL can’t render at all
• Assets carry a CC BY-NC 4.0 license—fine for research, a headache for commercial use
• The framework is tightly coupled to SAPIEN, so its platform limitations are your platform limitations

Verdict

Researchers building visuomotor policies or anyone who needs manipulation data faster than their robot hardware can provide should grab this. If you’re on Windows or macOS and were hoping for GPU acceleration, you’re stuck waiting—or switching OSes.