Meta’s 3D Robot Simulator Outran Its Own Maintainers

What it does

Habitat-Sim is a C++ 3D simulator built for embodied AI research. It loads scanned indoor environments and CAD scenes, drops in URDF robots like Fetch or Franka, and runs rigid-body physics through Bullet while streaming RGB-D sensor data. The stack is meant to be paired with Habitat-Lab for training and benchmarking navigation and manipulation agents.

The interesting bit

The project’s stated philosophy is speed over feature bloat: it reportedly hits several thousand frames per second single-threaded and crosses 10,000 FPS multi-process on one GPU when rendering Matterport3D scenes. For embodied AI, that velocity is the point—slow simulators leave GPUs idle.

Key highlights

• Native support for major datasets including HM3D, MatterPort3D, Replica, and Gibson.
• Simulates URDF robots—mobile manipulators, fixed-base arms, quadrupeds—with Bullet physics.
• Configurable RGB-D and egomotion sensors for agent perception.
• Achieves over 8,000 steps per second for a Fetch robot in a ReplicaCAD apartment scene.
• MIT licensed and designed to integrate with the higher-level Habitat-Lab framework.

Caveats

• Meta officially ended active development and maintenance after v0.3.4; future fixes depend on community forks.
• Headless GPU rendering relies on EGL, which the README explicitly notes does not work on macOS.

Verdict

Worth a look if you need a fast, open simulator for navigation or manipulation research and can tolerate maintaining your own fork. Avoid it if you require an actively supported, general-purpose engine or native macOS headless workflows.