What it does

VirtualHome is a dual-mode simulator for household activities. You write activities as plain sequences of instructions—“walk to kitchen, open fridge, grab apple”—and the system either runs a fast Python-only graph evolution or renders the whole thing in Unity with physics, lighting, and day/night cycles. It streams ground-truth data like segmentation masks, depth, and optical flow alongside the video, and supports multiple agents acting in the same scene.

The interesting bit

The split architecture is the clever part. The Evolving Graph simulator runs headless in Python for fast iteration and dataset generation, while the Unity Simulator handles photorealistic rendering when you need visual input for your models. Same program, two execution paths—swap them based on whether you’re training a policy or debugging why your robot keeps walking into walls.

Key highlights

• Procedurally generates environments for infinite training variation (v2.3)
• Exposes OpenAI Gym-style RL interface with Ray integration for parallel envs
• Multi-agent support with synchronized time management and outdoor terrain
• Ships with pre-collected household activity datasets and augmentation scripts
• Docker and headless X-server support for remote/cloud training

Caveats

• Some objects and actions in the Python graph simulator don’t yet exist in the Unity renderer; the README notes this mismatch explicitly
• Requires downloading a separate platform-specific Unity executable; not pip-only
• Photorealism and human interaction are listed as “currently in development”

Verdict

Grab this if you’re doing embodied AI, multi-agent RL, or need a controllable domestic environment with rich sensor output. Skip it if you want a drop-in Atari replacement or need guaranteed action parity between fast and rendered modes.