Procedural cities built to test your LLM agent's common sense

What it does SimWorld is a simulation platform built on Unreal Engine 5 for developing and evaluating LLM and VLM agents in complex environments. It provides a gym-like Python interface where agents receive multimodal observations—such as ego-view RGB images and spatial coordinates—and execute natural language actions through a local action planner. The base package includes lightweight city scenes and an empty map, with over 100 additional pre-built maps available for richer scenarios.

The interesting bit The environment layer supports language-driven scene editing and procedural city generation, so the world can be modified via natural language rather than manual level design. That turns a static game engine into a dynamic testbed where agents must handle both physical navigation and evolving social situations.

Key highlights

• Built on Unreal Engine 5 with realistic physics and traffic simulation
• Three usage tiers: base scenes, 100+ pre-built maps, or fully custom UE environments imported via .pak files
• Agent layer reasons over multimodal observations and history, feeding into a local action planner
• Gym-like Python API for task setup and agent control loops
• Accepted as a NeurIPS 2025 spotlight paper and CVPR 2025 demonstration

Caveats

• Requires a dedicated GPU with at least 6 GB VRAM, 32 GB RAM, and 50–200 GB of disk space
• Windows or Linux only; the hardware bar is non-trivial
• The quick-tour code in the README is explicitly simplified for demonstration, so real integration likely involves more boilerplate

Verdict Worth exploring if you need a realistic, open-ended benchmark for embodied AI that goes beyond grid worlds. Skip it if you were hoping for a lightweight, laptop-friendly simulator—this is a full Unreal Engine workload.