A field guide to world models that live in actual 3D, not just pixels

What it does

This is the companion repo for “3D and 4D World Modeling: A Survey” (arXiv 2509.07996, tagged TPAMI 2026). It’s an awesome-list with a thesis: world modeling has mostly been done on 2D pixels, and the survey argues the interesting frontier is native 3D/4D signals — RGB-D, occupancy grids, LiDAR point clouds, neural fields — where geometry and motion are encoded in the coordinates agents actually act in.

The interesting bit

The taxonomy is the value. Everything is sorted into three representation paradigms — VideoGen, OccGen, LiDARGen — and then subdivided by function (data engines, action interpreters, neural simulators, forecasters, autoregressive simulators). If you’ve lost track of which “world model” paper does what, this is the map.

Key highlights

• Unifies terminology, scope, and evaluation across 3D/4D world modeling research
• Covers benchmarks (WorldLens, VBench, WorldScore, OccStress) and a long chronological dataset list
• Tracks workshops across CVPR, ICLR, NeurIPS, ICML, and ICCV through 2026
• Application sections span autonomous driving, robotics, games/XR, and digital twins
• Includes codebases, toolkits, tutorials, and talks as supplementary resources

Caveats

• It’s a link collection plus a paper pointer, not software — no code to run here
• The README leans on tables of arXiv entries; depth comes from reading the survey itself

Verdict

Essential bookmarking if you work on world models, embodied AI, or driving simulation, and a good on-ramp if you’re entering the field. Skip it if you wanted a toolkit — this is a bibliography with structure.