First-person 3D vision for robots that follow vague instructions

What it does EmbodiedScan is a large-scale, ego-centric dataset and benchmark for 3D perception in embodied AI. It packages over 5,000 scans containing roughly one million first-person RGB-D views, one million language prompts, 160,000 oriented 3D bounding boxes across 760 categories, and dense semantic occupancy labels for 80 common categories. The repository also includes a baseline framework, Embodied Perceptron, that ingests an arbitrary number of multi-modal inputs—RGB-D sequences plus text—and tackles both fundamental 3D perception tasks and language-grounded challenges like visual grounding.

The interesting bit Traditional 3D datasets favor a tidy global viewpoint; EmbodiedScan forces models to parse messy first-person observations and align them with natural language instructions. The companion MMScan work piles on hierarchical grounded language annotations, so the benchmark effectively tests whether a model can grasp both the geometry of a room and the ambiguity of a human request.

Key highlights

• Over 5K scans, 1M ego-centric RGB-D frames, 1M language prompts, and 160K 3D boxes spanning 760 categories
• Baseline Embodied Perceptron fuses arbitrary multi-modal inputs via dense and sparse feature fusion
• Two benchmark tracks: fundamental 3D perception and language-grounded tasks such as multi-view 3D visual grounding
• Companion MMScan dataset adds hierarchical language annotations for finer-grained evaluation
• Work from CVPR 2024 and NeurIPS 2024

Caveats

• Data access requires filling out an application form; it is not an instant open download
• The README admits that installing heavy dependencies like PyTorch3D and Mink Engine can be error-prone
• Public benchmark splits differ slightly from the paper: the original validation set was retained as the test set, and the training set was re-split

Verdict A solid stop if you are building embodied agents that must reconcile first-person 3D sensing with natural language. Look elsewhere if you need a frictionless, immediately downloadable dataset for rapid prototyping.