Navigation instructions for robots that can't cheat with nav-meshes

What it does VLN-CE is a benchmark and training ground for vision-and-language navigation in continuous space. It ports the standard R2R and the larger multilingual RxR datasets into the Habitat simulator, replacing discrete graph-node movement with continuous navigation through Matterport3D scans. The repo includes baseline agents, training methods, and starter code for official challenge submissions.

The interesting bit The shift from discrete to continuous navigation turns high-level instructions like “turn left at the sofa” into low-level motor control problems. The RxR dataset raises the bar further with multilingual crowdsourced directions—English, Hindi, and Telugu—and deliberately non-shortest paths to prevent agents from gaming the task.

Key highlights

• Official implementation of the ECCV 2020 “Beyond the Nav-Graph” paper and its waypoint-model successor
• Supports both R2R and RxR, including preprocessed dataset versions and pre-trained Cross-Modal Attention baseline weights
• Includes challenge submission starter code for both VLN-CE (R2R) and RxR-Habitat leaderboards
• Runs locally in Habitat-Sim with Matterport3D reconstructions

Caveats

• Locked to Habitat-Sim 0.1.7 and Python 3.6, so modern environments may need dependency triage
• Waypoint models use a panoramic observation space that is incompatible with the constrained action space required by the RxR-Habitat challenge

Verdict A solid starting point for embodied-AI researchers working on instruction-following navigation, but the dated dependency stack makes it a poor fit for anyone looking for a modern, drop-in navigation library.