Dense SLAM that treats camera poses as 4×4 matrices, not vectors

What it does VGGT-SLAM takes a folder of RGB images (or a phone video converted to frames) and incrementally builds a dense 3D point cloud map while estimating camera trajectory. It wraps a pretrained feed-forward model—VGGT, a vision transformer for geometry—with back-end optimization that refines poses jointly. An optional open-set object detection mode lets you query the map with text like “coffee machine” and see 3D bounding boxes in the Viser viewer.

The interesting bit Most SLAM systems split pose estimation into rotation (SO(3)) and translation, then stitch them back together. VGGT-SLAM optimizes directly on the SL(4) manifold—4×4 homogeneous transformation matrices—using a custom GTSAM factor that was upstreamed into the official GTSAM repository. The authors claim this unified treatment avoids the usual representational headaches.

Key highlights

• Feed-forward dense reconstruction from RGB only; no depth sensor required
• SL(4) manifold optimization integrated into official GTSAM (August 2025)
• Optional open-set 3D object detection via SAM 3 and Perception Encoder
• Evaluated on TUM and 7-Scenes benchmarks with provided scripts
• VGGT-SLAM 1.0 accepted to NeurIPS 2025; 2.0 accepted to RSS 2026

Caveats

• Real-time camera streaming code is listed as TODO, though the authors note it has been tested on a Jetson Thor
• Visualization of large point clouds can crash Viser; the README suggests voxel downsampling as a workaround
• The setup script auto-downloads multiple third-party models with their own licenses (PE, SAM3, VGGT)

Verdict Worth a look if you’re building dense RGB-only mapping pipelines or curious about Lie-group alternatives to classic pose representations. Probably overkill if you already have reliable depth and just need odometry.