Tracking a camera while training a NeRF on the fly

What it does NeRF-SLAM runs dense monocular SLAM by pairing a Droid-SLAM-style tracking frontend with an Instant-NGP radiance field backend. While one thread estimates camera poses and geometry from incoming frames, another optimizes a neural scene representation. You can swap the mapping backend to Sigma-Fusion with a single flag if you prefer probabilistic volumetric fusion instead.

The interesting bit The project is essentially ambitious glue code: it tries to keep Droid-SLAM’s frame-to-frame correlation volumes and Instant-NGP’s renderer synchronized in real time, which is intrinsically memory-hungry. The author is refreshingly candid that the bundled GTSAM dependency—intended for IMU and stereo factor graphs—turned out to be unnecessary and never worked because the network’s learned scale broke when poses were updated.

Key highlights

• Real-time dense mapping from a single RGB camera using parallel tracking and mapping.
• Swappable backends: choose between NeRF (--fusion='nerf') or Sigma-Fusion (--fusion='sigma').
• Built on reimplemented Droid-SLAM components and a forked Instant-NGP with custom changes.
• Supports multi-GPU execution and a live GUI for visualization.

Caveats

• The pipeline is extremely GPU-memory intensive; the README warns that out-of-memory errors may require parameter tuning or simply better hardware.
• GTSAM is a compile-time dependency that the author admits is not actually needed, and getting its Python wrapper to build requires manual source patches.
• The author has graduated and explicitly stated there is limited time for ongoing maintenance or further development.

Verdict A solid reference for researchers bridging neural rendering and robotics, but expect research-grade rough edges and a hardware appetite that borders on voracious. Production engineers should look elsewhere.