One 2D prompt carves a 3D mask out of any NeRF

What it does

SA3D takes a pre-trained neural radiance field and lets you segment any object inside it with a single point or text prompt from one rendered view. It projects the initial 2D SAM mask into 3D mask grids using density-guided inverse rendering, then renders partial masks from other camera poses to generate cross-view self-prompts for SAM. The README claims the whole process finishes in roughly two minutes, yielding a complete 3D segmentation without redesigning the underlying radiance field.

The interesting bit

The clever part is that SA3D treats its own rendered masks as prompts: instead of asking a human to annotate dozens of views, it lets SAM refine itself across viewpoints until the 3D mask converges. The framework also adapts to different radiance field representations—NeRF, 3D Gaussian Splatting, and nerfstudio—without architectural redesign.

Key highlights

• One-shot prompting: a single point or text prompt in one view starts the process.
• Cross-view self-prompting: the system renders partial masks from novel views and feeds them back into SAM iteratively.
• Density-guided inverse rendering projects 2D masks into 3D grid structures aligned with the radiance field.
• Supports multiple backends, including MobileSAM and a 3D-GS variant that reportedly segments in seconds.
• Ships with a Dash-based GUI for interactive point-and-click or text-driven segmentation.

Caveats

• The README notes the GUI still needs refinement and that the two-pass stage currently has bugs.
• The nerfstudio branch supports text prompts only, so point-prompt fans are stuck on the mainline.
• It assumes you already have a trained NeRF; this is a post-hoc segmentation tool, not a scene reconstruction pipeline.

Verdict

Researchers and hobbyists sitting on pre-trained NeRFs who need quick object extraction without manual multi-view labeling should try this. If you are looking for an end-to-end reconstruction-and-segment pipeline, this is only half the equation.