Point-and-click 3D segmentation for Gaussian splats

What it does SAGA takes a pre-trained 3D Gaussian Splatting scene and lets you pull out individual objects by clicking on them. It translates 2D point prompts from SAM into binary masks over the underlying 3D Gaussians, so you can isolate, cluster, or render objects without ever extracting a mesh. The project also experiments with open-vocabulary segmentation via CLIP features and unsupervised 3D clustering.

The interesting bit Rather than segmenting geometry directly, SAGA trains contrastive affinity features on the Gaussians themselves, using SAM’s 2D masks as supervision. This lets the 3D primitives collectively signal object membership across rendered views. A dearpygui interface lets you preview 2D similarity maps before committing to a full 3D segmentation.

Key highlights

• Interactive GUI and Jupyter notebook for point-prompt segmentation
• Open-vocabulary queries via CLIP features (currently notebook-only)
• 3D clustering with HDBSCAN across the whole scene or inside masked regions
• Renders segmented objects by masking Gaussians directly
• Extends the original 3D Gaussian Splatting codebase with minimal architectural changes

Caveats

• Open-vocabulary segmentation is only implemented in the notebook, not the GUI
• Scene-wide clustering relies on CPU-based HDBSCAN, which the authors warn can be slow
• Requires an existing pre-trained 3DGS model and pre-extracted SAM masks before interaction

Verdict Researchers exploring editable 3D scene decomposition within Gaussian representations will find this a useful starting point. If you need a turnkey, real-time pipeline or production-grade open-vocabulary tooling, the current split between GUI and notebook may frustrate.