Honest glue: Segment Anything for Stable Diffusion

What it does

This extension for AUTOMATIC1111’s Stable Diffusion WebUI connects Meta’s Segment Anything (SAM) and GroundingDINO to the ControlNet pipeline. You can click an image to generate segmentation masks, type text prompts to get bounding boxes and masks via GroundingDINO, or run automatic segmentation. The resulting masks feed straight into ControlNet for inpainting, semantic segmentation, image matting, or generating LoRA/LyCORIS training sets. It is essentially integration plumbing between several existing heavyweights.

The interesting bit

The author is remarkably candid about the project’s nature, admitting the codebase is “ugly-enough” and that the extension has “almost moved into maintenance phase.” Rather than pretending to be a standalone research tool, it leans into being practical glue—solving the specific headache of getting SAM masks into ControlNet without exporting files between half a dozen tools. The README even ships a locally modified GroundingDINO fork to spare Windows users from C++ compilation hell.

Key highlights

• Click-to-mask generation inside the WebUI, plus text-driven mask creation via GroundingDINO.
• Direct handoff of SAM masks to ControlNet V1.1 for inpainting and semantic segmentation.
• Supports SAM variants including SAM-HQ and MobileSAM, with a CPU fallback for GPU-starved Macs.
• Full API coverage for single-image processing, plus batch layout generation and categorized masking.
• Bundles a local GroundingDINO workaround to bypass ninja, pycocotools, and _C build failures.

Caveats

• The author explicitly states the project is nearly in maintenance mode, with no huge updates planned unless new research justifies the effort.
• Extension list ordering conflicts with other WebUI extensions can break ControlNet integration; the offered fix involves manually reordering extensions rather than a code change.
• Some features like color inpainting and explicit mask editing are blocked by upstream Gradio and A1111 limitations, not by the extension itself.

Verdict

Worth a look if you live in AUTOMATIC1111’s WebUI and want SAM-powered masking without juggling external tools. Skip it if you need a standalone, actively evolving segmentation framework or if you aren’t already using ControlNet.