Mask, prompt, blend: text-driven local edits with old-school diffusion

What it does

This is the official implementation of a CVPR 2022 paper that performs local, region-based edits on natural images. The system expects an input image, an ROI mask, and a text prompt; it leverages CLIP to steer a denoising diffusion probabilistic model toward the description while trying to preserve the unmasked background. It fuses the edited region by spatially blending progressively noised versions of the original image with the text-guided diffusion latent.

The interesting bit

Instead of simply pasting new pixels over a mask, the method mixes noise levels: the edited area is blended across a progression of noised latents so the result does not look like a crude composite. The authors also found that injecting augmentations into the diffusion process helps suppress adversarial artifacts that CLIP guidance can otherwise encourage.

Key highlights

• Expects a text prompt, an input image, and an ROI mask to define the editable region
• Generates multiple candidate results ranked by CLIP similarity; the README suggests generating at least 64 to find the best ones
• Supports applications like object replacement, background swapping, altering existing objects, and image extrapolation
• The authors have since released Blended Latent Diffusion, which they explicitly note produces better results with a significant speed-up
• Built on top of OpenAI’s CLIP and guided-diffusion codebases

Caveats

• The authors themselves recommend their follow-up project for better quality and speed
• Batch processing can trigger CUDA out-of-memory errors, requiring batch size reduction
• Requires a separate pretrained diffusion checkpoint that is not bundled in the repository

Verdict

Worth a look if you are researching the history of text-guided image editing or need the official CVPR 2022 reference implementation. Skip it if you want state-of-the-art results or fast inference, since the authors’ own Blended Latent Diffusion successor is explicitly superior.