Hijacking diffusion to turn scribbles into photos

What it does

SDEdit takes a rough input—like a stroke painting or a photo with color edits—and injects enough noise to hide artifacts while preserving overall structure. It then initializes the reverse stochastic differential equation process from a downloaded generative model to denoise the result into a realistic image that follows your original layout. The repository provides pretrained checkpoints for CelebA-HQ, LSUN bedroom, and LSUN church outdoor, plus code for both stroke-based generation and masked editing.

The interesting bit

Instead of generating from pure noise, the method hijacks the reverse diffusion trajectory by starting from your own noisy, structured input. The README notes that this same intuition later powered large-scale pipelines like Stable Diffusion’s img2img, GLIDE, and distilled-SD.

Key highlights

• PyTorch implementation of the ICLR 2022 paper built on the DDIM codebase
• Automatically downloads pretrained SDE (VP) models for three datasets
• Generates multiple diverse photorealistic outputs from a single stroke painting
• Supports masked stroke editing on natural images using [image, mask] tensor pairs
• Editing aims to stay faithful to user strokes while avoiding undesired changes

Verdict

Diffusion researchers and anyone tracing the lineage of modern img2img pipelines should bookmark this; practitioners who need text-guided editing should skip ahead to the descendant models the authors graciously cite.