Edit images by tampering with Stable Diffusion's cross-attention

What it does This Jupyter notebook implements an unofficial, modified port of Google’s Prompt-to-Prompt paper for Stable Diffusion. It lets you edit generated or existing images by manipulating the model’s internal cross-attention maps during inference—no manual masks, no retraining. You can swap subjects, inject styles, shift seasons, or dial individual prompt tokens up and down.

The interesting bit The README demonstrates that simply adding “without fog and without rocks” to a prompt fails to remove fog and rocks, while this approach reaches into the diffusion process and edits attention weights directly. It also includes a modified inverse DDIM pipeline to find latent vectors for existing photos, plus a finite-difference refinement so you can invert images at higher CFG scales without everything breaking.

Key highlights

• Edits by patching attention maps instead of prompting harder or masking pixels
• Supports target replacement, style injection, global scene edits, and per-token weight tweaks
• Includes image inversion with a modified inverse DDIM compatible with schedulers like K-LMS
• Can invert at higher classifier-free guidance values via adaptive finite-difference refinement
• Exposes two main functions: stablediffusion(...) and prompt_token(...)

Caveats

• Fragile dependency: the notebook targets diffusers==0.4.1 and injects code into the model, so newer library versions are likely to break compatibility
• Requires prompt fine-tuning to keep results consistent; the README notes that changing hair color can accidentally introduce a smile, which must be edited out by adjusting the smile token weight
• Unofficial implementation with modifications from the original paper methods

Verdict Grab it if you’re experimenting with controllable diffusion and can tolerate pinned legacy dependencies. Skip it if you need a maintained, production-ready editing pipeline.