Diffusion models can find matching pixels without ever trying

What it does DIFT treats the intermediate feature maps of a frozen diffusion U-Net as dense visual descriptors. Feed it two images, and it can tell you which pixel in the second image corresponds to a pixel you clicked in the first, even if the objects differ in pose, category, or style. The repo includes evaluation pipelines for semantic correspondence, homography estimation, and video object segmentation, plus a notebook that propagates edits across matched images.

The interesting bit The correspondence isn’t learned; it falls out of the diffusion model’s existing geometry. The authors simply read off features from specific upsampling blocks at specific timesteps—t=261 and up_ft_index=1 for semantic matching in Stable Diffusion, for example—and find that these activations already encode meaningful spatial relationships. It is a neat reminder that generative models spend a lot of effort building internal scene understanding they never explicitly reveal to the user.

Key highlights

• Zero-shot semantic correspondence using Stable Diffusion or OpenAI’s ADM features.
• Interactive demo: click a point on a source image and watch the model predict the match on a target image with a heatmap.
• Benchmarked on SPair-71k, HPatches, and DAVIS 2017 without task-specific training.
• Edit propagation notebook shows how to copy edits from one image to semantically related others.
• Exposed hyperparameters (t, up_ft_index, ensemble_size) let you trade resolution and memory for accuracy.

Caveats

• Benchmark evaluation requires manual dataset downloads and multiple external repositories.
• The README warns that default image sizes and ensemble sizes can run into memory issues on modest GPUs.
• Optimal feature extraction settings differ by model and task, so some tuning is expected.

Verdict A compelling resource for CV researchers and diffusion hackers who want to probe what U-Nets already know about scene geometry. Skip it if you need a polished, turnkey correspondence API.