The paper that moved diffusion out of pixel space

What it does

This repository contains the reference implementation for Latent Diffusion Models (LDMs), which synthesize high-resolution images by applying the diffusion process in the latent space of a pretrained autoencoder rather than in raw pixel space. It ships with inference and training code for several tasks, including text-to-image generation, image inpainting, class-conditional synthesis on ImageNet, and unconditional generation. A retrieval-augmented variant (RDM) can also condition generation on nearest-neighbor CLIP embeddings drawn from external image databases.

The interesting bit

The core insight is decoupling perceptual compression from generative learning: an autoencoder handles the heavy lifting of compressing images, while the diffusion model only has to denoise a much smaller latent representation. The retrieval-augmented extension is a notable curiosity—it lets you steer generation not just with text prompts but with actual retrieved image neighborhoods from datasets like OpenImages or ArtBench, all through the shared CLIP embedding space.

Key highlights

• Pretrained 1.45B text-to-image model trained on LAION-400M
• Retrieval-augmented diffusion supporting text-only or text-plus-retrieved-image conditioning via CLIP embeddings
• Inpainting, class-conditional ImageNet generation (FID 3.6 with classifier-free guidance), and unconditional sampling
• Convolutional inference possible beyond the native 256² resolution, though the README notes controllability suffers
• Includes training configurations for autoencoders and latent diffusion models

Caveats

• Several capabilities are marked “Coming Soon,” including image-to-image retrieval and higher-resolution models
• The README is largely a quickstart and weight-download guide; architectural specifics are left to the arXiv paper
• Training requires substantial manual data preparation and disk space, particularly for ImageNet and LSUN

Verdict

Grab this if you need the original research codebase for latent diffusion, want to train custom autoencoders, or experiment with retrieval-augmented conditioning. If you are looking for a polished, packaged API, this is the upstream science project—expect rough edges.