Rebuilding Stable Diffusion from scratch in PyTorch

What it does

This repository contains a from-scratch PyTorch implementation of Stable Diffusion v1.5, organized as Jupyter notebooks. Instead of calling a one-line generate() helper, you load the original CompVis weights manually and step through the U-Net, VAE, and CLIP text encoder yourself. It effectively turns the familiar image generator into an interactive anatomy lesson.

The interesting bit

The project deliberately avoids wrapping logic in opaque abstractions; the README even points you to HuggingFace to download the raw vocab.json, merges.txt, and checkpoint files. That friction is the feature—it forces you to confront the actual tensor shapes and pipeline stages that most frameworks hide.

Key highlights

• Implements the full v1.5 latent diffusion pipeline in pure PyTorch
• Loads original CompVis .ckpt weights and compatible fine-tuned models (e.g., InkPunk, Illustration Diffusion)
• Notebook format makes each stage—tokenization, conditioning, denoising, decoding—inspectable
• No hidden helpers: you wire the tokenizer, encoder, and U-Net together explicitly

Caveats

• Requires manually downloading several gigabytes of weights and tokenizer files into a local data folder before anything runs
• README is minimal; expect to read the notebooks themselves rather than a guided tour
• Jupyter Notebook format means this is an interactive learning environment, not a packaged library or CLI tool

Verdict

Grab this if you are a student or researcher who wants to understand latent diffusion mechanics by touching the wires. Skip it if you just need an API or a polished GUI to generate images.