Teaching latent diffusion models where sleeves actually go

What it does StableVITON is the official implementation of a CVPR 2024 paper that performs virtual try-on—swapping clothes onto person images—using a latent diffusion backbone. It conditions generation on DensePose and agnostic masks to align garments with body geometry, synthesizing the dressed result in latent space rather than deforming the original garment pixels.

The interesting bit Instead of relying on geometric warping, the model learns semantic correspondence between clothing regions and body parts within the diffusion process. The authors build on Paint-by-Example but expand the U-Net’s first block to 13 input channels and fine-tune a VAE specifically on the VITON-HD dataset to keep fabric textures plausible.

Key highlights

• Full training and inference code released, including a two-stage pipeline with an optional ATV-loss fine-tuning stage
• Architectural tweaks to the latent backbone: modified U-Net input channels and a fashion-domain-fine-tuned VAE for sharper garment details
• Supports paired and unpaired try-on, plus a --repaint mode that preserves unmasked background regions while redressing the subject
• Usable on arbitrary images if you supply your own DensePose estimates and segmentation masks (the README suggests SAM)
• Non-commercial CC BY-NC-SA 4.0 license

Caveats

• Not a turnkey web app: it requires structured preprocessing (DensePose, agnostic masks, and specific dataset layout) even for inference
• Depends on modified third-party checkpoints and a custom-fine-tuned VAE hosted on SharePoint, so reproduction is tied to external file availability
• Explicitly non-commercial; building a product around it is off-limits

Verdict A strong reference implementation for computer-vision researchers and fashion-tech engineers who need a reproducible diffusion-based try-on baseline. Casual developers looking for a one-click API or commercial foundation should probably skip it.