Virtual try-on with SDXL, densepose, and a laundry list of checkpoints

What it does IDM-VTON is the official PyTorch implementation of an ECCV 2024 paper that synthesizes virtual try-on images: given a person and a garment, it generates a new image showing the person wearing that garment. The pipeline leans on densepose estimates, human parsing masks, and OpenPose keypoints to align clothing with body shape. It ships with training and inference scripts for the VITON-HD and DressCode benchmarks, plus a local Gradio demo.

The interesting bit Rather than training a bespoke diffusion architecture from scratch, the project bolts an IP-Adapter onto SDXL to encode garment appearance, letting the base model handle the heavy generative lifting. The “in the wild” pitch is that it aims for authentic draping and texture preservation outside sterile studio datasets—though the README itself offers little detail on how that robustness is achieved.

Key highlights

• Official implementation of the ECCV 2024 paper Improving Diffusion Models for Authentic Virtual Try-on in the Wild.
• Built on SDXL and IP-Adapter; uses auxiliary models for densepose, human segmentation, and pose estimation.
• Supports standard benchmarks VITON-HD and DressCode with provided data-preparation layouts.
• Includes a local Gradio demo for interactive generation.
• Released under a CC BY-NC-SA 4.0 license, so it is strictly non-commercial.

Caveats

• The README is essentially a quickstart manual: it lists directory structures and launch commands but explains almost nothing about the model architecture or training logic.
• Getting started requires downloading several separate checkpoint sets—IP-Adapter, densepose, human parsing, and OpenPose models—in addition to the base weights.
• Non-commercial license (CC BY-NC-SA 4.0) means you cannot legally use the code or checkpoints for commercial products.

Verdict Worth a look if you are a researcher reproducing virtual-try-on benchmarks or experimenting with IP-Adapter conditioning for fashion generation. Skip it if you need a production-ready, commercially licensable tool or a self-contained package with minimal dependencies.