ControlNet with 90% fewer parameters and a LoRA habit

What it does

ControlNeXt wrangles Stable Diffusion XL, SD 1.5, and Stable Video Diffusion by injecting control signals—pose sequences for video, structural guidance for images—without ballooning the trainable parameter count. The repo bundles several backbone-specific variants, each with its own training scripts and inference setup.

The interesting bit

The authors claim up to 90% fewer trainable parameters than ControlNet, which allegedly translates to faster convergence. The architecture is also built to coexist with existing LoRA weights, so style adapters and control can share the same model without stepping on each other’s toes. For video, the SVD variant chases AnimateAnyone-style human motion, but with what the authors describe as a more concise architecture.

Key highlights

• Image control for SD 1.5 and SDXL, plus video generation via SVD and SVD-v2
• Explicitly designed to integrate with LoRA techniques for style control
• SVD-v2 upgrades include higher-resolution generation, larger inference batches, and pose alignment for human video
• Online Hugging Face demo available for the SDXL variant
• Training tips and battle scars documented in experiences.md and GitHub Issues

Caveats

• The project is actively iterating; the authors warn that code and models may change at any time
• The SD3 variant exists but cannot be released due to private training data and code
• No online demo for video variants because SVD inference is too resource-intensive to host

Verdict

Try it if you want ControlNet-style steering with a smaller training footprint and LoRA compatibility. Look elsewhere if you need a stable, frozen release—the README waves a polite “here be dragons” disclaimer.