Steer video diffusion by warping noise, not architecture

What it does

Go-with-the-Flow is a fine-tuning and inference method for video diffusion that lets you dictate how a scene moves. A cut-and-drag GUI lets you sketch a crude animation by moving pieces of a still image, and the diffusion model renders it into a finished video. It can also transplant motion from one existing video onto another.

The interesting bit

Rather than bolting on new layers or re-engineering the architecture, the trick is to replace the usual i.i.d. Gaussian noise with warped noise. Inference runs at the same speed as the base model because the pipeline itself is unchanged.

Key highlights

• CVPR 2025 Oral; built by Netflix Eyeline Studios with Stony Brook, UMD, and Stanford collaborators.
• Local GUI for no-GPU animation templating, plus text-to-video and image-to-video motion transfer modes.
• Community ports already exist for ComfyUI, AnimateDiff, and HunyuanVideo.
• Works with CogVideoX and requires no structural changes to the underlying diffusion network.

Caveats

• Depth-warping inference, 3D-to-video conversion, fine-tuning code, and a Replicate instance are still on the TODO list.
• The modified Apache 2.0 license demands that all paper authors be credited in any motion-picture, film, or credited production using the work.

Verdict

A strong choice for researchers and creative coders who want steerable video generation without retraining entire models. Less appealing if you need a fully productized pipeline or commercial-friendly licensing out of the box.