Keyframe Propagation Keeps Diffusion Video Coherent

What it does Rerender A Video takes an input video and a text prompt, then translates the footage to match the prompt’s style using Stable Diffusion. It generates stylized keyframes with an adapted diffusion model, then fills in the remaining frames using temporal-aware patch matching and blending. The goal is a full video translation that preserves global style and local textures across frames.

The interesting bit Instead of training a video-specific diffusion model, the project hijacks existing image generators by applying hierarchical cross-frame constraints during keyframe synthesis and borrowing classical video propagation techniques for the in-between frames. It also plugs directly into the existing SD ecosystem, supporting ControlNet for spatial guidance and LoRA for custom subjects.

Key highlights

• Zero-shot: no retraining or fine-tuning on video data is required.
• Compatible with off-the-shelf tools like ControlNet and LoRA.
• Enforces temporal consistency at both global style and local texture levels.
• Available as a Diffusers community pipeline and a local Gradio WebUI.
• Requires 24 GB of VRAM, though the README notes workarounds like xformers and dynamic resolution scaling.

Caveats

• The README’s troubleshooting section is extensive, covering ebsynth build failures, path restrictions (no non-ASCII characters), Gradio version conflicts, and out-of-memory errors.
• 24 GB VRAM is the stated requirement, which puts it firmly in high-end GPU territory.
• Some WebUI features require manually downloading specific community models.

Verdict Worth a look if you have the hardware and patience to tinker with diffusion-based video stylization. Skip it if you need a lightweight, plug-and-play video editor or lack a high-end NVIDIA GPU.