Editing videos by text without forgetting the geometry

What it does StableVideo performs text-driven consistency-aware diffusion video editing by leveraging Neural Layered Atlases to separate foreground and background into shared 2D representations. It applies ControlNet-conditioned diffusion—using canny and depth guidance—to these atlas layers rather than to raw frames directly. A Gradio interface lets you adjust foreground masks and generate the final .mp4, which lands in ./log.

The interesting bit Instead of generating each frame in isolation and hoping the edits align, the system manipulates persistent atlas layers that the underlying NLA representation maps back into video space. This effectively gives the diffusion model a memory of the scene geometry across the entire clip.

Key highlights

• VRAM footprint ranges from roughly 14 GB to 29 GB depending on mixed precision, CPU offloading (save_memory), and xFormers
• Built atop Text2LIVE and ControlNet, reusing their pretrained canny and depth checkpoints
• Supports custom videos only after external NLA atlas training; pre-computed examples (car-turn, blackswan, etc.) are provided
• Includes a Hugging Face demo, plus a CPU-only variant
• Foreground atlas masks can be edited interactively in the Gradio UI

Caveats

• The README warns of a Gradio bug where the editable foreground atlas preview can desynchronize from the display, requiring a full restart
• Raw video cannot be edited directly; you must supply or train Neural Layered Atlases first

Verdict A solid research artifact for anyone studying atlas-based video manipulation or hunting for temporally consistent diffusion edits. If you need a push-button, training-free video editor, this is not it.