Make-A-Video in PyTorch: image-first training wheels for video U-Nets

What it does This repo implements the core building blocks of Meta AI’s Make-A-Video paper in PyTorch. It provides PseudoConv3d, SpatioTemporalAttention, and a SpaceTimeUnet that can be trained on still images and then promoted to video generation without architectural surgery. The temporal layers initialize as identity functions, so image pretraining Just Works™.

The interesting bit The insight is almost insultingly simple: instead of building a native 3D model from scratch, bolt temporal attention and pseudo-3D convolutions onto a text-to-image backbone like DALL-E 2. The README notes this “skimp on compute cost” approach has even shown up in protein contact prediction, which is a nice reminder that video and biology occasionally share math.

Key highlights

• SpaceTimeUnet accepts both 2D images and 3D video tensors, automatically skipping temporal logic when there are no frames to attend to.
• Temporal modules initialize as identity, so you can pretrain on images and fine-tune on video without rewriting your pipeline.
• Includes Flash Attention support and recent positional-embedding tricks.
• Explicitly designed to slot into existing diffusion U-Nets rather than stand alone.

Caveats

• The repo is marked wip, and integration with dalle2-pytorch for end-to-end training is still on the todo list.
• You get the Lego bricks, not the finished castle: no training loop, dataset loader, or pretrained weights are provided.

Verdict Grab this if you are building a custom diffusion video model and want a battle-tested spatial-to-temporal upgrade path. Skip it if you need a turnkey text-to-video API; this is library code, not a product.