Pyramidal flow matching cuts video diffusion waste

What it does

Pyramid Flow is an autoregressive video generation model built on flow matching. It produces up to 10 seconds of 768p video at 24 FPS, and handles image-to-video generation, all trained exclusively on open-source datasets. The codebase includes inference scripts, a Gradio demo, training code for the VAE, and DiT finetuning utilities.

The interesting bit

Instead of running diffusion at full resolution from start to finish like most video models, it interpolates between latents of different resolutions and noise levels in a pyramid structure. This simultaneous generation and decompression avoids wasting compute on heavily noised full-resolution latents. The authors claim the whole thing was trained end-to-end with a single DiT in roughly 20.7k A100 hours.

Key highlights

• Two model backbones are available: a newer miniFLUX variant (which replaced the original SD3 architecture to fix human structure and motion stability) and the original SD3-based weights.
• Memory-frugal inference options exist: sequential CPU offloading can push requirements below 8 GB of VRAM, and standard CPU offloading runs under 12 GB, though both trade speed for memory.
• Multi-GPU sequence parallelism is supported for inference, cutting a 5-second 768p clip from 5.5 minutes on one A100 to 2.5 minutes across four.
• The project also supports Apple Silicon via an MPS backend and offers a Hugging Face demo, though the online version is capped at 25 frames due to GPU limits.

Caveats

• The codebase currently locks you to Python 3.8.10 and PyTorch 2.1.2, with wider version support described as “actively working.”
• bfloat16 is the only supported inference dtype; fp16 is not yet available.
• The 384p checkpoint is limited to 5-second generations, and CPU offloading noticeably sacrifices efficiency.

Verdict

Researchers and hackers looking for an open, training-efficient video generator with published training code should take a look. If you need a polished, one-click consumer tool or broad hardware compatibility today, this is still a research release.