Nvidia's recipe for 45 FPS long-video generation

What it does

LongLive 2.0 is Nvidia’s open infrastructure for generating long videos autoregressively — think minutes, not seconds — at speeds that approach real-time. It wraps training (teacher-forcing AR, then DMD distillation) and inference in a single stack, with sequence parallelism and NVFP4 quantization wired throughout. The project also preserves the original LongLive 1.0 branch, which handles interactive streaming generation via attention sinks and KV-recache.

The interesting bit

Most quantization stories stop at “we ran quantize() and it was faster.” LongLive actually trains through NVFP4 — both the autoregressive stage and the few-step DMD distillation — and keeps a BF16 fallback path intact. The inference engine then runs W4A4 with a quantized KV cache, fused Triton kernels for RoPE and adaLN, and pinned VAE transfers. A recent optimization pass squeezed another 18.6% throughput out of that pipeline. They also bothered with a safer “LoRA before quantization” setup, which suggests someone learned a hard lesson so you don’t have to.

Key highlights

• Claims 45.7 FPS for the 2-step NVFP4 5B model, 29.7 FPS for the 4-step variant; BF16 5B does 24.8 FPS
• Supports both T2V and I2V training, including multi-shot video sequences
• NVFP4 training and inference with TransformerEngine or FourOverSix backends
• Sequence-parallel inference plus async decoding and multi-shot attention sinks
• VBench scores hold within ~2 points across the quantization ladder (85.06 BF16 → 83.14 2-step NVFP4)
• Apache 2.0 license

Caveats

• The README notes a TODO to replace a placeholder video embed on the project page, so the demo polish is still incoming
• I2V training configs require careful latent-sequence bookkeeping (96, not 96 + 1) and masking the first latent from loss — easy to trip over
• NVFP4 setup demands attention to which checkpoint format you downloaded; the wrong model_quant_use_transformer_engine flag will quietly break things

Verdict

Worth a look if you’re shipping autoregressive video models and need to claw back GPU memory without training a separate tiny model. Skip it if your videos are short enough that standard diffusion sampling already finishes before your coffee does.