Turning stick figures into footage, assuming you own a server farm

What it does

vid2vid is a PyTorch implementation of a NeurIPS 2018 method that translates structured inputs—semantic label maps, face edge maps, or human pose skeletons—into photorealistic video at resolutions up to 2048×1024. It essentially chains image-to-image translation across time, using previously generated frames to condition the next so the output stays temporally coherent. The repo includes pre-trained models for street-view scenes, talking faces, and dancing bodies.

The interesting bit

The clever part is the sequential, multi-GPU training pipeline: frames are generated one after another in a relay across GPUs, with each new frame conditioned on its predecessors, then passed to discriminator GPUs for loss computation. The authors also use a coarse-to-fine strategy, progressively scaling resolution from 512×256 to 2048×1024 to keep training stable.

Key highlights

• Generates 2048×1024 video from semantic maps, edge maps, or pose data
• Extends the pix2pixHD/SPADE lineage into the temporal domain
• Requires multi-GPU setups for full-resolution training (8 GPUs with 24 GB VRAM for 2K)
• Includes single-GPU training scripts, though the authors note performance is “not guaranteed”
• Ships with pre-trained checkpoints for Cityscapes, FaceForensics, and pose datasets

Caveats

• PyTorch 0.4 dependency dates the codebase; expect archaeology when setting up modern environments
• Full-resolution training demands 8 GPUs with at least 24 GB memory each, and single-GPU scripts are explicitly marked as not fully tested
• Frame generation is strictly sequential (current frame depends on previous frames), so parallelism at inference is limited

Verdict

Researchers or VFX tinkerers who need high-resolution synthetic video from structured data should look here; hobbyists with a single consumer GPU should probably admire the demo GIFs and move on.