Finally, a report card for generative video models

What it does

VBench is a comprehensive benchmark suite that breaks “video generation quality” into specific, measurable dimensions rather than reducing it to a single vague score. It supplies curated prompt suites, evaluation pipelines, and human-annotated preference data to automatically score models on technical fidelity, temporal consistency, and prompt adherence. The repository unifies three generations of the benchmark: the original VBench (CVPR 2024), VBench++ (adding image-to-video, long-form video, and trustworthiness metrics), and VBench-2.0 (evaluating intrinsic faithfulness like physics realism and commonsense reasoning).

The interesting bit

Instead of trusting vibes, the authors validated their automated metrics against human preference annotations to ensure the scores actually reflect human perception. The suite has also evolved from basic technical benchmarks toward evaluating whether generated videos obey physical laws and commonsense—precisely the frontier where next-generation models currently struggle.

Key highlights

• Decomposes video quality into hierarchical, fine-grained dimensions rather than a single aggregate score
• Human preference annotations verify that automated metrics align with actual perception
• Three suites in one repo: VBench (technical quality), VBench++ (I2V/long video/trustworthiness), and VBench-2.0 (intrinsic faithfulness)
• Includes prompt suites, sampled videos from various models, and HuggingFace leaderboards/arenas
• Peer-reviewed: CVPR 2024 Highlight and TPAMI 2025; VBench-2.0 is on arXiv

Caveats

• VBench++ modules use legacy path names (vbench2_beta_*) for backward compatibility, which can confuse navigation
• The distinction between the three versions and their respective code locations requires careful reading of the overview section

Verdict

Researchers and engineers building or comparing video generation models should treat this as essential infrastructure. Casual users who just want to generate clips without systematic evaluation can skip it.