Consistent depth for arbitrarily long videos, no diffusion required

What it does Video Depth Anything extends Depth Anything V2 to monocular video, estimating temporally consistent depth across arbitrarily long sequences without drifting. It ships both relative and metric depth checkpoints in three sizes—Small, Base, and Large—and includes an experimental streaming mode that processes one frame at a time by reusing cached hidden states from prior temporal attention layers.

The interesting bit The authors skip the current diffusion-model trend and stick to a transformer backbone, claiming faster inference, fewer parameters, and better consistent-depth accuracy. The experimental streaming mode is essentially a memory hack: it lets the model pretend it has seen the whole video while ingesting only a single new frame, though the README admits this trick currently costs accuracy.

Key highlights

• Three model scales (28.4 M to 381.8 M parameters) with both relative and metric depth checkpoints.
• Benchmarked against MoGe-2-L, UniDepthV2-L, and DepthPro on KITTI, NYUv2, and ScanNet; the metric Large model posts a 1.09 TAE on ScanNet.
• FP16 inference on an A100 takes 7.5 ms for the Small model and 14 ms for the Large, using 6.8 GB and 23.6 GB of VRAM respectively.
• A training-free streaming mode caches temporal-attention hidden states to support single-frame inference, though the authors note a measurable accuracy drop (e.g., ScanNet δ1 falling from 0.926 to 0.836).
• Licensing is split: the Small model is Apache-2.0, while Base and Large are CC-BY-NC-4.0.

Caveats

• The streaming mode is explicitly experimental and suffers a performance gap because it was not trained for incremental inference; the authors suggest finetuning would help but have not released that.
• Only the Small model is fully open-source (Apache-2.0); the larger checkpoints carry a non-commercial license.

Verdict Worth a look if you need fast, consistent depth from long video clips and can live with transformer-level rather than diffusion-level accuracy. Skip it if you require a fully open commercial license for large models or production-grade streaming inference today.