Teaching video models to know where things are, not just what they are

What it does Cambrian-S is a set of video understanding models (0.5B to 7B parameters) built on Qwen2.5 and SigLIP2, plus a curated instruction-tuning dataset called VSI-590K. The project also ships VSI-SUPER, a benchmark designed to test whether models actually understand spatial relationships in video, or are just exploiting textual shortcuts.

The interesting bit The authors noticed that standard spatial benchmarks are leaky—models can score well without looking at the video. Their fix is “predictive sensing,” a training objective where the model learns to predict latent future frames, forcing it to build genuine spatial representations. One variant, Cambrian-S-7B-LFP, is trained explicitly with this objective.

Key highlights

• Four model sizes (0.5B–7B) plus an LFP variant; all weights and training code are open
• VSI-590K: 590K video QA pairs focused on spatial reasoning, with task-type and question-type breakdowns visible in the repo
• VSI-SUPER benchmark plus evaluation code in lmms-eval/
• Competitive on general video benchmarks (Perception Test, EgoSchema) while outperforming prior work on spatial tasks
• Training pipeline is four-staged, from vision-language alignment through spatial video tuning; scripts provided for each stage
• TPU-first training stack via TorchXLA

Caveats

• Training is TPU-oriented (TorchXLA); GPU support is not mentioned in the README
• The “predictive sensing” mechanism is described at a high level; architectural details require reading the paper
• Dataset construction diagram suggests heavy curation, but exact filtering logic isn’t specified in the README

Verdict Worth a look if you’re building spatially-aware video agents, or if you’re suspicious that your current MLLM is just reading captions. Skip if you need a drop-in GPU training recipe today.