Your image segmenter, now with temporal memory

What it does

DEVA is a decoupled video segmentation system: a task-specific image model—such as SAM, Grounding DINO, or your own detector—generates per-frame masks, while a separate, universal temporal module propagates and fuses those masks bidirectionally through time. The result is open-vocabulary, long-term video segmentation driven by text prompts or automatic detection, without retraining the video backbone for every new task.

The interesting bit

Instead of the usual end-to-end monolith, DEVA treats temporal consistency as a generic fusion problem. It stitches together segmentation hypotheses from different frames using a (semi-)online fusion process, so the image model can focus on what it sees while the propagation model handles where things went. This design is deliberately modular: swap in a new image segmenter and the video tracker keeps working.

Key highlights

• Open-vocabulary tracking out-of-the-box when paired with Grounded Segment Anything; text prompts like “capybara” or “person.hat.horse” just work.
• The temporal propagation model is trained once and generalizes across tasks; no fine-tuning is needed to adapt your own image model.
• Supports both text-prompted and fully automatic grid-based prompting modes, with variants for HQ or efficient SAM backbones.
• Semi-online inference fuses hypotheses across chunks; optional Gurobi solver for integer programming, with a PuLP fallback.

Caveats

• The authors note that on closed-set benchmarks, end-to-end approaches still likely win; this is a trade-off for open-world flexibility.
• Temporal propagation amplifies false positives, so a conservative detector threshold matters, and busy scenes with constant object turnover (e.g., driving footage) can bloat the memory bank.
• Text-prompted demos depend on a specific fork of Grounded-Segment-Anything, and the setup is tested on Ubuntu only.

Verdict

Worth a look if you have a favorite image segmentation model and want to see it track objects through long videos without building a custom video pipeline. Skip it if you are optimizing for a narrow, closed-domain benchmark where joint training is viable.