SAM meets DeAOT: video tracking via clicks, captions, and even audio

What it does

SAM-Track is a pipeline that marries Meta’s Segment Anything Model with DeAOT, a NeurIPS 2022 transformer-based tracker. SAM handles key-frame segmentation—automatically or via user clicks, brushstrokes, and text prompts—while DeAOT propagates those masks through time and manages long-term memory to avoid exploding RAM on long clips. The result is a system that can detect new objects mid-stream and keep their masks attached as they move.

The interesting bit

The project is essentially well-engineered glue code, but the authors keep adding sensory modalities: you can now point at a sound-making object via an audio-grounding feature, or prompt with class names like “panda” and spatial descriptions like “panda on the far left.” That turns a pure computer-vision pipeline into something closer to a multi-modal video query engine.

Key highlights

• Combines SAM for key-frame segmentation with DeAOT (NeurIPS 2022) for temporal tracking and mask propagation.
• Supports automatic discovery of new objects plus interactive refinement via clicks, brushstrokes, and text prompts.
• Includes an audio-grounding mode that can track the object producing sound in the soundtrack.
• Offers a WebUI and a Colab notebook for browser-based experimentation without local setup.
• Provides long-term memory controls (long_term_memory_gap, max_len_long_term) to cap memory use on longer videos.

Caveats

• The README frames the 1.0 WebUI as a “developer version” and explicitly invites bug reports, so rough edges are expected.
• It is a pipeline built atop two external repositories (SAM and aot-benchmark), so maintenance depends on upstream changes.

Verdict

Worth a look if you need interactive video segmentation for research, smart-city footage, or autonomous-driving datasets. Skip it if you want a single, self-contained model with published benchmarks and guaranteed production stability.