SAM 2 treats images as single-frame videos

What it does

SAM 2 is Meta’s sequel to Segment Anything, turning promptable segmentation from a static-image tool into a video tracker. You nudge it with a click or bounding box, and it either masks the object in a single frame or propagates that mask across an entire clip. The repo ships inference APIs, pretrained checkpoints ranging from 39 million to 224 million parameters, and—since late 2024—the full training code, a web demo, and Hugging Face loaders.

The interesting bit

The model handles both domains with the same transformer backbone by processing a still image exactly like a one-frame video, using streaming memory to keep context as it processes footage in real time. A recent refactor of SAM2VideoPredictor also decouples per-object inference, so you can add new targets to an ongoing track instead of locking in every object at the start.

Key highlights

• SAM2ImagePredictor and SAM2VideoPredictor provide unified APIs for stills and clips, including automatic mask generation.
• Four checkpoint sizes offer a direct speed-accuracy trade-off, with the smallest running at over 90 FPS and the largest at roughly 40 FPS on published benchmarks.
• The December 2024 update brings full-model torch.compile support for faster video object segmentation and relaxes multi-object tracking constraints.
• Training scripts, web demo source code, and Hugging Face from_pretrained integration are all included.

Caveats

• Installation compiles a custom CUDA kernel, and the README warns that if nvcc is missing or the build fails, some post-processing features will be limited.
• Native Windows installation is effectively unsupported; the maintainers strongly recommend WSL.

Verdict

Ideal for researchers and product engineers building interactive video annotation, editing, or tracking pipelines. If your workload is strictly static images and the original SAM already suffices, the video capabilities are the main reason to migrate.