Meta’s SAM 3 segments nearly any concept you can describe

What it does SAM 3 is a unified foundation model for segmentation in images and video. You prompt it with text, points, boxes, or existing masks, and it detects, segments, and tracks the corresponding objects. It handles open-vocabulary concepts—phrases like “a player in white”—without task-specific retraining. The architecture pairs a shared vision encoder with a DETR-based detector and a tracker inherited from SAM 2.

The interesting bit A presence token in the architecture helps the model discriminate between closely related text prompts, while a decoupled detector–tracker design is meant to reduce task interference. Meta also says an automated data engine annotated over 4 million unique concepts to produce the training set, which they describe as the largest open-vocabulary segmentation dataset to date.

Key highlights

• Open-vocabulary segmentation across 270K concepts on the SA-CO benchmark, where the model reportedly hits 75–80% of human performance.
• Supports both image and video segmentation with text, geometry, or exemplar prompts.
• 848M parameters: a DETR detector and a SAM 2-style transformer tracker sharing a vision encoder.
• SAM 3.1 checkpoints on Hugging Face introduce a faster shared-memory multi-object tracking mode.
• Model weights are gated behind Hugging Face access requests; anonymous download is not possible.

Caveats

• You must request and be approved for access to download checkpoints from Hugging Face.
• It expects a CUDA-compatible GPU with CUDA 12.6 or higher, plus Python 3.12 and PyTorch 2.7.
• The repository is primarily inference and fine-tuning scaffolding around large pretrained weights; most of the novelty sits in the trained model and data engine, not the surrounding tooling.

Verdict Worth a look if you need open-vocabulary segmentation in video or want a drop-in replacement for SAM 2 with text prompting. Skip it if you are hoping for lightweight edge deployment or an ungated, fully open model.