SAM on a diet: 40× faster and still pocket-sized

What it does

EdgeSAM is a shrunk-down, hardware-friendly rebuild of SAM. It swaps the original Vision Transformer encoder for a pure CNN, then distills the full pipeline—prompt encoder, mask decoder, and all—using actual box and point prompts rather than just raw image features. The result is a model that reportedly runs 40× faster than the original SAM and 14× faster than MobileSAM on edge hardware, while nudging COCO mIoU up by 2.3 points over MobileSAM.

The interesting bit

The authors found that distilling only the image encoder loses too much of SAM’s magic; the prompt-to-mask relationship gets mangled. By keeping prompts “in the loop” during distillation, the student model learns how user input actually affects segmentation, not just what the image looks like. It’s a reminder that the boring part—how you supervise the student—matters more than the architecture swap alone.

Key highlights

• Runs at 38.7 FPS on an iPhone 14 (vs. MobileSAM’s 4.9 FPS), with roughly half the GFlops (22.1 vs. 38.2)
• Exports cleanly to CoreML and ONNX; the authors ship pre-converted weights and an iOS app called CutCha
• Drop-in API compatibility with SAM: SamPredictor and sam_model_registry work the same way, just with an extra mask candidate option (1, 3, or 4 masks per prompt)
• Training and eval code released as of July 2024
• Already integrated into labeling tools (X-AnyLabeling, ISAT) and the Grounded-Segment-Anything ecosystem

Caveats

• CoreML exports omit pre- and post-processing (resize, pad, norm) because CoreML lacks dynamic-size interpolation, so you’ll need to handle that yourself
• IoU prediction scores are considered unreliable; the authors recommend using the stability score for mask selection instead
• The CoreML export script currently needs a workaround (commenting out an assertion in coremltools) as of version 7.1

Verdict Worth a look if you need interactive segmentation on mobile hardware or in browser demos without a GPU backend. If you’re doing server-side batch processing on beefy GPUs, the original SAM or heavier distillations are probably simpler.