Segment Anything: now with less waiting around

What it does segment-anything-fast is a fork of the original Segment Anything library tuned strictly for offline batched inference. It acts as a drop-in replacement: swap your import and the sam_model_fast_registry automatically flips on eval mode, bfloat16, torch.compile with max-autotune, and a custom Triton kernel for scaled dot-product attention with relative positional encodings. The README shows a waterfall of stacked optimizations—ending with NestedTensors, dynamic int8 quantization, and 2:4 sparsity—though it leaves the exact speedups to a bar chart rather than hard numbers.

The interesting bit The custom Triton kernel is the unusual part: it was written specifically for A100s, and if you’re on different hardware the library will attempt to autotune locally, which may or may not behave. That kind of hardware-specific optimization is rare in a drop-in library, and it suggests the authors are chasing every last FLOP rather than shipping generic defaults.

Key highlights

• Drop-in API compatibility with the original segment_anything package.
• Automatically applies torch.compile, bfloat16, and a custom Triton SDPA kernel.
• Stacks advanced techniques: NestedTensors, dynamic int8 symmetric quantization, and 2:4 sparse weights.
• Targets batched offline inference, not interactive or online use.
• Requires the latest PyTorch nightly builds.

Caveats

• The custom Triton kernel was built for A100 GPUs; other hardware may see degraded performance unless you disable it via SEGMENT_ANYTHING_FAST_USE_FLASH_4=0.
• Expect a slow first run while torch.compile generates kernels.
• It depends on PyTorch nightly, so stability is implicitly tied to bleeding-edge builds.

Verdict Worth a look if you’re already running Segment Anything at scale and need throughput without rewriting your pipeline. Skip it if you need real-time interactive segmentation or if you’re allergic to nightly PyTorch builds and compilation delays.