Segment Anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick

★ Must read April 2023

TL;DR SAM is a promptable segmentation model trained on SA-1B, 1.1B automatically generated masks on 11M images, and applied to new tasks zero-shot by prompting. From a single point it beats the strongest interactive segmenter, RITM, on 16 of 23 unseen datasets.

What they do

Segmentation has no web-scale data source, so the authors build three linked pieces: a promptable segmentation task (return a valid mask for any point, box, mask or text prompt, even an ambiguous one), a model for it (SAM), and a model-in-the-loop data engine that produced SA-1B, 1.1B masks on 11M images. New tasks are then solved zero-shot by prompting.

How it works

Model. An MAE-pretrained ViT-H encodes the image once. Prompts are embedded (CLIP for text) and a lightweight two-way-attention decoder turns them into masks in about 50 ms on a CPU in a browser.
Ambiguity. Each prompt yields 3 masks (whole, part, subpart) ranked by a predicted IoU; training backpropagates only the lowest loss among them.
Data engine. Assisted-manual (4.3M masks; time per mask fell from 34 to 14 s), semi-automatic (5.9M more), then fully automatic from a 32x32 point grid, keeping confident, stable masks. SA-1B holds only automatic masks, and SAM trains on them.

Results

In a 500-image sample, 94% of automatic masks have over 90% IoU with professionally corrected versions.
From one point, SAM beats RITM’s mIoU on 16 of 23 unseen datasets, by up to about 47 IoU, and on all 23 when an oracle picks among its 3 masks. Human raters score its masks 7-9 out of 10, consistently above RITM.
Zero-shot: BSDS500 edges at ODS .768 (HED, trained on BSDS500: .788); LVIS proposals at AR@1000 59.3 vs 63.0 for ViTDet-H; box-prompted instance segmentation at 46.5 / 44.7 mask AP on COCO / LVIS vs 51.0 / 46.6, though annotators rate SAM’s masks higher.

Limitations

SAM misses fine structures, sometimes hallucinates small disconnected components, and draws less crisp boundaries than “zoom-in” methods; dedicated interactive segmenters do better with many points. The full model is not real-time with the heavy encoder, text-to-mask is not robust, and it is unclear how to prompt for semantic or panoptic segmentation.

Figures

From Figure 1: Three interconnected components: a promptable segmentation task, a segmentation model (SAM) that power... — From Figure 1: Three interconnected components: a promptable segmentation task, a segmentation model (SAM) that powers data annotation and zero-shot transfer via prompt engineering, and a data engine for collecting SA-1B, a dataset of over 1 billion masks.

From Figure 4: SAM overview. A heavyweight image encoder outputs an image embedding that can be efficiently queried b... — From Figure 4: SAM overview. A heavyweight image encoder outputs an image embedding that can be efficiently queried by point, box, mask or text prompts to produce object masks at amortized real-time speed. For ambiguous prompts, SAM outputs multiple valid masks with confidence scores.

From Figure 2: An example image from SA-1B with overlaid masks, all annotated fully automatically by SAM.

From Figure 9: Point-to-mask mIoU over 23 datasets with a varying number of center-point prompts. SAM significantly o... — From Figure 9: Point-to-mask mIoU over 23 datasets with a varying number of center-point prompts. SAM significantly outperforms prior interactive segmenters with 1 point and is on par with more points; the cross marks the oracle pick among SAM's 3 masks. Low absolute mIoU at 1 point is the result of ambiguity.

Abstract

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong