SAM 2: Segment Anything in Images and Videos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

Seminal August 2024

A soccer ball and a player's boot, each selected with one click, tracked through the clip.

TL;DR SAM 2 extends Segment Anything to video with a streaming memory that carries an object's mask from frame to frame, trained on a new 35.5M-mask video dataset collected with the model in the loop. It beats SAM-plus-tracker baselines with over 3x fewer interactions, and on images it is more accurate than SAM and 6x faster.

What they do

SAM 2 is one model for promptable segmentation of images and videos. Clicks, a box or a mask on any frame define an object; the model returns its mask on that frame, propagates it through the video as a “masklet”, and accepts corrective clicks on other frames. An image is a one-frame video, where the model behaves like SAM.

How it works

Streaming memory. A Hiera encoder embeds each frame once. Memory attention conditions the frame on a memory bank holding up to N recent frames, up to M prompted frames and object-pointer vectors from the decoder; a memory encoder writes each new prediction into the bank.
Decoder. SAM’s promptable mask decoder, plus a head that predicts whether the object is visible at all (for occlusions) and skip connections from high-resolution encoder features.
Data engine. Annotation time fell from 37.8 s/frame with SAM alone to 4.5 s with SAM 2 in the loop (8.4x). The resulting SA-V dataset has 50.9K videos, 642.6K masklets and 35.5M masks, 53x more masks than any existing video segmentation dataset.

Results

Interactive segmentation on 9 zero-shot video datasets: better accuracy than SAM+XMem++ and SAM+Cutie with over 3x fewer interactions.
First-frame prompts, 17 zero-shot video datasets: 64.7 J&F with one click versus 56.9 and 56.7 for the baselines.
Semi-supervised VOS (Hiera-B+, 43.8 FPS on an A100): 76.6 on MOSE val versus 71.7 for Cutie-base+, and 76.8 / 77.0 on SA-V val / test where no prior method exceeds 61.4 / 62.8.
Images, SAM’s 23 datasets, trained on SA-1B only: 58.9 mIoU at one click versus 58.1 for SAM, at 130.1 versus 21.7 FPS.

Limitations

It can fail across shot changes, lose or confuse objects in crowded scenes, after long occlusions or in long videos, and struggles with thin fast-moving details and look-alike neighbours. Objects are processed separately, with no communication between them. The data engine still relies on human annotators to verify masklets.

Videos

A mug and the hand holding it, segmented and tracked beside a campfire.

One microorganism in a microscope video, selected with a click and tracked.

Figures

From Figure 1: The task (prompts on one or more video frames), the model (image encoder, memory attention, memory ban... — From Figure 1: The task (prompts on one or more video frames), the model (image encoder, memory attention, memory bank, mask decoder) and the data engine that built the SA-V dataset.

From Figure 2: Clicks on frame 1 select the dog's tongue and SAM 2 propagates it. When the track is lost after frame ... — From Figure 2: Clicks on frame 1 select the dog's tongue and SAM 2 propagates it. When the track is lost after frame 2, a single click on frame 3 recovers it, thanks to the memory.

From Figure 3: The SAM 2 architecture. Each frame is encoded once, cross-attends to memories of the object from earli... — From Figure 3: The SAM 2 architecture. Each frame is encoded once, cross-attends to memories of the object from earlier frames, and is decoded into a mask; a memory encoder stores the prediction for future frames.

From Figure 5: Zero-shot accuracy averaged over 9 video datasets in the interactive offline setting, against the numb... — From Figure 5: Zero-shot accuracy averaged over 9 video datasets in the interactive offline setting, against the number of frames annotated with 3 clicks, for SAM 2, SAM+XMem++ and SAM+Cutie.

Abstract

We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing our main model, dataset, as well as code for model training and our demo.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong