SAM 2: How an Interactive Demo Shaped Video Segmentation

Finding an outline is only part of the job

Counting visible protein reactions in wet-lab images presents two problems: isolate the individual items, then decide which ones matter. Joseph Nelson uses this example to explain Segment Anything Model (SAM). It produces masks—the image regions belonging to selected objects—while a person or another system identifies the relevant targets and supplies class labels.

The original SAM works on images; [SAM 2](https://github.com/facebookresearch/segment-anything-2) extends segmentation to video. Nelson describes the advance as near-zero-shot segmentation: producing useful outlines without training specifically for each application. Performance still varies across domains, and users may need to specify which object they want.

In the August 2024 Latent Space conversation, Nikhila Ravi describes arriving at AI research through engineering rather than a PhD. After studying engineering at Cambridge, she reconsidered a planned medical degree, spent a gap year coding and contracting, and received a scholarship to study at Harvard. Computer science classes at Harvard and MIT and computer vision projects led her to Facebook. By the interview, she had spent seven years at Facebook and Meta, working from 3D reconstruction toward understanding objects and pixels in 2D images.

Roboflow’s labeling workflow makes SAM’s value concrete: a click can replace the many polygon vertices needed to outline an object manually. Nelson reports approximately 49 million images labeled with SAM on Roboflow’s hosted platform over the preceding year, including five million in the preceding 30 days. He estimates roughly 35 years of time saved, explicitly as a back-of-the-envelope calculation assuming around a dozen seconds saved per annotation. The figures exclude self-hosted and offline usage.

Ravi identifies class-agnostic annotation as a deliberate design choice: training annotations describe boundaries without assigning category names. Learning boundaries around many small objects may transfer to structures such as cells without explicit cell labels. That flexibility does not eliminate domain adaptation—additional training for a particular application—especially where choosing the correct boundary requires specialist knowledge.

But target selection is a separate problem. A retailer may want a shirt mask while SAM selects the whole person. Visual prompting can specify the shirt without necessarily changing the model. Even a model capable of outlining every object perfectly would still need a signal identifying the region the application cares about.

Click, track, correct

The SAM 2 web demo carries visual prompting into video. In a football clip, the demonstrator selects the ball and tracks its mask. A shoe partially covering the ball causes an error, making the recovery workflow concrete: 1. Click the target in any video frame. 2. Press Track to propagate its mask through the video. 3. Inspect the error and add refinement clicks; negative clicks remove unwanted regions. 4. Press Track again to incorporate the correction.

Foreground and background effects show how those masks can feed video editing tools. An underwater octopus example tests tracking across moving tentacles, with background desaturation making the mask easier to inspect. A shuffling-cup example adds three visually identical cups and a ball hidden underneath one. The narrated result is that SAM 2 maintains the selected cup’s identity and finds it again at the end. These demonstrations illustrate capability without establishing quantitative accuracy.

Timeline swim lanes change thickness according to an object’s visibility. This exposes occlusion, when another object hides the target, and its later reappearance. The display helps annotators locate predictions worth inspecting or correcting. Recovery is part of the ordinary interaction.

The demo was also the annotation tool

The demo doubles as the annotation tool, giving interface work a direct role in model development. Faster, clearer interactions improve annotation efficiency and data quality, which can improve the model. The original SAM demo also helped people outside machine learning try their own images. That experience made the SAM 2 demo a priority from day one, with interactive video segmentation shaping both annotation and model design.

Real-time interaction brought efficiency decisions forward. The original SAM demo and Roboflow’s integration computed an image embedding—a model representation queried to produce masks—on a GPU, then queried it in the browser as the user moved the mouse. That split became impractical for video: the original embeddings occupied approximately 4 MB per frame, making an entire video’s embeddings burdensome to send to the frontend.

SAM 2 changed its image encoder from ViT-H to [Hiera](https://github.com/facebookresearch/hiera). Its largest model is described as approximately 224 million parameters, compared with roughly 630 million for the original SAM model discussed. The interview reports about a sixfold speedup over applying SAM independently to every video frame, though hardware, model variants and benchmark settings are unspecified. The efficiency improvements enabled the demo to run model inference entirely on the server.

The release-era demo focused on video; a notebook in the project repository demonstrated image segmentation with SAM 2. Browser and on-device execution were discussed as possible future community developments.

Adding names and choosing the right regions

Nelson’s Smart Poly demonstration returns to the gap between a boundary and its meaning. The Roboflow tool, then using SAM 1, initially proposes only the bass drum in an image of a drum set. Adding and subtracting regions expands the mask toward the intended whole set. The annotator then assigns a class label so downstream systems know what the region represents.

Grounding connects a requested concept to a location in an image. Nelson demonstrates this with Grounding DINO, using person and microphone as text prompts to obtain bounding boxes. A grounding model can supply target selection while SAM supplies segmentation. He describes existing combinations of Grounding DINO or Florence 2 with SAM 2.

AutoDistill accepts a directory of images and an ontology specifying the concepts of interest. In the shipping-container example, Florence 2 locates the container concept and SAM produces its mask, returning class labels with regions. Video can enter as individual frames; that frame-by-frame labeling workflow does not itself establish tracking between frames.

Ravi explains that native text prompting fell outside SAM 2’s deliberately narrow focus. SAM targeted class-agnostic image segmentation; SAM 2 targeted the corresponding video capability. The development recipe aligns data, model and demo around one substantial improvement. Community combinations are an intended benefit of releasing the work. Asked whether SAM 3 would add text prompting, she makes no commitment.

Website screenshots expose a practical limitation. In the demonstration, SAM 2 outlines people inside images and some screen text, while a web-navigation agent needs buttons, article headlines and individual image elements. A plausible segment can be the wrong unit for the application. The suggested remedy is to annotate domain data and fine-tune the released model, following the research team’s approach of providing general models and adaptation tools rather than building a custom solution for every use case.

A unified model that remembers corrections

The data engine, the workflow for creating training annotations with model assistance, evolved in three stages: 1. Apply SAM independently to video frames. 2. Feed SAM’s masks into a video segmentation model accepting a first-frame mask. 3. Replace the two-model pipeline with a unified SAM 2 model for image and video segmentation. Each stage reportedly improved annotation efficiency and quality. Unification made correction incremental: positive clicks add regions and negative clicks remove them. In the earlier pipeline, correcting a frame required deleting its prediction and annotating the object again, an expensive operation for complex shapes.

SAM combines an image encoder, prompt encoder and mask decoder: it uses a click prompt together with the image embedding to predict a mask. SAM 2 treats an image as a single-frame video and adds memory attention, a memory encoder and a memory bank. Those memory components are effectively unused for a single image. In video, they provide context about the target from other frames.

Memory draws on both prompted frames, containing input from a user or another model, and surrounding frames. It retains information in two forms: - Spatial memory captures high-resolution local detail. - Object-pointer memory captures longer-term, higher-level information about the tracked object. The team describes six-frame spatial memory combined with longer-term object pointers as a useful speed–accuracy trade-off, without giving a numerical accuracy result for that configuration.

Refinement clicks become memory. If tracking loses an object after an occlusion, a prompt on a later frame provides a recovery path. Prompted frames are described as always retained, keeping the correction available. Hands-on use of earlier video segmentation models exposed the need: once a prediction failed, users lacked a convenient way to intervene. SAM 2 makes feedback part of the prediction process.

Why not retain tens or hundreds of frames? The response argues that tracking an object probably needs less context than maintaining a long language-model conversation. Lighting, appearance and shape change, but short-term spatial detail plus longer-term object information had been sufficient in the team’s experience. This is a design judgment about the task, rather than a general limit on useful video context.

Tracking anything depends on what was learned

Crowded scenes containing similar-looking objects remain difficult. Refinement clicks offer a workaround, and better motion estimation is suggested as a possible improvement. The successful cup demonstration illustrates one case; it does not settle the broader identity-tracking problem.

Applications should also choose the output they actually need. Tracking a point or a bounding box may suffice when accurate boundaries are unnecessary. A full segmentation mask can exceed the needs of a simpler tracking workflow.

The discussion looks ahead to stronger zero-shot recognition and multimodal understanding, combining images with text or video with audio. Yet specialized knowledge remains a constraint. Rivian’s custom internal parts and their correct assembly, for example, may require manufacturer-specific knowledge absent from general training data. Dataset curation and validation on the intended application remain essential as general models improve.

The SAM 2 dataset under discussion contains approximately 51,000 videos. Ravi emphasizes coverage as well as scale: SAM had a billion masks across 11 million images without class labels, while video segmentation datasets covered relatively few categories. She cites YouTube VOS’s 94 object categories to explain why broader video annotations matter. Giving an existing tracker an arbitrary mask does not automatically give it the knowledge to follow arbitrary objects.

A reported comparison makes that distinction concrete: SAM 2 and baseline trackers receive the same initial T-shirt mask. Ravi describes SAM 2 following the shirt throughout the video while baselines may drift toward the whole person, attributing the difference to the baselines’ training on whole objects rather than arbitrary parts. The first mask specifies the target; training affects whether the tracker preserves it.

Video frames also differ from still images through motion blur, lower resolution and a different distribution of objects. Joining an image segmenter to an existing tracker therefore only goes so far. The project’s lesson is to design the model and training data together for arbitrary-object tracking, including the conditions found in video.

Beyond familiar benchmarks

RF-100 broadens evaluation toward unusual objects and settings, including thermal and aerial imagery, as a counterpoint to COCO’s common objects in context. The discussion also characterizes Object Detection in the Wild (ODinW) as revealing how quickly models can adapt to a domain. That separates two useful questions: what works immediately, and what becomes useful after additional training?

The team’s emphasis remains real-world, zero-shot capability beyond familiar benchmark categories. The released models, [SA-V dataset](https://ai.meta.com/datasets/segment-anything-video), SAM 2 paper, interactive demo and dataset visualizer provide starting points for experimentation. The closing invitation is to build on those resources, address unresolved limitations and share improvements that could inform future versions.