SAM 3: Finding Every Instance—and Defining What Counts

From clicking players to naming a team

Selecting every white-shirted player in a football video requires more than finding them in the opening frame. Players move, and others enter the shot. In Nikhila Ravi’s demonstration, SAM 3 takes a jersey-color prompt, finds matching players, tracks them and detects new matches throughout the video.

That is the change introduced by concept prompting: a short phrase specifies a category, and the model attempts to find every instance. In the image demonstration, watering can produces a segmentation mask—the region belonging to the object. Clicks and visual examples refine the request: a box around a missed flower tells the model that it also belongs to the target category. SAM 1 and SAM 2 required selecting instances individually.

In the Latent Space conversation, Ravi traces the project to late 2021, followed by SAM 1 in 2023 and SAM 2 in July 2024. The latest launch includes three separate models: SAM 3 handles images and video, while SAM 3D Objects and SAM 3D Body are distinct products.

Masks and tracks give other software regions to operate on. The football demo adds trails, clones players and changes the background to spotlight selected people. Templates combine prompts with effects, and users can upload their own media. The interface also exposes prompt sensitivity: in a busy scene, a disappointing result for tablecloth leads to a trial of table.

Speed depends on the workload and hardware. A host cites 30 milliseconds for an image with 100 detected objects on an H200, without knowing the live demo’s hosting configuration. Ravi explains that video cost grows with object count and that the demo uses multiple GPUs for crowded scenes. The companion article reports real-time capacities of 10 objects on two H200s, 28 on four and 64 on eight, using parallel inference and fast-mode tracking. It does not specify frame rate, resolution or fast mode’s effect on accuracy.

A broad vocabulary, a bounded language task

Text prompting appeared as an early proof of concept in SAM 1 and became its most requested feature, Ravi says. For SAM 3, the team focused on atomic visual concepts: short descriptions of objects and attributes, such as yellow school bus or purple umbrella. Accepting text does not make the model an unrestricted language-reasoning system.

Even that bounded task needs broad vocabulary coverage. SACO—Segment Anything with Concepts reportedly contains more than 200,000 unique concepts, compared with about 1,200 in the earlier benchmark discussed. The aim is to capture the diversity of natural-language requests. Producing the training examples at that scale made the data engine, the process that generates and verifies annotations, a central part of the project.

When “vehicle” works and “Waymo” does not

Reported applications span cancer research, solar-panel counting in aerial imagery, underwater trash-collection robots and fish-population monitoring. Roboflow reports roughly 106 million smart-polygon examples powered by SAM 1, 2 or 3, and about eight million SAM 3 inferences in its first five days. These figures describe usage; Ravi values the resulting deployment feedback because it reveals how models behave in settings the research team otherwise sees less directly.

Broad coverage still leaves specialized gaps. A visitor to the CZI Imaging Institute describes researchers fine-tuning SAM to segment individual lysosomes in human-cell imagery, where structures can be difficult to distinguish. In a street-scene example, SAM 3 recognizes a Waymo under vehicle but returns nothing for Waymo: it finds the broad category but misses the requested subtype.

A ten-second clip can supply examples of that narrower category, and a single image can provide text and visual prompts for subsequent inference. The Waymo discussion blends fine-tuning with prompt-based adaptation, so it does not establish which steps update model parameters. The practical point is that examples can communicate a distinction the category name alone fails to convey.

Negative examples help define the boundary. One speaker reports that even one negative can help, and that three to five negatives alongside positives can substantially change predictions. Suggestions that adaptation can begin with roughly ten data points are starting points, not accuracy guarantees. Coverage matters: new object subtypes or a change from overhead to side views can expose failures. The recommendation is to test those examples against real deployment data.

Two separations inside the architecture

The first separation is recognition from localization: does the requested concept appear anywhere in the image, and, if so, where? SAM 3 adds a learned global presence token for the recognition decision. Individual detector proposals no longer carry the entire burden of deciding both whether a concept exists and where its instances lie.

Training reinforces that distinction with absent concepts as well as positive concept–mask pairs. The team reports that more than 70% of annotations were negative phrases describing things absent from the corresponding image. A valid answer to a concept prompt can be no detections.

The second separation is detection from tracking. A detector needs a shared representation that recognizes different dogs as dogs. A tracker needs distinct representations to preserve each dog’s identity through video. After experimenting with a unified approach, the team kept separate components, retaining SAM 2 tracking components and sharing Perception Encoder, a visual backbone aligned with text and images.

This explains why video compute scales with detected objects, rather than classes: every dog must be tracked independently. SAM 3 combines interactive segmentation, text prompting, detection and tracking in one interface, while the components retain different responsibilities.

Adding reasoning through a visual tool

Locating the larger character requires a comparison beyond naming an atomic category. Pengchuan explains that SAM 3’s small language encoder cannot supply advanced language understanding and reasoning. SAM 3 Agents pair it with a multimodal language model: the language model handles the complex request, while SAM 3 provides visual grounding—the relevant locations and regions.

The relationship works in both directions. Pengchuan says the language model can correct some SAM 3 errors, and cautions that SAM 3 is not as robust as human vision. He reports that standalone SAM 3 performs close to the agent on short phrases resembling its training data, while the agent gains much more on long reasoning requests. This comparison tests the language model’s contribution to SAM 3; the interviewer’s separate question about an agent versus a language model alone remains unanswered.

A live instrument-panel demonstration compares SAM 3, Gemini 3 Pro and Florence 2 on numbers, indicator lights and needles. The presenter highlights SAM 3’s quick response and individual segmentation masks. Gemini’s displayed detections miss some numbers and add erroneous boxes; Florence 2 groups the instrument region instead of separating the requested parts. These are qualitative observations, with Gemini using its hosted API, a possible timeout interrupting the initial comparison and no numerical accuracy measurements. The presenter also acknowledges Florence 2’s age.

The number example surprises a participant involved in the data engine: images heavy in text had intentionally not been prioritized for optical character recognition, or OCR. A subsequent glass-mug demonstration highlights a central design goal. SAM 3 reportedly finds a partially visible mug in a corner. Such easy-to-miss objects are the target of exhaustivity: finding every instance of the requested concept.

Checking what the model missed

A set of accurate masks can still be incomplete. The data engine therefore checks both individual masks and the coverage of the whole set: 1. Source images and noun phrases, including phrases extracted from captions generated by Llama. 2. Use a model to propose candidate masks. 3. Check each mask and discard bad ones. 4. Check whether the remaining masks cover every relevant instance. 5. Have humans add missing masks to incomplete examples.

The team fine-tuned AI verifiers on human judgments, naming Llama 3.2 as an example base model. It reports better-than-human performance on mask-quality and exhaustivity checks, allowing those stages to run automatically, though no evaluation protocol is given. The companion article reports annotation time falling from two minutes per image with human annotation to 45 seconds with model proposals, then 25 seconds with AI verification.

The dataset retains which examples the model completed exhaustively and which needed human intervention. The verifier can also classify new examples as easy or difficult. This makes automation useful for directing human effort toward cases where models still need additional information.

One proposed next step is reinforcement learning from human feedback (RLHF), using people’s judgments about which output is better. The argument is that choosing between outputs can be easier than drawing an annotation from scratch, potentially enabling progress beyond learning only from manually created labels. This remains a research direction. Even the aspiration of full automation comes with an expectation that humans will supply information on the hardest unresolved tasks.

Video makes accuracy a timing decision

Image automation does not transfer automatically to video. The team reports that SAM 1’s released SA-1B dataset was annotated fully automatically, but SAM 2 did not achieve a fully automated video data engine. Improving model-generated video labels remains a priority for SAM 3. It requires stronger perception models and video-capable multimodal verifiers; collecting enough video-mask annotations to train those verifiers is itself time-consuming.

Video offers additional evidence: an ambiguous object at the edge of a frame can become recognizable as it enters fully. Information gathered across its sequence can correct an early classification. But later evidence costs latency. A streaming application must decide before all that evidence exists; a workflow that can wait can make a more robust recognition decision.

The output requirement also matters. Knowing that 11 players are on the pitch differs from preserving each player’s identity. Two design questions follow: - Does the application need per-frame detections or counts, or persistent identities? - Must decisions arrive immediately, or can they use later frames? The answers determine whether tracking and delayed recognition serve the task.

Native perception and the next research steps

Pengchuan illustrates the perception–reasoning gap with a six-finger hand: he describes frontier models answering five and proposes using SAM 3 detections to supply the correct count. The example has no measured success rate, but motivates an architectural question: should a frontier model call a perception tool, or contain that capability natively?

His preference is hybrid. Basic localization and counting—fewer than 20 objects is his example—should be immediate, native capabilities. Counting thousands of objects in a crowded image may justify extra tools and steps. He describes an AI annotator that notices ten detected people are an incomplete set and can call another model to continue. Recognizing incompleteness becomes part of visual reasoning.

Community work already feeds back into the model: the team says SAM 3 incorporated datasets, benchmarks and inference optimizations built on SAM 2. At the time of the conversation, SAM 3 had one model size. Smaller models and more efficient video inference were proposed, potentially within the SAM 3 series.

Video remained a substantial distance from human performance. The decoupled system was not trained end to end; joint training and a larger data engine using AI video annotators were directions to investigate. Connecting perception to decisions was another priority—for example, helping a biology-lab robot judge whether liquid in a test tube is at the correct level.

An open text box also invites requests beyond atomic concepts: reading documents, selecting an object to the left of another, or finding the moment a player kicks a ball. These are anticipated user demands, not promised SAM 3 capabilities. They expose the distance between naming visible objects and reasoning about relationships or actions.

The last mile is defining what counts

Roboflow puts SAM 3 into automatic labeling, custom adaptation and hosted inference. When a concept is already within the model’s knowledge, a text prompt can generate a first pass of labels for images or video. Specialized categories can require fine-tuning; a participant reports that Med-SAM 3 appeared within a week of release for medical adaptation. Endpoints serving base or adapted models were available at launch, while edge deployment remained dependent on smaller models or distillation.

The closing Rapid demonstration uses a ten-second street clip near Roboflow’s San Francisco office. Prompting for vehicle finds vehicles across frames. A confidence control broadens the output enough to include vehicle reflections in a building. Yet the user might want only cars before the crosswalk—or might deliberately want reflections. Detection confidence does not specify user intent.

A host questions whether a category name and a less/more slider can express that distinction, and doubts preference learning alone will resolve it. One proposed workflow would have another perception model inspect the output for reflections and iteratively adjust the threshold. Such a loop still needs a target. Vehicles on the street might help, but someone installing a car-counting camera may not anticipate reflections until seeing the results.

The same issue appears in annotation definitions. On customized datasets, apparently poor SAM 3 performance sometimes reflected disagreement about hand: some annotations covered the palm, while others included the arm. Human-guided adaptation can align the model with the user’s definition. Here, human involvement supplies task meaning, even when visual knowledge is already present.

The team closes by crediting the scientists, engineers, interns and other contributors behind what it describes as its largest project to date, including the demo and launch. Its invitation is practical: try the demo and code, report uses and failures through GitHub issues or Twitter, and suggest future directions.

SACO is offered as a contribution that may outlast SAM 3 itself. The team reports measuring human performance on the benchmark and hopes it will guide future models toward matching or exceeding that level. Alongside application feedback, it keeps attention on both requirements: finding every relevant instance and understanding what the user considers relevant.