Segment Anything and the Work Between a Vision Model and a Useful App

A phone, a Sudoku board, and the hard part

Point a phone at a Sudoku puzzle, recognize its digits, and overlay the completed board. That was Magic Sudoku, which Joseph Nelson traces to a call from his longtime collaborator Brad after Apple released its augmented-reality toolkit, ARKit, in 2017. Brad was comfortable with the mobile mechanics but uncertain about digit recognition. Nelson saw digit recognition as a familiar introductory computer-vision problem. Each thought the other was worrying about the easy part.

Nelson had been working on Represently, applying natural language processing to sort constituent messages for congressional offices. Magic Sudoku brought a related ambition to physical objects: turn what a camera sees into something software can act on. The app attracted press and won Product Hunt’s AR App of the Year, Nelson recalls, but its popularity was short-lived.

The larger idea survived. In this April 2023 conversation, programmable physical objects connect Roboflow’s origins to the newly released Segment Anything Model. The engineering problem runs through both: converting visual recognition into a useful application requires data preparation, a clear task, and a model that can run where it is needed.

The tools outlasted the games

In summer 2019, the collaborators returned to the idea: an app that recognized physical objects, paired with a platform for developers to build interactions around them. Board games offered a familiar, social starting point. Their Boggle feature revealed high-scoring words players had missed among adjacent letter tiles. Unlike automatically completing Sudoku, it left room for the game itself. A Canadian English-as-a-second-language program reportedly adopted it for vocabulary practice.

At the 2019 TechCrunch Disrupt hackathon, they added chess support in 48 hours. The app recognized a physical board’s state, converted it into a standard chess notation, and passed it to Stockfish for move recommendations. Users also wanted to record games for later review. Evan Spiegel, visiting the demo, reportedly suggested another use: cheating against friends.

Boggle had taken most of a summer; chess took a weekend. Although these were different projects, the team credited reusable internal tools for the acceleration. The consumer apps kept attracting sharp adoption spikes followed by gradual decline. The tools looked like the more durable opportunity.

The week before Thanksgiving 2019, they wrote a six-phase manifesto into Roboflow’s repository README: help engineers select images and video frames, annotate data, choose models, and monitor performance. The tooling began gaining traction after its early-2020 release. During Y Combinator that year, Nelson and Brad lived and worked in a Des Moines house, returning to the city where they had grown up and first met.

Recognition becomes useful when it supports a decision

Computer vision can identify sushi ingredients, check roofs for hail damage, monitor protective equipment around machinery, or track riverbank changes in satellite imagery. Rivian is cited as a customer using visual inspection while seeking to scale manufacturing. Another customer measures traffic and curb utilization to support changing a curb’s permitted use throughout the day. Each application connects something visible to an operational decision.

Home automation offers an accessible entry point: send a text when a package arrives, identify birds visiting a feeder, or build an alert for a gate left open. One hobbyist reportedly trained a model to recognize his cat and connected it to a robotic arm carrying a laser pointer, aimed about ten feet from the animal. The proposed gate alert and reported cat project illustrate how recognition can drive an action.

Those applications depend on annotations: labels that mark the objects or regions a model should learn to recognize. Roboflow’s early dataset tools exposed missing annotations in Udacity’s self-driving dataset, intended for academic coursework, including unlabeled strollers. The team recounts relabeling it by hand on a train from Hualien to Taipei to meet a Pioneer deadline—and acknowledging that the rerelease probably still contained mistakes.

How much annotation is enough?

The answer depends heavily on variation in the scene. Scratches on one known part under controlled lighting present a narrower problem than objects appearing in many conditions. For the controlled scratch example, the estimate is 30–50 images for an initial working model; a broader starting heuristic is roughly 200 images per class. These are rough planning estimates, with no defined evaluation metric or test conditions behind the suggested reliability level.

The deployment criterion is whether the model creates more value than the available alternative. That alternative might be a person doing the work, or it might be leaving the task undone. Cost and tolerance for errors belong in the decision alongside predictive quality.

Sports analytics makes the tradeoff concrete. Reviewing footage from a team, its competitors, and scouting prospects consumes many hours. A model that finds a player’s appearances or tracks when the ball enters a region could be useful despite imperfect results, provided the task tolerates those errors and automated review costs substantially less.

For the porch camera, the baseline is a homeowner who does not continuously watch the feed. An always-on detector that recognizes boxes but initially misses flat packages can still provide a useful new capability. Its shortcomings remain opportunities for improvement without making the first version worthless.

Valid annotations can still mean the wrong thing

Roboflow’s first released tool addressed computer-vision annotation formats. A bounding box, the rectangle locating an object in an image, can be encoded using opposite corners or a center point plus dimensions. Coordinates may be absolute pixels or values normalized between zero and one. Training pipelines add different file expectations, including TFRecord, COCO-JSON, and YAML paired with text files.

A converter can produce a file the training pipeline accepts while placing every box incorrectly. Visualizing converted annotations checks that their meaning survived. Roboflow automated the conversions to eliminate repeated script-writing and concentrate the work of finding silent errors in a shared library.

The required annotation also depends on the application’s output: - Image classification assigns tags to an entire image. - Object detection locates individual objects with bounding boxes. - Instance segmentation describes individual object shapes using masks—selected pixel regions—or polygon outlines. - Keypoint detection locates landmarks such as hand joints. - Visual question answering answers questions about an image, such as which foods it contains.

Counting players does not inherently require segmentation; individual boxes can suffice. Masks become useful when precise outlines matter, as in estimating a building’s area from aerial imagery. Detection occupies a practical middle ground, supporting counting and tracking with less painstaking annotation than masks. Running in real time on edge devices—local hardware with limited compute—adds another reason to choose a smaller model.

Why fast detectors still matter

Faster R-CNN uses a two-stage approach: propose candidate regions, then classify them. Single-shot detectors predict detections in one model pass, with training objectives for objectness—whether a region contains an object—and classification. The appeal is reduced compute time, potentially at an accuracy cost; actual latency depends on the model and deployment conditions.

YOLO, short for “You Only Look Once,” is a popular family of single-shot detectors. Joseph Redmon introduced the original in Darknet, whose low-level C implementation offered speed but was difficult to work with. The family later branched: Darknet maintainer Alexey AB released YOLOv4, while YOLOv5 grew from improvements to a PyTorch implementation of YOLOv3. Using the YOLO name outside the original Darknet effort sparked a naming controversy.

By April 2023, YOLO covered multiple fast detection variants, with YOLOv8 offered as a preferred choice. Broadly pretrained foundation models suggested a complementary workflow: use their visual capabilities to prepare data, then train a smaller detector suited to the application. The conversation loosely calls this distillation—transferring a useful portion of a large model’s capabilities into a narrower architecture through generated training examples.

SAM changes the starting point

Segment Anything Model (SAM) generates masks without first requiring manually drawn examples and training for each new object. This is zero-shot segmentation: applying an already trained model to new images without task-specific training. Its early-April release included a permissively licensed model in three sizes and a dataset of 11 million images with 1.1 billion masks. The excitement concerned its segmentation quality; zero-shot transfer itself was already established.

For the porch detector, the proposed workflow is to collect images from the actual camera, use SAM to help generate annotations, and train a smaller detector for deployment. That preserves a model tailored to the porch while reducing manual drawing. Crucially, the model preparing the data need not be the model running continuously. A Raspberry Pi motivates the smaller-model route in the discussion, though no device version or runtime measurements establish a precise hardware limit.

One step remains: a mask does not name the object. SAM may outline a region without identifying it as a package or the door behind it, and it can miss objects in unfamiliar positions or appearances. Applications that need object identities must supply semantic labels separately. For proposed uses such as background removal and video editing, selecting the region itself can be valuable.

SAM separates a computationally substantial image encoder from lighter prompting and mask-decoding components. The encoder produces an embedding, a representation of the image that subsequent interactions use without retraining its weights. Point-and-click prompts indicate the desired region. This split supports interactive browser use: substantial image processing happens separately, while the lightweight prompting stage returns rapid feedback.

Annotation becomes a learning loop

Model-assisted annotation brings training forward, before the entire dataset is labeled. The workflow is iterative: 1. Label a small initial subset. 2. Train a model on those examples. 3. Use its predictions to help annotate the next batch. 4. Supervise and correct the predictions, then repeat. An example starts with 100 labeled images from a collection of 10,000 and uses the resulting model to assist with the next 1,000. Those numbers illustrate the process rather than prescribe batch sizes.

SAM’s own annotation process progressed through three stages: 1. Assisted manual labeling: 4.3 million masks from 120,000 images. 2. Semi-automatic labeling: 5.9 million masks from 180,000 images, with people identifying masks the model missed. 3. Fully automatic generation: 1.1 billion masks across 11 million images. Human attention helped extend coverage before automatic generation supplied the largest scale.

For application developers, the recommendation is to label where the model is likely to fail. Annotation serves learning, so the next batch should address what the current model still struggles with. Roboflow’s workflow follows that relationship: the model makes initial guesses, and the user supervises and improves them.

Grounding DINO supplies a proposed route to semantic labels. It uses text prompts to locate objects with bounding boxes; a hypothetical request to find the left can in a studio image illustrates the interaction. Pairing that text-guided localization with SAM’s masks connects object descriptions to precise regions. Novel datasets can still require review, but the starting point becomes candidate regions and descriptions instead of annotation from scratch.

Prompting specifies which object—and how much of it

In the narrated Roboflow demonstration, the first target is a welded joint between two pipes. The existing Smart Poly tool includes unwanted pipe regions, requiring positive and negative points to refine the mask. The demonstrator reports four clicks to select the weld, compared with one using SAM on the same example. This is a single interaction comparison, not an aggregate speed benchmark.

In the next example, clicking a brick wall reportedly selects its visible regions on both sides of an intervening pole. A shoe selection can be expanded to the person in two clicks, but a missed portion of the face needs another click. The interaction lets the user change the selection’s scope and repair remaining errors.

The chihuahua example makes object-part granularity clear. Clicking an eye initially selects the whole dog. A negative point excluding the rest of the dog narrows the mask to the eye; a positive point adds the other eye. A similar correction narrows a head selection to one ear. Positive and negative prompts resolve the ambiguity between a plausible object boundary and the boundary the user actually wants.

Carrying that intent across images returns to the learning loop: label a few examples, train a small task-specific model, and use it to continue the work. An apron detector provides another proposed application. Large models help establish the labels; a custom architecture learns the particular task. The suggested ten-image starting point is illustrative, not an accuracy threshold.

Broader visual coverage still leaves a long tail

Two deployment routes emerge: run SAM alongside a model that supplies semantic labels, or use both to prepare data for a custom model. Compute requirements, proprietary imagery, and ownership can favor the custom route. The conversation predicts that segmentation and semantic identification will soon share one architecture; in April 2023, that remains a forecast.

The optimism extends to GPT-4 and optical character recognition, or OCR—reading text from images. One participant recounts GPT-4 reading every word in a Discord screenshot; another argues that OCR is effectively solved and predicts a leading model from OpenAI. These are judgments prompted by demonstrations, without comparative OCR measurements. Generating React components from website screenshots is likewise a six-month forecast.

The broader argument describes an expanding center of visual coverage. Common Objects in Context, or COCO, supplies 80 common-object categories; its broad sports-ball category illustrates their coarseness. The long tail contains a manufacturer’s unique product, a logistics company’s expectations for its goods, or one particular house. CLIP, which links text and images, broadens generalization, and GPT-4 is expected to extend it further. Proprietary visual information can still require custom training.

Coverage also differs from deployability. Even a highly capable model may need to be distilled for offline use, edge hardware, or constrained compute. Free-form image descriptions may need further processing before an application can act on them. These integration problems persist as recognition improves.

At recording time, Roboflow explicitly lacked GPT-4 visual access. Its uses were text-only: generating dataset descriptions from known class information and powering a support bot with 480 blog posts, 120 YouTube videos, open-source projects, documentation, and help-center material. The bot generated code snippets and directed users to relevant resources.

Keep up by building and explaining

Programmer assistance also explains the company’s name. Stack Roboflow began as a 2019 project to translate Stack Overflow material for programmers working in other languages. A suggestion to drop “Stack” left Roboflow. The older question-answering tool was still available at recording time, although newer tools were acknowledged to work much better.

The closing advice pairs consuming research with producing working explanations and implementations. Papers shared on Twitter, community Discords, internal discussions, and conversations with friends supplied inputs. Early in Roboflow’s development, the team maintained a goal of three blog posts and two YouTube videos each week for seven months. Training tutorials and Colab notebooks pushed them to reimplement the papers they read.

Explaining something simply tests understanding; hackathons force ideas into working software. Curiosity also means seeking other people’s perspectives after achieving success. Expertise in a framework is never a permanent status: everyone continues learning, concentrated effort can close gaps, and rapid change creates opportunities for teams that turn new capabilities into user value.