Nikhila

Nikhila Ravi is a research and engineering leader in Meta’s Segment Anything (SAM) work. She was lead author of SAM 2 and shares SAM 3 project leadership with Piotr Dollár, Kate Saenko, Pengchuan Zhang and Christoph Feichtenhofer. Her contributions include developing promptable image-and-video segmentation and explaining how interactive tools shape model design, annotation and recovery from mistakes.[2][5][7]

As a shared project lead on SAM 3, Ravi helped develop a model that detects, segments and tracks objects using text or visual prompts. Its extension beyond SAM 2 is concept segmentation: a short phrase or visual exemplar can request all instances of an open-vocabulary concept, rather than requiring the user to select each object individually.[7]

SAM 3’s detector and tracker share a vision encoder but remain separate components to reduce interference between their tasks. A presence token helps distinguish closely related prompts, such as “a player in white” and “a player in red”; the tracker inherits SAM 2’s transformer encoder-decoder architecture and supports interactive refinement. These are collaborative project contributions, with Ravi one of five named project leads.[7]

Before SAM 3 added concept prompts, Ravi led the authorship of SAM 2, which extended promptable segmentation to video. In August 2024 she was a Research Engineering Manager at Facebook AI Research. Her account of that project explains the interactive tracking and correction mechanisms that SAM 3 later inherited.[2][5][7]

Ravi described the public demo as part of the development machinery: “the demo is actually the annotation tool.” Improving the interface accelerated annotation and improved data quality, which in turn helped improve the model. For SAM 2, the team prioritized the demo from day one. She argued that real-time interaction brought efficiency, image-encoder selection and hardware considerations forward: “No one wants to wait.”[2]

Video also changed the demo’s architecture. Ravi explained that sending image embeddings for every frame of an entire video to the browser was impractical. The team changed SAM’s ViT-H image encoder to Hiera, and SAM 2’s efficiency allowed the demonstrated system to run entirely on the server.[2]

The annotation workflow evolved through three stages: applying SAM separately to video frames; pairing SAM with a video model that accepted an initial mask; and replacing that pair with one model for both images and video. Ravi explained the practical advantage of the unified model: annotators could add positive or negative refinement clicks to correct a prediction. In the preceding two-part workflow, they had to delete the frame prediction and annotate the object again from scratch.[2]

SAM 2 adds streaming memory through memory attention, a memory encoder and a memory bank. Ravi described a combination of short-term, high-resolution spatial memory and longer-term object-pointer information. The team used a six-frame spatial-memory window alongside longer-lived object pointers as a speed–accuracy trade-off. Additional prompted frames enter memory, giving users a way to correct tracking failures, including after an object disappears behind something else.[2][5]

In her August 2024 demonstration, Ravi selected a football with a click and tracked it through a video. When a shoe covered the ball and the model made a mistake, she added negative refinement clicks and ran tracking again with the correction. The interface also used timeline “swim lanes” whose thickness indicated whether an object was visible, helping users locate predictions that might need attention. She acknowledged that crowded scenes containing similar-looking objects remained difficult.[2]

Ravi identified omitting class labels as a deliberate SAM design decision. Annotation followed object boundaries rather than a fixed vocabulary of categories, allowing the model to segment unfamiliar objects without first assigning them names. She attached a qualification to that generality: specialized domains requiring expert judgment could benefit from additional fine-tuning data.[2]

Asked in 2024 why SAM 2 did not include native text prompting, Ravi said the team concentrated each release on one capability: class-agnostic segmentation for SAM, then video segmentation for SAM 2. She welcomed community combinations with other models—“That's exactly why we open source our work”—and advocated releasing tools for domain adaptation alongside the base model. She credited research-group head Piotr Dollár with bringing historical perspective and emphasizing general, real-world uses.[2][5]

Her advice to practitioners was to choose the representation their application actually needed. Tracking a point or bounding box could be sufficient; a full, high-fidelity segmentation mask was not necessary for every task.[2]

Ravi’s route into this work began before Segment Anything. She earned bachelor’s and master’s degrees in engineering at Cambridge and spent a year at Harvard on a Kennedy Scholarship. Recalling that period in 2024, she said she had held a place to study medicine at Oxford, but took a gap year involving coding, projects and freelance contracting. During her Harvard year she took computer science classes at Harvard and MIT and worked on AI projects, especially computer vision. She then joined Facebook without a PhD, describing her background as primarily engineering.[2][3][4]

At Facebook, her earlier work included internal tools for machine-learning training and deployment, followed by computer vision and 3D understanding. She coauthored the July 2020 paper Accelerating 3D Deep Learning with PyTorch3D with Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson and Georgia Gkioxari. The library addressed heterogeneous 3D data and differentiable graphics operations with modular operators and a differentiable renderer for meshes and point clouds. This earlier work supplies context for her 3D engineering experience before image and video segmentation.[4][6][2]