Pengchuan

Pengchuan Zhang is an AI researcher whose work includes concept-prompted segmentation and visual grounding. He was one of five project leads for SAM 3 at Meta Superintelligence Labs in November 2025, alongside Piotr Dollár, Nikhila Ravi, Kate Saenko and Christoph Feichtenhofer. He also described leading the visual grounding effort for Llama 3 and continuing that work for Llama 4.[3][2]

Zhang and his collaborators extended Segment Anything from visual prompts for individual objects to finding, segmenting and tracking all instances of a concept. SAM 3 accepts simple noun phrases such as “yellow school bus,” image exemplars, or both. A positive bounding box around a dog can prompt it to find all dogs in an image; positive and negative exemplars let users refine missed or unwanted detections. In video, the model preserves object identities across frames.[3]

The jointly led team separated tasks that impose competing demands on the model. A global presence token predicts whether the requested concept appears anywhere in an image, leaving individual object queries to localize matching instances. Each proposal's final score multiplies its local score by the global presence score. Detection and tracking are also separate: the detector recognizes matching objects without preserving identity, while the tracker distinguishes particular objects over time. They share a vision backbone.[3]

The team's data engine checks both mask quality and exhaustivity—whether every matching instance has been masked. Human accept/reject labels were used to fine-tune Llama 3.2 into AI verifiers for these checks. Cases that fail exhaustivity verification go to humans, who add, remove or edit masks. This arrangement directs manual effort toward model failures while automating verification of other examples.[3]

SAM 3 itself is constrained to simple noun phrases and struggles with fine-grained concepts outside its training domains, including specialized medical terms. For more complex requests, the team built SAM 3 Agent: a multimodal language model proposes noun phrases, calls SAM 3, examines the returned masks and iterates. The language model handles reasoning and mask selection; SAM 3 supplies segmentation.[3]

Before SAM 3, Zhang worked on visual grounding in Llama and on image representations in VinVL. These earlier projects provide context for his work connecting language with visual objects.[2][5]

Zhang received his PhD in Applied and Computational Mathematics from Caltech in 2017 before joining Microsoft. He later worked as a principal researcher at Microsoft Research in Redmond; Microsoft's January 2021 VinVL account identifies his title at that time as Senior Researcher.[2][4]

Zhang was the first-listed author of VinVL, published at CVPR 2021, with Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi and Jianfeng Gao. The team addressed an overlooked part of vision-language systems: image encoding. They pretrained an object detector on merged public datasets, then fine-tuned it with an attribute branch on Visual Genome to produce richer representations of objects and their attributes.[5][4]

In their January 14, 2021 account, Zhang, Lei Zhang and Jianfeng Gao reported improvements from replacing visual features while retaining the fusion module's architecture, training data and recipe; the fusion module was still trained. They cautioned that benchmark success did not establish human-level understanding: “our model is by no means reaching the human-level intelligence of VL understanding.” They identified perception-grounded language models as a future research direction.[4]

In a July 31, 2024 entry on his personal website, Zhang described himself as a core Llama 3 contributor who led its visual grounding effort. On April 5, 2025, he wrote, “I’m proud to continue leading the visual grounding effort from Llama 3 to Llama 4.”[2]