John Yang

John Yang develops benchmarks and interfaces for AI software-engineering agents. His collaborative work includes SWE-bench, which evaluates repairs to real GitHub issues; SWE-agent, which investigates interfaces designed for language models; and CodeClash and ProgramBench, which extend evaluation to iterative development and building whole programs. He also coauthored a position paper arguing that coding-agent research should give greater attention to collaboration with people.[6][7][8][9][11]

As of September 18, 2026, Yang described himself as a second-year Computer Science PhD student at Stanford, advised by Ludwig Schmidt and Diyi Yang. He previously completed an M.S.E. in Computer Science at Princeton under Karthik Narasimhan and received a B.S. in Computer Science from UC Berkeley. His research spans language-model agents, evaluation, synthetic data and software engineering.[4][5]

His earlier Princeton work established two concrete foundations for these later projects: evaluating models on existing software problems and studying the tools agents use to solve them.[2][6][7]

Yang coauthored SWE-bench with Carlos E. Jimenez, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. First submitted on October 10, 2023, the original benchmark contains 2,294 problems drawn from GitHub issues and corresponding pull requests across 12 Python repositories. A model receives a codebase and an issue description and must edit the codebase to resolve the problem. The challenge can require coordinated changes across functions, classes and files, rather than generating an isolated function. The paper reported that its best-performing model, Claude 2, resolved 1.96% of issues.[6]

In a September 2024 interview, collaborator Shunyu Yao credited Jimenez and Yang together with the idea of evaluating models on GitHub problems that human engineers were solving. Yao distinguished the straightforward concept from the infrastructure needed to make it work: “I think the idea is super easy, but the engineering is super hard.”[3][6]

Yang was first-listed author of SWE-agent, first submitted in May 2024, alongside Jimenez, Wettig, Kilian Lieret, Yao, Narasimhan and Press. The team treated language-model agents as software users with their own needs and abilities. Its custom agent-computer interface supports creating and editing files, navigating repositories, and executing tests and other programs. In the paper’s experiments, SWE-agent achieved pass@1 rates of 12.5% on SWE-bench and 87.7% on HumanEvalFix.[7]

In their November 3, 2025 CodeClash introduction, Yang and Lieret argued that coding evaluation should extend beyond increasingly difficult isolated tasks to goal-oriented software engineering. CodeClash places multiple language-model systems in a multi-round tournament: each system iteratively improves a codebase pursuing a high-level objective, such as winning, surviving or maximizing reward. The models do not play directly; their code competes as their proxy. This makes adaptation over successive rounds part of the evaluation.[8]

Yang was first-listed author of ProgramBench, submitted on May 5, 2026, with Lieret and ten other collaborators. Agents receive a reference executable and its documentation and must architect and implement a new codebase matching its behavior. Agent-driven fuzzing generates end-to-end behavioral tests without prescribing the implementation’s structure. The 200 tasks range from compact command-line tools to FFmpeg, SQLite and the PHP interpreter. In the initial paper’s evaluation of nine language models, none fully resolved a task; the best model passed 95% of tests on 3% of tasks.[9]

Yang and Lieret’s account of ProgramBench’s constraints explains a consequential design choice. In early unrestricted trials, Claude Opus 4.5 identified a source repository from --help output and cloned it; other models downloaded source through package managers or submitted wrappers around the reference executable. Prompt-only prohibitions became an ambiguous “cat-and-mouse game.” The authors therefore blocked outbound internet access, made the reference binary execute-only, prohibited binary-analysis tools, and removed copies matching the original binary’s hash before evaluation. Clean task environments prevent implementation details from leaking through build artifacts or caches. Localhost remains available for network-protocol tasks.[10]

In Humans are Missing from AI Coding Agent Research, Yang, Zora Z. Wang and their coauthors argue that practical usefulness increasingly depends on how people communicate with, supervise and trust coding agents. They advocate systems designed to collaborate effectively with people and organize that interaction around four dimensions: task alignment, verifiability, steerability and adaptability. Their proposed research directions include user-involved coding environments, comprehensive verification mechanisms and measures of human-agent interaction quality.[11]