Why SWE-Bench Verified Stopped Measuring Coding Progress

When a failing test does not mean a failed solution

Give an agent a repository and a GitHub issue, ask it to produce a patch, then run the tests. SWE-Bench Verified uses this setup to evaluate real software maintenance. But when OpenAI examined the original SWE-Bench, some agent failures came from flawed task descriptions and tests. A failing test did not always mean the solution was wrong.

In the Latent Space conversation, Olivia argues that Verified has reached another measurement limit: saturation and contamination now make it a poor guide to further coding improvements. The benchmark’s creation and subsequent problems explain why verification alone could not preserve its usefulness.

The original verification effort curated SWE-Bench, an academic benchmark from a Princeton lab. OpenAI’s Frontier Evals and Human Data teams collaborated on the project; almost 100 software engineers reviewed task specifications and test fairness to produce a set of 500 tasks. Mia describes three independent expert reviews for each problem. Reviewers had to understand the issue and reference patch in the context of the surrounding codebase. She considers the repeated reviews necessary and suggests more might have helped.

Repository knowledge can supply a missing requirement

Contamination occurs here when prior familiarity with benchmark material or its source repositories helps a model solve the evaluation. Filtering the published benchmark does not close every route. Canary strings—markers added to evaluations so they can be filtered out of training data—do not mark the original open-source material. Popular repositories such as Django also appear repeatedly across GitHub, complicating attempts to exclude that material.

Olivia describes a GPT-5.2 run in which a test expected a particular argument that the task never specified. The model’s reasoning referred to that argument appearing in a later version of the repository and proposed adding it. She interprets this as later repository knowledge supplying a missing requirement, making an otherwise very difficult test possible to pass. Mia says the observation prompted an investigation across OpenAI and other frontier models.

Narrow tests create a misleading ceiling

A renewed fairness review focused on problems o3 could not solve reliably. Human reviewers investigated whether these failures reflected model limitations or defective tasks. Olivia reports defects in more than half of the problems examined in that targeted review—a finding about selected difficult problems, not a defect-rate estimate for all 500 tasks.

Two recurring defects made the tests demand more than the task description: - Unspecified implementation details: a working feature could fail because an argument or function used a reasonable name different from the expected one. Olivia identifies overly narrow tests as the most common defect. - Unrequested features: some tests required additional behavior that the issue never asked the agent to implement.

These tests accept only a subset of viable solutions. A capable agent’s failed implementation can make that restriction easier to spot: comparing a concrete, plausible solution with the expected outcome exposes assumptions that reviewers may miss when assessing a task in the abstract.

Saturation can arrive below 100%. Once remaining gains depend on guessing hidden details, higher scores become less informative about coding ability. After GPT-5.2’s release, an examination reportedly found it solving 31 problems classified as very hard to solve without contamination. This raised the possibility that models had already reached the uncontaminated ceiling, though that ceiling remained difficult to estimate.

Mia still regards Verified as a strong benchmark when it was published. Producing a satisfactory patch from a GitHub issue is a real capability. The concern is whether further gains around the 80% level still measure improvements in that capability: a useful benchmark can outlive its ability to distinguish progress.

SWE-Bench Pro offers more headroom

Olivia endorses SWE-Bench Pro as a better next evaluation for three reasons: - Larger, harder problems: roughly 90% of Verified tasks were estimated to take an expert engineer less than an hour. Pro’s larger tasks offer more room to measure improvement. - Broader coverage: Pro spans multiple repositories, languages and qualitatively different problem types. - Less detected contamination: the same auditing approach found much weaker evidence of familiarity on Pro than on Verified.

The contamination auditor was itself an agent. Given a task description, reference patch and task ID, it explored questions that might expose a target model’s familiarity with the material. On Verified, the audit reportedly elicited reference solutions and sometimes task IDs from OpenAI models, Claude Opus 4.5 and Gemini Flash—evidence of at least familiarity with the source repositories.

On Pro, the auditor found only weak indications that a couple of models might have limited familiarity with one or two repositories. That supports a relative comparison, not a claim of zero contamination. Pro is also expected eventually to need replacing as capabilities advance; harder and more representative tasks remain an ongoing requirement.

From passing tests to code a maintainer would accept

An ideal coding evaluation would test design decisions under underspecified requirements. One proposed task is to speed up part of a codebase when several approaches could work. The evaluator must allow reasonable choices while judging the result. Leaving room for design judgment differs from omitting a requirement and then testing for one hidden implementation.

The desired scope extends beyond small issues to work lasting hours or days. It also includes qualities that a pass/fail score does not capture: - Design taste and team fit: does the agent choose an approach the team would want? - Code quality and maintainability: is the implementation well written and suitable for future work? These qualities matter to users but are harder to measure. Passing tests does not establish that a maintainer would merge the pull request.

GDPVal illustrates how professional knowledge can enter the grading process. Built through a collaboration between Human Data and Frontier Evals, it evaluates varied white-collar work across professions and specializations. Practitioners helped create tasks, reference solutions and grading rubrics—the criteria used to judge a result—because reliable grading required knowledge of each professional context.

Applying human judgment to coding introduces a practical tradeoff: - Human review assesses quality in context, but makes evaluation slower and more complex. - Executable tests provide speed and comparability. Once tests run, pass/fail results are easy to aggregate across the industry without human involvement, though important questions about quality remain unanswered.

Measuring research workflows and longer autonomy

Evaluating AI research coding presents another obstacle: realistic task data may be private. Many state-of-the-art AI codebases are proprietary. This is one proposed reason public benchmarks underrepresent research workflows: realistic tasks are harder to build and share when the underlying code cannot readily be released.

Monetary value, human completion time and task complexity offer related ways to frame the capability question: how much complex, long-running work can an agent be entrusted with? They are useful proxies in this discussion, without an established quantitative relationship that would make dollars and hours interchangeable.

This question also connects coding evaluations to OpenAI’s Preparedness Framework, a public framework for tracking risks from dual-use capabilities—those that can serve beneficial or harmful purposes. At the time of the conversation, it tracked biological risk, cybersecurity, and research automation and model autonomy.

SWE-Bench Verified originated in the model-autonomy evaluation workstream. Coding is an important component of research automation, but patch production covers only part of it. The proposed next step is evaluating whether models can automate research workflows themselves.

Harder work needs credible grading

The closing appeal is to build and share evaluations across the field, with a willingness to adopt stronger work from elsewhere. The requested directions are concrete: - Tasks lasting weeks or months: work that would take top engineers months or teams weeks, paired with reliable grading—for example, rubrics sourced from and validated by many practitioners. - End-to-end product creation: evaluate the ability to deliver a complete product. - Real-world outcomes: track AI adoption, worker augmentation, speed improvements and job replacement, even when these measurements fall outside conventional benchmarks.

Asked what OpenAI will release next, the respondent offers real-world impact as a general direction without specifying forthcoming work. The open challenge is to make harder, more realistic tasks reliably gradable, then connect those results with what agents enable people to do.