OpenAI Frontier Evals & Human Data

OpenAI’s Frontier Evals and Human Data teams collaborate on evaluations of AI capabilities, including SWE-bench Verified for software engineering and GDPval for economically valuable work. Their work combines benchmark construction with expert judgment about whether tasks and grading criteria reflect the capabilities being measured. In February 2026, OpenAI recommended moving to SWE-bench Pro, describing Verified as increasingly contaminated.[1][3][4]

In the February 23, 2026 interview, Olivia Watkins identified herself as a Frontier Evals team member. Mia Glaese introduced herself as an OpenAI VP of research whose teams were Codex, Human Data and Alignment, working closely with Olivia’s team. Glaese attributed SWE-bench Verified to collaboration between Frontier Evals and Human Data; Watkins credited the same teams with GDPval.[1]

On February 23, 2026, OpenAI’s assessment stated: “SWE-bench Verified is increasingly contaminated. We recommend SWE-bench Pro.” Watkins argued that saturation and contamination meant Verified no longer measured coding improvements well. She favored Pro’s larger, harder tasks, greater repository and language diversity, and less evidence of contamination in the team’s probing.[4][1]

Watkins described two ways tests could reject reasonable solutions: requiring an implementation detail that the issue never specified, such as a particular function or argument name, and demanding additional features absent from the problem description. Glaese explained that these tests accepted only a narrow portion of the space of viable solutions. Watkins credited other team members with much of the later analysis.[1]

Watkins also recounted a GPT-5.2 solution whose reasoning supplied a specific argument expected by a test even though the task had not requested it. Glaese said this observation prompted a broader investigation of contamination in OpenAI’s models and other frontier models. The concern was that benchmark success could reflect prior familiarity with repository changes rather than solving the supplied issue.[1]

That recommendation followed the teams’ earlier effort to make software-issue evaluation more reliable: SWE-bench Verified, released in 2024, used human review to curate the original benchmark.[2][1]

OpenAI released SWE-bench Verified on August 13, 2024 as a human-validated subset evaluating models’ ability to solve real-world software issues. In the benchmark setup Watkins described, an agent receives a codebase and a task drawn from a repository’s GitHub issue, produces a solution, and is graded by tests.[2][1]

Watkins recalled OpenAI hiring almost a hundred software engineers to assess whether tasks were well specified and tests were fair, producing a curated set of 500 tasks. Glaese described three independent expert reviews as necessary because assessing an issue and its patch required understanding the surrounding codebase. These figures describe the review campaign and benchmark size.[1]

Watkins said Verified was initially created for the model-autonomy evaluation workstream in OpenAI’s Preparedness Framework. In February 2026, she described the tracked categories as biological risk, cybersecurity, and research automation and model autonomy. Coding was one important component of the broader question of whether models could automate research workflows.[1]

OpenAI introduced GDPval on September 25, 2025 to measure model performance on economically valuable, real-world tasks across 44 occupations. Watkins described Human Data recruiting practitioners from those professions to help create tasks, gold solutions and grading rubrics, because judging the work required domain knowledge.[3][1]

Glaese emphasized the practical value of automated tests: they are quick to run and their results are easy to aggregate and compare. But passing tests does not establish whether a maintainer would merge a change, whether its design suits a team, or whether the code will remain maintainable. Human judgment can address those questions, she said, at the cost of a slower, more complex evaluation process.[1]

Watkins’s February 2026 priorities included tasks that would take top engineers months or teams weeks, reliable grading with expert-validated rubrics, end-to-end product-building benchmarks, and metrics of real-world AI use. These were research directions: measuring longer and more open-ended work, alongside how AI augments or replaces human activity.[1]