OpenAI Frontier Evals

OpenAI Frontier Evals is an internal research team that collaborates with OpenAI’s human data team on evaluations of AI capabilities, including SWE-bench Verified and GDPval. In a February 2026 interview, researcher Olivia Watkins identified herself as a team member and explained how coding evaluations contribute to OpenAI’s assessment of research automation and model autonomy.[1][4]

On February 23, 2026, OpenAI recommended moving to SWE-bench Pro, stating: “SWE-bench Verified is increasingly contaminated. We recommend SWE-bench Pro.” Watkins described Verified as effectively saturated and no longer measuring coding improvements well. She favored Pro’s larger, harder tasks, broader repository and language coverage, and lower contamination in the team’s investigation.[3][1]

The reassessment exposed two interacting problems. Watkins said that over half of the problems investigated in a deeper review had an issue: some tests required an undisclosed function or argument name, while others demanded features absent from the task description. A reasonable implementation could therefore fail. She also recalled a GPT-5.2 solution whose reasoning drew on knowledge of a repository version to supply an argument the task had not requested. Such knowledge could help a model pass a test without demonstrating the intended coding capability.[1]

That recommendation revisited a benchmark Frontier Evals had helped create. Its earlier collaboration with human data explains both the effort behind Verified and the limits that later scrutiny revealed.[1]

OpenAI introduced SWE-bench Verified on August 13, 2024. The benchmark gives an agent a codebase and a task drawn from a real repository and GitHub issue, then grades its solution through tests that are hidden from the agent. Mia Glaese attributed its creation to collaboration between Frontier Evals and human data. In their February 2026 retrospective, Watkins described OpenAI hiring almost a hundred software engineers to review task specifications and test fairness, producing a curated set of about 500 tasks. Glaese described three independent expert reviews; these figures describe the validation campaign, rather than Frontier Evals staffing.[2][1]

On September 25, 2025, OpenAI introduced GDPval, an evaluation of economically valuable, real-world tasks across 44 occupations. Watkins also attributed this evaluation to collaboration between Frontier Evals and human data. Professionals helped create tasks, gold solutions, and grading rubrics because assessing their work required domain knowledge. This approach extended the collaboration beyond coding tasks with automated test results to professional work requiring contextual judgment.[4][1]

In February 2026, Watkins described OpenAI’s Preparedness Framework as tracking biological risk, cybersecurity, and research automation and model autonomy. She said Verified had initially been created for the model-autonomy workstream: coding is an important component of automating research, but evaluating complete research workflows requires more than measuring coding alone.[1]

Watkins and Glaese advocated evaluations of longer tasks, open-ended design decisions, code quality, and maintainability. Glaese emphasized the tradeoff: automated tests are fast and easy to compare across models, but do not establish whether a project maintainer would accept a patch. Watkins called for tasks that could take expert engineers months or teams weeks, reliable professional rubrics, end-to-end product creation, and measurements of actual AI usage. These were research priorities discussed in February 2026, rather than announced benchmark releases.[1]