Olivia Watkins

Olivia Watkins is an AI researcher at OpenAI whose work includes evaluating frontier capabilities and cybersecurity risks. In February 2026, she represented OpenAI’s Frontier Evals team in publicly advocating a move away from SWE-bench Verified: she argued that saturation and contamination had weakened its ability to measure improvements in coding agents, and endorsed SWE-bench Pro. Her collaborative research also includes assessing open-weight model risks and teaching agents to interpret human communication.[2][1][6][4]

In the 23 February 2026 Latent Space interview, Watkins explained that SWE-bench gives an agent a repository and a task drawn from a GitHub issue, then grades its solution through tests. Mia Glaese credited the creation of Verified to collaboration between Watkins’s Frontier Evals team and the Human Data team. Watkins described the original curation effort as a human review of whether tasks were well specified and tests were fair, producing a set of 500 tasks.[1]

Watkins described a revealing GPT-5.2 example: a test required a particular argument that the task had never specified, yet the model’s chain of thought drew on knowledge of a repository version to suggest adding it. She considered the task nearly impossible to pass without that prior knowledge. Glaese connected the observation to a broader contamination investigation; Watkins credited other team members with most of the subsequent analysis.[1]

Watkins distinguished two failures of test fairness. Narrow tests could reject a reasonable implementation because it used a different function or argument name from an unspecified expected name. Other tests demanded additional features absent from the problem description. She said these defects were easier to recognize when reviewers could compare a capable agent’s attempted solution against the tests.[1]

In February 2026, Watkins favored SWE-bench Pro for harder tasks, greater diversity of repositories and languages, and substantially less contamination observed by the team’s auditor. OpenAI’s recommendation published that day likewise urged using Pro because Verified was increasingly contaminated.[1][5]

Watkins argued that coding evaluations should test whether agents make reasonable design decisions when a problem is underspecified. Her example was speeding up part of a codebase where several approaches might work. She also called for reliably graded tasks that would take leading engineers months or teams weeks, rubrics validated by practitioners, and benchmarks for building products end to end.[1]

Watkins coauthored GDPval, an evaluation of economically valuable knowledge work. In the interview, she described it as another Frontier Evals–Human Data collaboration and emphasized the domain knowledge needed to grade such work. She credited Human Data with recruiting professionals to help create tasks, reference solutions, and rubrics. Beyond benchmarks, she wanted measurements of actual AI use: whether systems replace work, augment people, or help them work faster.[2][1]

On her personal website, as captured in September 2026, Watkins described her recent focus as measuring cybersecurity risks posed by frontier AI systems and accelerating cyber defense. She coauthored EVMbench, listed there as a 2026 publication, which evaluates whether agents can detect, patch, and exploit high-severity smart-contract vulnerabilities in realistic local Ethereum environments. Its programmatic graders support both tracking frontier cyber capabilities and developing defensive uses.[2]

Before these 2026 evaluations and public discussions, Watkins had worked on model-release risk assessments and, at Berkeley, on human supervision and adversarial communication. These earlier projects provide context for her technical background in evaluating and guiding agents.[6][4][3]

Watkins was a PhD student at Berkeley’s Center for Human-Compatible Artificial Intelligence from fall 2019 through spring 2024, co-advised by Trevor Darrell and Pieter Abbeel. Her PhD thesis, Towards Agents Which Can Understand Rich Communication, was published as a Berkeley technical report on 2 September 2024. It addresses grounding verbal and nonverbal human communication, using language to guide common-sense exploration, and distinguishing trusted communication from malicious instructions or facts. Its second part benchmarks semantic prompt injection and jailbreak vulnerabilities.[3][4]

In Teachable Reinforcement Learning via Advice Distillation, presented at NeurIPS 2021, Watkins worked with Darrell, Abbeel, Jacob Andreas, and Abhishek Gupta. The method first teaches an agent to interpret human advice that can be composed to coach it through complex tasks, then distills a policy that operates independently of the advice. The researchers described this as a way to train new-task policies with less human effort than behavioral cloning or reinforcement learning would require.[2]

Her collaborative 2024 work included Tensor Trust and StrongREJECT. Tensor Trust collected human-designed prompt injection attacks through an online game and curated them into a benchmark. StrongREJECT addressed jailbreak evaluations that could award high scores to low-quality responses, using a revised question set and grading method that the researchers reported better matched human judgment.[2]

In August 2025, Watkins coauthored Estimating Worst-Case Frontier Risks of Open-Weight LLMs with Eric Wallace, Miles Wang, Kai Chen, and Chris Koch. They used malicious fine-tuning to try to elicit maximum biology and cybersecurity capabilities from gpt-oss. Biology training used reinforcement learning with web browsing on tasks related to threat creation; cybersecurity training used capture-the-flag challenges in an agentic coding environment.[6]

The authors reported that maliciously fine-tuned gpt-oss underperformed o3 on their frontier risk evaluations. Relative to open-weight models, they found a possible marginal increase in biological capabilities without a substantial advance of that frontier. They stated that these results contributed to the decision to release gpt-oss and proposed the method as guidance for estimating harm from future open-weight releases.[6]