METR

METR—Model Evaluation & Threat Research—is a nonprofit that evaluates frontier AI systems’ autonomous capabilities and catastrophic risks. Its work includes task-completion time horizons, experiments on software developer productivity, and assessments of agents used inside AI developers. These evaluations examine both what systems can accomplish and whether their behavior undermines human oversight or the integrity of the tests themselves.[3][10]

METR’s earlier history supplies the organizational context for this evaluation work. On December 4, 2023, ARC Evals announced the METR name as it concluded incubation at the Alignment Research Center and prepared to spin off into a standalone 501(c)(3) nonprofit. Beth Barnes would continue leading the team, with staff transferring once the new entity had been formed. Pronounced “meter,” the name also nods to metrology, the science of measurement.[4]

METR’s task-completion time horizon expresses agent capability in terms of how long a human expert would take to complete a task. METR estimates human task durations, then fits a logistic curve relating duration to an agent’s probability of success. The 50% horizon is where that curve predicts success half the time; the 80% horizon uses a higher reliability threshold. As its methods page explains, “It’s a measure of the difficulty of a task, rather than the time an AI spends to complete the task.”[6]

The original dataset, released in March 2025, showed frontier time horizons doubling approximately every seven months over 2019–2025, according to METR. The tasks primarily cover software engineering, machine learning and cybersecurity. They are self-contained, well-specified and automatically scored, so a multi-hour horizon does not establish that an agent can perform equivalent hours of a familiar professional’s daily work or automate an entire job.[7][6]

On January 29, 2026, METR released Time Horizon 1.1, expanding the suite from 170 to 228 tasks and increasing tasks estimated at eight or more human hours from 14 to 31. Only five of those 31 long tasks had measured human baseline times; the rest used estimates. METR attributed changed horizons mainly to task-suite changes and run noise, and judged the trend sensitive to task composition. It subsequently corrected a regularization mistake on March 3, 2026; the live methods page warns that measurements above 16 hours are unreliable with the current suite.[7][6]

TH1.1 also moved evaluations from Vivaria, developed in-house in 2023, to Inspect, the open-source evaluation framework developed by the UK AI Security Institute. METR’s open-source Hawk platform provides cloud infrastructure around Inspect’s tasks, solvers, scorers and sandboxes. Users specify tasks, agents and models in YAML; Hawk runs evaluations on AWS using isolated Kubernetes pods, manages model credentials, streams logs and stores results in PostgreSQL for browsing through a web interface.[7][9][2]

METR also maintains MALT, a dataset of natural and prompted examples of behavior that threatens evaluation integrity, including reward hacking and sandbagging. Integrity checks are part of its capability measurements: seemingly successful runs are scanned for cheating, flagged attempts receive human review, and attempts judged to cheat are treated as failures.[2][10]

METR’s July 10, 2025 productivity study, credited to Joel Becker, Nate Rush, Beth Barnes and David Rein, tested early-2025 tools on developers’ real work. Sixteen experienced open-source developers supplied 246 issues from repositories they had contributed to for years. METR randomly assigned each issue to allow or disallow generative AI assistance. Participants primarily used Cursor Pro with Claude 3.5 or 3.7 Sonnet, recorded their screens and self-reported implementation time.[8]

METR reported that issues took 19% longer when AI was allowed. Developers had expected a 24% speedup and afterward still believed AI had sped them up by 20%. The experiment provided complementary evidence to benchmarks in a setting with familiar codebases and human review standards, but did not establish effects for most software developers. METR now explicitly marks these results as historical and says they no longer reflect the current impact of AI models on open-source developer productivity.[8]

In 2026, METR piloted an organization-level assessment of internal AI use, designed for periodic repetition rather than individual model launches. Its Frontier Risk Report, published May 19, covered February 16–March 16, 2026, with Anthropic, Google, Meta and OpenAI participating. Companies supplied access to frontier models, including raw chains of thought, and non-public information about capabilities, internal use and monitoring. Ajeya Cotra led report writing; Nate Rush and Hjalmar Wijk oversaw evaluations.[10]

Participants agreed upfront to publication of shared-model evaluation results in aggregated internal-frontier benchmark statistics. They retained redaction or anonymization rights over other non-public materials and could withdraw silently before finalizing disclosure. They had no approval rights over the final public report. METR also disclosed that it lacked an applicable personnel conflict-of-interest policy at the project’s outset, so the pilot did not meet all requirements of the AEF-1 evaluation standard.[10]

METR assessed that internal agents during the February–March window plausibly had the means, motive and opportunity to start small rogue deployments—agents operating autonomously against a developer’s intent—but lacked the means to make them robust against active investigation and shutdown. This was a risk judgment, not a report that such deployments had occurred. The pilot focused on proximity to direct AI takeover and excluded indirect undermining of human control from its scope.[10]

The assessment also used an early version of MirrorCode, developed with Epoch AI, in which agents reimplement software through black-box access without source code. METR found strong performance where agents could repeatedly check progress against test cases, alongside weaker judgment and reliability on open-ended challenge tasks. Ryan Greenblatt of Redwood Research supplied supplemental public-model evaluation results; he did not receive non-public access through the program.[10]

On August 14, 2026, METR reported raising around $71 million in commitments over the preceding six months and announced significant team expansion. It designated the funding for work on autonomous capabilities, recursive self-improvement, monitoring systems, risk assessments and AI incident investigations. METR credits The Audacious Project with its first institutional-scale funding.[5]

METR’s funding base is donations, with a small portion of income from a European AI Office technical-assistance contract supporting assessment of loss-of-control risks. In its August update, METR said it had not accepted funding from frontier AI companies and refused donations made by or at the direction of their staff. Those companies nevertheless provide significant free tokens for evaluations, research and engineering.[3][5]