Joel Becker

Joel Becker is an AI evaluation researcher whose work at METR tests how model capabilities translate into useful software development. He contributed to a randomized study that found experienced open-source developers took longer with early-2025 AI tools, and to a follow-up that prompted changes to the experiment as growing AI adoption complicated measurement.[4][5]

Becker, Nate Rush, Beth Barnes and David Rein published their developer-productivity study on July 10, 2025. Sixteen experienced developers supplied 246 real issues—bug fixes, features and refactors—from large open-source repositories they had contributed to for years. The researchers randomly assigned issues to allow or disallow generative AI assistance. Developers recorded their screens and reported implementation time; when AI was allowed, they primarily used Cursor Pro with Claude 3.5 or 3.7 Sonnet.[4]

The team reported that tasks took 19% longer when AI was allowed. Before the experiment, developers expected a 24% speedup; afterwards, they still believed AI had sped them up by 20%. The finding concerned experienced contributors working in familiar repositories with code expected to satisfy review requirements, including style, testing and documentation. It did not establish that AI slowed most programmers, and METR subsequently described the result as historical rather than representative of current tools.[4]

Becker and his collaborators presented the experiment as complementary to benchmarks for understanding AI’s potential to accelerate AI research and development. Their distinction was practical: passing automated tests, producing code a user finds helpful, and delivering work ready for repository review measure different things. They argued for using multiple evaluation methods rather than treating any one as a complete account of capability.[4]

A follow-up begun in August 2025 collected data from 57 developers across 143 repositories and more than 800 tasks. In a February 24, 2026 update, Becker, Rush, Tom Cunningham, Rein and Khalid Mahamud judged it an unreliable signal of AI’s current productivity effect and announced work on redesigning the study.[5]

Developers increasingly declined participation because they did not want to work without AI. In surveys, 30% to 50% said they withheld some tasks for the same reason, potentially excluding work where they expected the largest benefits. Compensation had also fallen from $150 to $50 per hour, which the authors believed contributed to selection. Concurrent agent use created another problem: developers could work on unrelated tasks while an agent ran, making time spent on a single issue difficult to report.[5]

The authors believed developers were likely benefiting more from AI in early 2026 than in early 2025, but called their data “only very weak evidence for the size of this increase.” They proposed alternatives including shorter, more intensive experiments, observational data, carefully designed questionnaires and fixed-task experiments. The update therefore documented a change in measurement strategy rather than a clean causal reversal of the original finding.[5]

Becker also contributed with Parker Whitfill, Cheryl Wu and Rush to a March 10, 2026 study of SWE-bench Verified pull requests. Four maintainers from three repositories reviewed 296 AI-generated pull requests. The team reported that roughly half of test-passing patches from mid-2024 to mid/late-2025 agents would not be merged, after adjusting for noise in maintainer decisions using reviews of 47 originally merged human patches.[7]

The reviews were hypothetical, lacked continuous integration and waived testing requirements. Agents had one submission without an opportunity to revise in response to feedback, unlike the usual human development process. Becker and his coauthors therefore did not claim a fundamental capability limitation or an equal-condition comparison with humans; their argument concerned the difficulty of translating benchmark scores directly into usefulness.[7]

In the paper Forecasting AI Time Horizon Under Compute Slowdowns, Becker’s individual contribution is specified as project guidance, data analysis and draft writing. His research page also lists contributions to long-task measurement, HCAST and RE-Bench.[6][2]

Before his AI evaluation work, Becker worked in economics, genomics and sports analytics. These earlier roles provide context for his research background.[2][3]

Becker’s CV lists him as an Analytics FC co-owner and consultant from 2016 to 2018, a Harvard University/NBER predoctoral fellow from 2018 to 2020, and an economics graduate student at New York University from 2020 to 2023. On his personal page, he describes earlier economics and genomics research at NYU, Oxford and Harvard, and a statistics consultancy advising professional soccer teams. He also lists the 2021 Nature Human Behaviour paper Resource profile and user guide of the Polygenic Index Repository among his publications.[3][2]