After SWE-bench: Evaluating What Happens After the Patch

SWE-bench: from slow adoption to broader coverage

SWE-bench evaluates coding agents using existing software issues and pull requests. Released in October 2023, it initially attracted little use. John Yang credits Cognition’s Devin launch with helping turn it into a focus of competition. Adoption brought another problem into view: a Django-heavy benchmark covers a narrow slice of development.

The benchmark’s expansion has taken several forms: - Independent benchmarks. SWE-bench Pro was developed outside the original team. Yang welcomes it while making clear that his team was not involved. - Broader repository coverage. The team deliberately diversified repositories in its multimodal and multilingual follow-ups. The multilingual extension covers roughly nine programming languages across 40 repositories, including JavaScript, Rust, Java, C, and Ruby.

Adding languages or repositories gives a straightforward reason for a new task set. Authors increasingly justify new sets through curation techniques intended to produce harder tasks. For Yang, where that extra difficulty directs evaluation—and how durable this approach will be—remains an open question.

CodeClash makes earlier changes consequential

Yang’s motivation for CodeClash starts with two limitations he sees in SWE-bench: unit tests as verification, and independent task instances. Once a model submits a solution, the episode ends. Its later work never reveals what earlier decisions made easier or harder. CodeClash evaluates development on a codebase that persists across rounds.

Two or more models each maintain a codebase. Their programming tournament repeats this sequence: 1. Choose improvements. Each model edits its code as it sees fit. 2. Compete in an arena. The codebases run against one another in an environment that determines which performs better. 3. Continue from the resulting code. The next round builds on earlier changes, making their consequences part of the evaluation.

A language model acting as a judge is one possible evaluation mechanism, but the initial release centered on existing programming games. In Halite, programs control fleets, attack and defend, and collect resources. Changes to code produce strategies whose consequences emerge through competition.

Economically valuable arenas were still being developed at the time of the interview. Yang sees proximity to real work as a major strength of SWE-bench and Terminal-bench, and a goal CodeClash needed to pursue beyond its initial games.

Runtime, evaluation cost, and impossible tasks

SWEfficiency evaluates speed improvements while preserving behavior. Models modify an existing codebase—for example, through parallelization—to improve runtime while keeping unit tests passing. AlgoTune is related optimization work; SciCode addresses scientific coding. Completion-only tests also retain a practical role: SWE-bench and other agentic evaluations are expensive to run, making simpler coding tasks useful before moving to costly, multi-turn evaluation.

User simulation raises a realism problem: an evaluation samples only one interaction path. That concern accompanies interest in Tau Bench and Vending Bench. Allegations that some Tau Bench tasks are underspecified or impossible remain disputed in the conversation; a rebuttal is mentioned, with task ambiguity and simulator behavior identified as possible causes to investigate.

Deliberately impossible tasks could serve two distinct purposes: - Flag suspicious success claims. Reporting success where none is possible could expose cheating. The suggestion that scores above 75 on Tau Bench v2 indicate cheating is a hypothetical joke, not an established threshold. - Test acknowledgment of limits. ImpossibleBench modifies SWE-bench Verified issues to make them impossible, then checks whether models recognize that they cannot complete them. A participant tentatively recalls models claiming completion instead, without supplying model-specific scores or evaluation settings.

Long autonomy meets interactive development

Terminal-bench allows more creative task construction. SWE-bench’s reliance on existing issues and pull requests supplies realism but constrains task diversity. A proposed direction for Terminal-bench is to broaden its contributors and environments, including coding environments used for non-coding tasks and environments outside coding altogether.

Broader environments also invite a vision of long-running agents: specify a goal and a verifier—a signal that approximates the desired outcome—then let the agent optimize for hours. Judging a codebase by how much money it makes in a particular setting is offered as a hypothetical example. The attraction is delegating implementation decisions and returning to an improved result.

The host pushes back: underspecified goals call for fast human–agent interaction. Repeated back-and-forth may better represent useful development, though it is harder to benchmark. An agent running for a day would demonstrate a capability; whether people would use that mode in real work is a separate question. Benchmark choices matter because they influence which capabilities the industry pursues.

The response favors human involvement matched to the task: - Close control: hands-on development in Windsurf illustrates work where a developer wants to stay involved. - Extended delegation: routine data processing with extensive JSON parsing illustrates work someone might prefer to hand off and walk away from.

Measuring collaboration and codebase understanding

Studying interaction creates a data problem for academic research. The closing research request seeks insight into real coding-assistant interactions and ways to evaluate them at scale. Two difficult routes are identified: - Attract real users. Build a compelling product people use consistently, with LMArena offered as an example. - Build credible user simulators. Reproduce realistic interaction behavior; simply asking ChatGPT to act like a human is insufficient.

CodeClash could support these studies through agents sharing a codebase and human–agent teams competing against teams composed entirely of agents, across one or several arenas. The companion article makes the proposed comparison explicit: hold model capability fixed while varying the collaboration setup, then repeat as models improve. This would help isolate interaction effects and reveal how useful collaboration patterns change.

The host closes with codebase understanding beyond retrieval: helping people understand their repositories and enabling human–model collaboration on tasks neither could accomplish alone. This is described as an anticipated direction for Cognition, alongside work on a research sub-agent for automatic context engineering—preparing context for a large language model.

How to benchmark understanding remains open. Freezing a repository, manually curating answers, and asking trivia questions is an obvious approach, but the host expects models to exhaust its difficulty quickly. The final exchange notes Google’s then-recent code-wiki offering, clarifying that it came from a team separate from Antigravity. Repository explanation tools provide a concrete setting for the unresolved evaluation question.