ICLR 2024, Part 2 of 5: Compute, Context, and Benchmarks for Working Agents

Compute and credit shape academic research

Academic researchers can find compute providers, but they compete with everyone else for the same resources. Early in Part 2 of the ICLR 2024 recording, a participant argues for substantially expanding US academic compute through government investment. The constraint is timing: government moves more slowly than the field.

Industrial secrecy also weakens visible academic credit, a participant argues. When companies publish less about their models, graduate students have fewer ways to demonstrate that industry adopted their research. UW’s Matryoshka embeddings become a contested example: one participant alleges use by OpenAI without adequate acknowledgment, while another considers an oversight more plausible than deliberate exclusion. Sora receives a more favorable assessment for citing its inspirations, including diffusion transformers.

The discussion identifies three valuable academic contributions: - Simple modeling changes that work. A participant criticizes novelty-focused reviewing and praises DPO as a useful, widely used method whose mathematical presentation obscures its practical simplicity. - Benchmarks that make progress measurable. Building evaluations is substantial research work in its own right. - Open-source engineering that accumulates across projects. Academia could strengthen open alternatives to large companies, but dispersed effort and poor coordination limit its contribution.

Hugging Face, OpenDevin, and MergeKit, a model-merging project, illustrate how contributions can come together. A conference organizer describes including a MergeKit talk to give engineering work a venue, arguing that such projects receive little visibility at ICLR while acknowledging the overlap between engineering and research.

Small experiments can miss improvements that appear at scale

One educator describes expanding a language-processing syllabus’s coverage of distillation and synthetic data, alongside a comparative tour of language models. The striking observation is how similar their architectures have become. The educator interprets convergence around the Llama architecture as reaching at least a local architectural optimum: engineering has produced a strong shared design.

Another participant challenges the suggestion that architecture work is largely finished, citing mixture of experts as a significant scaling improvement. A Meta experiment offers a second example: training models to predict the next four tokens instead of only the next token. The participant recalls no improvement at small scales but gains beyond a threshold tentatively remembered as 13 billion parameters; neither a metric nor the size of the gain is given.

A small-scale negative result may not settle whether a method works at scale. Testing some ideas requires sufficiently large models and enough training data, strengthening the case for academic compute. Meanwhile, the educator places more emphasis on data selection, cleaning, deduplication, and training methods. Architecture teaching increasingly explains why existing designs work.

Mamba and RWKV deserve time to mature, the researcher argues, because comparisons with transformers also reflect unequal engineering investment. They recall neural machine translation taking a year or two to surpass phrase-based systems that had accumulated roughly a decade of engineering. An alternative’s early performance need not reveal its eventual ceiling.

Long context needs recall without scanning everything

Forgetting makes the architectural tradeoff concrete. Participants identify information recall as a limitation of pure Mamba and RWKV models. Jamba combines seven Mamba layers with one transformer layer, offering a possible way to recover information that the recurrent state may fail to retain.

The proposed approaches address different parts of the problem: - Hybrid attention may improve recall, but retaining one attention layer in eight only reduces the quadratic attention burden by a constant factor. A participant argues that it remains a bottleneck at contexts of one million to ten million tokens. - Retrieval could preserve access while limiting work. For ten million to one hundred million tokens, a participant proposes keeping information available while making work per generated token grow more slowly than context size, rather than compressing everything into a state.

Unlimiformer implements retrieval-based attention by encoding previous context into a FAISS retrieval index. A researcher involved in the work reports that it worked well with T5, without giving a numerical result. They identify rotary positional embeddings, or RoPE, as an obstacle: retrieval-based attention must handle their relative positional encodings appropriately, a detail left unexplored in the conversation.

The broader direction is efficiency through approximation. A k-nearest-neighbors operator, abbreviated kNN, would retrieve a few relevant items instead of attending across millions of keys and values. The participants favor investigating this approach; they do not present it as a solved design for extremely long context.

The interview closes by inviting contributions to OpenDevin: test agents, add agents suited to particular tasks, and continue developing open-source AI. Its stated May roadmap calls for a strong open agent. Collaboration with members of the Qwen team makes Qwen a plausible foundation, while Llama 3 remains under consideration. A separate participant describes Cursor’s recruiting at the time for researchers, engineers, and machine-learning engineers working on code generation and assistive coding.

SWE-Bench tests changes inside real repositories

SWE-Bench asks a model to resolve a GitHub issue in an existing codebase. Carlos Jimenez introduces the benchmark, developed with collaborators at Princeton and the University of Chicago and co-led with John Yang. His evaluation criteria are challenge, practical relevance, and easy verification. HumanEval checks functions written from signatures and docstrings, but Jimenez argues that software engineering also requires fitting changes into a larger project.

Each task supplies a repository and an issue describing a bug, requested feature, or behavior change. The model edits one or more files, and tests associated with the issue’s resolution evaluate its patch. Locating the change and understanding its effects elsewhere are part of the task.

The collection process converts accepted development work into executable evaluations: 1. Collect accepted pull requests from 12 popular Python repositories. 2. Keep requests that change both source and tests, pairing a solution with associated checks. 3. Verify automatic installation and test behavior, including relevant tests that fail before the accepted source change and pass afterward.

A SymPy issue involving unexpected identity-matrix output illustrates the structure. The repository snapshot corresponds to the issue’s submission. The gold patch is the accepted source-code solution; the test patch contains tests added or updated in the same pull request. The gold patch supplies one valid solution, while tests evaluate the model’s alternative.

The collection contains more than 2,000 verified tasks. Codebases average about 3,000 files, while accepted patches typically edit only one or two: a small change can require searching a large project. Another 19,000 unverified instances, described as lacking test cases, were collected for training.

The initial retrieval-augmented generation (RAG) baseline selects files with a BM25 sparse retriever, then gives the model the issue, project README, and complete contents of the highest-ranked files. The model generates a patch from that material. Fine-tuning CodeLlama on long-context retrieval examples also produces the released SWE-Llama 7B and 13B models.

The experiments expose three obstacles: - File selection matters. Claude 3 Opus resolves 3.8% of issues in the presented baseline. An oracle condition supplying every file changed by the gold patch raises the reported Claude 3 result to 9.1%, but assumes knowledge of the accepted solution’s file locations. - Longer inputs correlate with worse performance. Jimenez reports a strong relationship for Claude 2 and a similar pattern across essentially all tested models. The experiments do not establish context length as the cause. - Patches miss repository conventions. Qualitative inspection finds shorter, simpler model patches that overuse Python built-ins and overlook library-specific utilities and APIs.

SWE-Agent makes issue resolution interactive, giving the model an interface for working with a computer. Jimenez reports up to 12.5% of SWE-Bench resolved with the framework. The model and detailed evaluation configuration are unspecified, so that result does not isolate the interface’s contribution.

Cost, task selection, and human control change the evaluation

In the following interview, a project contributor credits Devin with making SWE-Bench progress tangible: a demonstration could show what resolving roughly 15% of issues meant in practice. SWE-Agent served a related purpose—showing that initially low scores on a difficult benchmark could improve through agentic approaches.

The collection strategy favored depth in well-maintained repositories, yielding a reported 2,294 instances from 12 projects. Manual inspection of contribution guidelines and tests helped develop heuristics the team judged reliable for Python packages on PyPI. Extending the evaluation harness to Rust or Scala would require substantial engineering and collaboration with language experts or repository maintainers.

The interviewer challenges the initial characterization of human performance as 100%. Every issue has a human-written solution, but those solutions came from many people; they do not measure one engineer’s performance. Patch size is also a poor difficulty measure. A one-line edit can require deep repository knowledge. The team describes ongoing work to characterize difficulty using both problem statements and code changes.

Evaluation costs constrain repeated experiments, but the quoted figures measure different things: - Retrieval-based evaluation: an estimated $0.20 per instance, with no model or configuration specified. - SWE-Agent: roughly $1–$2 for successfully resolved issues, with a $4 spending cap. This is cost on resolved issues, not an average across all attempts.

SWE-Bench Light, as it is called in the discussion, offers a more accessible starting point: 300 tasks filtered for criteria including one-file changes and reproducible code in the issue. The split deliberately removes some task diversity. The contributor relays a claimed 21% result for CodeAct, associated with the OpenDevin team, on this subset. Its evaluation configuration is unspecified, and the score is not directly comparable with full-benchmark results.

The SWE-Agent version discussed operates without human intervention. A next research question is how to preserve control while offering autonomy: why people intervene, when they want to do so, and which interfaces help. The contributor treats interface design empirically because some intuitive choices worked for language agents and others did not, while declining to make a detailed comparison with competing systems.

Memorized example order can reveal contamination

Yonatan introduces test-set contamination with a hypothetical accident: someone uploads Codeforces problems to GitHub, a crawler collects them, and they enter a model’s training data. A later benchmark score becomes difficult to interpret. Checking for that overlap is especially hard when models are available only through APIs or their training datasets remain undisclosed.

He recounts observations of GPT-4 performing differently on older and newer Codeforces problems and Phi 1.5 completing a GSM8K math example. These are circumstantial signals. The proposed statistical audit instead requires a test set X and access to text log probabilities under model θ, aiming to bound the false-positive rate, also called the type I error rate, by a chosen level α.

The null hypothesis is that model and test set are independent, with model randomness arising from the draw of training data. The crucial prerequisite is exchangeability: permuting the test examples leaves their joint distribution unchanged. Under exchangeability and independence, the model should have no inherent preference for the benchmark’s original ordering.

Training on the benchmark can break that symmetry. Several adjacent examples may fit into one training context window, allowing the model to memorize something about their order. An unusually strong preference for the original ordering provides contamination evidence, under the stated assumptions.

The permutation test follows that mechanism: 1. Score the original sequence using its log likelihood under the model. 2. Shuffle the examples and rescore repeatedly, changing their order while preserving their contents. 3. Rank the original among the shuffles to obtain a p-value measuring the statistical evidence against independence. An original ordering that outranks a million random shuffles illustrates an approximately one-in-a-million event under the null. Repeated whole-dataset scoring is expensive, so the proposed sharded rank comparison test aggregates smaller tests; its implementation is left to the paper.

For controlled validation, the researchers train a 1.4-billion-parameter model on 20 billion Wikipedia tokens, injecting benchmarks at different duplication rates. At a rejection threshold of 0.05, detection depends strongly on how often the benchmark appears: - Ten or more copies: every tested case is detected. - Four copies: test sets are detected about half the time. - Two copies: some test sets are detected. - One copy: detection remains unsolved for these statistical tests.

The public-model audit reports a signal for Mistral 7B on ARC-Easy that remains just significant after correction for multiple testing. MMLU requires separate treatment because some of its test sets violate exchangeability and are excluded. Yonatan describes the findings as consistent with the Llama 2 report’s mild evidence of MMLU contamination.

A high p-value does not establish clean training data. These tests are most sensitive to repeated verbatim contamination; testing many model–benchmark pairs also requires correction for chance significance. Yonatan interprets the audit as suggesting that popular benchmarks were unlikely to appear many times in the audited models’ training data. Low-copy contamination and its effect on scores remain unresolved. The researchers make their models and benchmarks available to support further detection work.

GAIA tests assistance through operations on the world

GAIA is a benchmark for general AI assistance: multistep reasoning, combining information, browsing, and parsing material in an open world. It grew out of a Meta workshop debate about whether language models alone were sufficient. Its designers wanted to test tasks ordinary people could solve with enough time, even when models that perform well on expert questions struggle.

GAIA has three difficulty levels. Level one requires roughly one or two reasoning or browsing steps; level three can require ten to twenty, potentially including multimodal processing. The guest reports roughly 90% performance for human participants and about 10% for GPT-4 on level-one tasks. The human result’s subset and GPT-4’s tool configuration are unclear, so the figures do not establish a comparison under identical conditions.

Memory, tools, and planning are associated with reported agent progress from roughly 10% to 40%. The guest mentions Friday in OS-Copilot and Microsoft’s AutoGen without assigning separate scores or specifying exact configurations. GAIA also offers a way to measure how changing an agent’s underlying language model affects task performance. In this interview’s context, the guest says Llama 3 is still in development and has not yet been tested on GAIA.

The benchmark supplies a prompt plus access to the world, without requiring a dedicated synthetic environment. Open-ended browsing nevertheless needs a checkable destination: questions are constrained to have unique answers for automatic evaluation. Specifying a Wikipedia source and date can resolve annotator disagreements, at the cost of making prompts less natural than everyday requests.

Asked about contamination and leaderboard cheating, the organizer describes three safeguards: - Withhold the test set. It was not publicly released; manually answering its questions is estimated to take roughly an hour each. - Create follow-up questions. Fresh tasks could test whether a submitted system can reproduce its claimed capability. - Request answer traces. Submitters are generally asked to show how the model reached its answer. These are proposed defenses, rather than demonstrated guarantees against cheating.

The closing example makes the task design concrete. Louis XIV’s birth date may be recalled from training. Counting occurrences of a particular spelling of his name on a Wikipedia page requires inspecting the page and deriving an answer that is not explicitly stated there. GAIA’s questions are oriented toward such operations on information.

Asked what limits agent performance, the respondent proposes adding planning and backtracking to a system powered by a language model. Further investigation is planned: these are hypotheses for improvement, not an established diagnosis of GAIA’s bottlenecks.