ICLR 2024, Part 1 of 5: Finding the Tasks Agents Still Cannot Do

What survives a curriculum overhaul?

An advanced natural language processing (NLP) instructor returns to a course last taught before ChatGPT and must rebuild much of the syllabus. The problem is deciding what deserves scarce teaching time: researchers need techniques they can use in their next projects, while older algorithms may remain instructive without belonging to the modern model-building stack. That tension opens this reading edition of Latent Space’s ICLR 2024 conversation, Part 1 of 5.

One survivor is the n-gram language model, which uses counts of word sequences to predict what follows. It gives students a concrete way to understand language modeling before introducing neural networks. Its difficulties also help explain why neural methods matter.

Another participant points out that n-grams can supply proposals for speculative decoding: a draft mechanism proposes several future tokens so generation can process tokens together. N-gram statistics can provide those proposals in place of a smaller neural draft model. The instructor agrees—an older technique can return as a component of a newer system.

WebArena: completing tasks on a sandbox web

The discussion moves to three projects: WebArena, Sotopia and performance-improving code edits. Their research motivation begins with a problem that emerged around 2022: stronger GPT models made it harder to identify worthwhile unsolved tasks. Evaluation became a way to decide what to build, avoiding work on problems the strongest models could already solve.

WebArena tests agents on realistic website tasks. Its first author, identified as Shuyan, originally wanted a robot to help with housework. The interviewee considered robotics the immediate bottleneck, so the team chose web tasks that still required planning across many actions and applying world knowledge.

The benchmark evaluates task completion rather than imitation of human behavior. Its sandbox uses production-grade open-source sites, including OneStopShop as an Amazon counterpart, Postmill as a Reddit counterpart and GitLab for activities associated with GitHub. The authors generalized activities from a month of their own Chrome histories into tasks, then wrote validators to check success.

A representative task asks how much the user spent on food in the past month. The agent must navigate to purchase records, identify food purchases, filter them by date and add their costs. Arithmetic is only the final step: the request combines navigation, filtering and calculation, all described as weaknesses of the models being evaluated.

Initial model success was below 15%, against 78% for humans. Over roughly six months, the reported state of the art rose from about 14% to 25–30%, using GPT-4 as the base model. The researcher attributes the improvement mainly to how agents use the model, with several concrete changes: - Better action interfaces: refine prompts and the available ways to scroll, click and navigate. - Reflection with rollback: assess whether a step helped, then back out if it did not. - Relevant site documentation: supply instructions about where to perform an activity based on the agent’s location and goal.

The human score also exposes validator errors. The researcher tentatively attributes about half the human shortfall to instruction-following mistakes and half to overly strict checks, such as rejecting answers over rounding differences. Fixing validators could raise both human and model scores; the expectation that a similar gap would remain had not been tested in a re-evaluation.

Action history, reusable tools and the meaning of “myself”

Action history is not observation history. The described agent receives its entire previous action sequence, but not previous page contents. Adding those pages could preserve useful information while greatly increasing context size. Whether longer context would help this setup remained open; the researcher had not yet seen papers demonstrating an improvement.

Follow-on work was moving toward tasks representative of actual jobs, including software engineering. Proposed directions included synthetic training data, reinforcement learning and better website interfaces. The interface discussion distinguishes two approaches: - Use existing interfaces: a participant recounts Adept’s argument that point-and-click interaction reaches systems without application programming interfaces (APIs), giving it broader coverage. - Build reusable tools: only GitLab among WebArena’s four covered sites reportedly had an API. The researcher proposes creating functions for repeated activities on other sites, such as navigating to the purchases page. The tool-building work was unfinished, but it extended the site-documentation idea into executable actions.

WebArena represents pages through an accessibility tree, a textual representation created for screen readers. Visual WebArena is a follow-up that operates visually. Both require the agent to understand what it can do on a page. Reported failures fall into distinct categories: - Interface understanding: missing that a dropdown is clickable, or failing to connect a desired action to the correct visual element. - Recovery: reaching the wrong page without recognizing the need to go back. - Situational meaning: interpreting an instruction to assign an issue to itself as a request to select the literal username myself, rather than the current user’s account.

Sotopia: social ability can outrun its evaluator

Sotopia evaluates whether models can negotiate and cooperate effectively without causing harm. Its semi-automatically created tasks span negotiation, exchange, competition, collaboration, accommodation and persuasion. In one scenario, two people are cold and only one has a blanket. The owner wants to keep it; the other wants to share it. Each agent receives a persona, background information and a secret to protect.

Conversations are evaluated on goal achievement, believability, social-rule violations and secret disclosure, among other dimensions. Experiments included model–model and model–human conversations. Comparing human and model judgments through their correlation suggested that models were moderately capable at both participating and evaluating, but imperfect at both.

The partner changes measured performance. On Sotopia’s composite scale out of seven, the scores distinguish the evaluated party from its conversational partner: - GPT-4’s score: about 3.3 with another GPT-4, versus approximately 4.8 with a human; the latter was recalled tentatively. - The human’s score: 6.15 with another human and about 5.95 with GPT-4. The proposed explanation is that human partners help repair conversational errors, making GPT-4 more successful and believable.

Follow-up work trained Mistral 7B using two methods: - Behavior cloning: train Mistral to imitate conversations generated by GPT-4. - Self-reinforcement: have Mistral generate conversations, ask GPT-4 to grade them and retain only highly rated conversations for training.

The trained model matched GPT-4 under model grading while remaining far behind under human evaluation. Matching the judge’s score did not establish matching social ability. The researcher interprets this as over-optimization to model judgments and proposes alternating improvements to models with improvements to evaluators. Prompting GPT-4 remained the team’s best available evaluation method; better social-skill evaluators were still in development.

Faster code requires trustworthy tests and timing

The performance-improving code edits project asks whether language models can make programs more efficient. Its dataset pairs competitive-programming submissions where an initial solution timed out and a revision passed. Contestants under time pressure have a reason to preserve much of their implementation, making successive submissions useful examples of related slower and faster programs.

The project combines prompting and fine-tuning with a runtime evaluation harness, described as using virtualized CPUs to address confounds such as system load. Models reportedly made programs faster on average and exceeded the human reference solutions. But the human baseline was optimized to pass a time limit, not to achieve the best possible performance. The result does not establish general superiority over human optimizers.

Performance-conditional generation teaches the model to associate implementations with performance ratings: 1. Prefix training sequences with ratings, illustrated as zero out of ten versus ten out of ten. 2. Fine-tune on slower and faster implementations paired with those ratings. 3. At inference time, request generation using the fastest category’s tag. Low-quality examples remain useful because the model learns what distinguishes them from desirable outputs. The participants present this as a way to gain some benefits of learning from negative examples when only supervised fine-tuning is available.

Quark provides an analogous approach for text: label examples by toxicity, then request the non-toxic category during generation. The researcher reports that its tagged training outperformed training only on high-quality examples, without specifying an effect size.

Applying optimization to ordinary development exposes two constraints: - Isolate the runtime being measured. In Cursor, optimization was described as a user-directed workflow: request improvements and supply timing information. Laptops and remote development machines lack the controlled environment of the research harness. - Test correctness beyond the observed case. With weak coverage, a model can replace computation with a constant return value that passes the test quickly but fails other inputs. Comprehensive tests become harder to construct for real-world code with richer data structures.

Coding agents: useful autonomy still needs intervention

Devin’s demo renewed one participant’s interest in code generation after incremental completion improvements had become less compelling as a research direction. Combining coding with agents offered difficult, measurable problems on SWE-Bench and WebArena. A more skeptical participant expected useful results on simpler pull requests, with human judgment remaining necessary for some time.

Asynchronous work with intervention was a particularly promising aspect of Devin’s interaction design: an agent could execute its plan while the user retained the ability to redirect it. This made supervised autonomy seem more plausible to one participant than complete autonomy.

OpenDevin began with enthusiasm ahead of functionality. A contributor recounts a README-only repository attracting 1,000 GitHub stars, followed by a nonfunctional React interface resembling Devin’s, without chat. Contributors then worked to make it functional; for roughly the first four weeks, the speaker says, nothing actually worked.

Using OpenDevin to resolve its own issues exposed a substantial prerequisite: the agent must first set up the repository. Interest from people who were not developers suggested a potentially nearer-term use case—building simple web apps. Another participant expected better results on new projects than on changes to existing ones, depending on task size. Their hypothetical value calculation was simple: eight hours of agent execution could be worthwhile if it saved an hour of human attention.

Delegation also changes the developer’s work. Reviewing contributions, checking progress and redirecting an agent that has drifted resemble managing junior engineers. One participant welcomed that role: they could inspect the work and ask the agent to re-plan.

Planning, retrieval and what benchmark scores conceal

OpenDevin initially implemented planning, but its best-performing agent at the time reportedly achieved 21% on SWE-Bench Lite without explicit planning. The contributor still considered planning important. Whether it would outperform an agent equipped with effective code-search and editing tools remained an open comparison. The team had not run the full benchmark, citing setup effort and a reported $6,000 cost per GPT-4 run.

Retrieval was an active experiment: OpenDevin was trying Morph while using search based on SWE-Agent. Another participant described a retrieval setup combining embeddings, rerankers and occasional language-server-protocol information, arguing that autonomous agents have stricter requirements for selecting exactly the right context.

Public repositories complicate score interpretation. In one informal test, Claude Opus reportedly identified the correct file to edit from only a pull-request name. The participant suspected pretraining exposure without establishing its source or measuring its effect. They still expected SWE-Bench improvements to help private-repository work, while cautioning that the same absolute success percentage may not transfer.

OpenDevin was expanding into a platform with contributed agents and pluggable models through LiteLLM, its model integration layer. Pluggable evaluations were also planned. Only SWE-Bench was implemented at the time; web-navigation evaluation would test agents’ ability to gather information online.

Two projects offered additional evaluation approaches: - R2E: turns repositories into code-agent evaluation environments. OpenDevin’s team was discussing integration with its creators as part of work toward evaluations less vulnerable to leakage. - LiveCodeBench: groups programming problems by release period, with adjustable date cutoffs for comparing model performance.

Component tests explain failures that aggregate scores hide. Visual WebBench tests about eight web-agent capabilities, including reading text from pages, locating elements and predicting click effects. The coding discussion identifies similarly concrete needs: - Environment navigation: a GPT-4 agent cloned a repository, failed to change into its directory and tried to clone it again. - Reliable edit application: a participant recounts a SWE-Agent demo in which a model appeared to understand the change but applied incorrect indentation repeatedly—reportedly seven times despite feedback. A skills benchmark could expose these weaknesses, though optimizing a fixed checklist could leave other failures unresolved.

Background execution preserves developer attention

An OpenDevin contributor forecasts rapid initial progress from open experimentation, followed by a plateau on tasks beyond the available model backends. A complementary product vision keeps the developer coding while delegating substantial units of work: write a call to a needed helper function, for example, and have its implementation generated in the background.

More computation during generation calls for a different interaction model. More capable models, chains and loops cannot all fit comfortably into continuously supervised autocomplete, streamed diffs or chat. The proposed design moves work into the background while preserving developer flow and control. The speaker expected these tasks to require highly capable remote models; that was a deployment expectation, not a demonstrated requirement.

Does code teach reasoning—or supply more useful data?

The discussion turns from agent systems to their training material. One participant questions whether evidence that code improves reasoning isolates a special property of code, or simply the benefit of adding more data after other sources run out.

A researcher recounts initially misunderstanding the relationship between Code DaVinci 002 and Text DaVinci 002: the text model was fine-tuned from the code model, rather than the reverse. In their experiments, Code DaVinci 002 performed better on some structured-reasoning benchmarks even though Text DaVinci 002 had received more training data. No benchmark names, scores or experimental controls are specified here.

The proposed mechanism is structure and repetition that require reference to earlier context. Code may encourage a model to capture dependencies that also matter for structured reasoning. The researcher treats this as unproved and suggests that similarly structured text could confer comparable benefits; legal text is one candidate. Another participant hypothesizes that textbooks might be more useful still. These are hypotheses about useful training material, not established rankings.

Academic research needs suitable compute—and a way to allocate it

The closing discussion traces academia’s compute squeeze from Google’s 2014 sequence-to-sequence work through BERT and the GPT models. Modeling innovation remained possible, but training large base models increasingly exceeded academic resources. After ChatGPT’s release, an emergency workshop at CMU confronted what researchers should work on next.

Two opportunities helped restore direction: evaluating unresolved reasoning and multistep tasks, and fine-tuning open models. Training tools including DeepSpeed, Llama Factory and Axolotl made useful work more accessible on academic hardware. The researcher estimates that a roughly $100,000 machine could support interesting training runs, without specifying model sizes or configurations.

Cluster access alone does not solve the resource problem. Two requirements remain: - Suitable hardware: participants cite V100-based infrastructure and 40GB A100s as resources that did not meet their particular training needs. - Concentrated allocation: institutions need a way to direct substantial compute toward selected projects, which sits uneasily with universities’ customary support for many independent ideas.

This part ends with possible routes to additional compute. Participants point to Eleuther’s success in collecting compute grants and producing research with them. A suggestion that Andromeda might support research meets a practical question: might existing users already consume its capacity? Neither availability nor saturation is established. Obtaining enough suitable hardware remains an open constraint on the experiments researchers want to pursue.