2024 in Agents: From Executable Actions to Useful Software Work

Three jobs running during a talk

Graham Neubig gives OpenHands, an open-source coding-agent framework, a concrete task: inspect the SWE-Bench repository, plot benchmark scores over time, and label the top three systems for each dataset. SWE-Bench evaluates coding work drawn from real GitHub pull requests. Neubig continues presenting; when he returns, the agent has produced the visualization. The result demonstrates work completed while his attention was elsewhere, though the chart’s rankings remain outputs of the agent’s analysis.

Two more jobs show the range of work he wants to delegate: - Create a tool: Read the Resend.com API documentation and build an email-sending script that accepts a CSV file, with Jinja2 templates for the subject and body. - Extend existing software: Add pie charts and period totals to a dashboard tracking agents’ OpenHands contributions, separated into merged, closed, and open pull requests. Those agents are still working when the visualization arrives. Neubig says he uses coding agents five to ten times daily and gets better results from relatively concrete prompts.

This 2024 retrospective reflects Neubig’s experience as a CMU professor, All Hands AI chief scientist, OpenHands maintainer, and software developer at the time. His starting point is the programmer’s everyday environment: a browser, terminal, filesystem, and editor. Agents that can browse, write code, and execute it successfully can use that small toolset for a broad range of work.

Let the model write the tool-calling program

The agent–computer interface determines which capabilities an agent receives and how it invokes them. Neubig illustrates the choice with a smartphone-price comparison across the USA, Japan, Germany, and India. With granular API tools, the model repeatedly requests information for individual countries and coordinates the results across turns.

OpenHands exposes operations through executable Python, so one program can coordinate many API calls. Execution handles repeated operations without requiring a separate model turn for each one. The agent runs its program, receives results or errors, and repairs its code when necessary. Neubig considers this instrumental to OpenHands’ success. The example illustrates reduced model-call overhead; it supplies no measured latency or cost improvement.

The core tools cover three kinds of action: - Execute programs: Run Bash commands and Jupyter notebook cells. - Inspect and edit files: Read or overwrite file contents and perform global search and replace. - Browse the web: Scroll, type, and click—several actions grouped under one capability. General execution extends this small interface through existing software. Python libraries provide HTTP requests, PDF-to-text conversion, and visualization; command-line tools clone repositories; the GitHub API supplies issue comments and GitHub Actions results.

Make work understandable; choose models that recover

The human–agent interface helps the user understand and direct the work. Neubig considers it unresolved, including in OpenHands. His guiding principle is to describe actions in plain English without filling the default view with large commands or notebooks.

Interaction also needs to fit where users already work: - Chat supports direct exchanges with the agent. - GitHub comments let users invoke it inside an issue or pull request, such as asking it to repair failing tests. - A remote runtime API launches jobs without a chat interface, including agents tackling several separate problems concurrently.

Behind those interfaces, model selection requires several capabilities to work together: - Instruction following: Carry out the requested workflow. - Tool use and coding: Turn instructions into valid operations. - Environment understanding: Interpret the relevant text or visual information. - Error recovery: Recognize failure, diagnose it, and try another strategy. Recovery is especially consequential: a model that repeats a failed action can squander an otherwise capable execution environment.

Claude powers the demos, and the presenter particularly values its willingness to change approach after an error. An evaluation within the coding-agent framework had placed it ahead of the tested alternatives, with Llama 3.1 405B the strongest open-source model tested. That comparison was already several months old at the talk, with no numerical scores or detailed protocol given: a historical, framework-specific assessment.

Plans need structure and room to change

Planning involves two independent choices: - Who supplies the plan: A developer curates a workflow in advance, or the model generates one for the task. - How execution is structured: Code coordinates separate stages or agents, or a prompt directs one agent through the stages. A curated workflow therefore does not require a multi-agent architecture.

For a GitHub issue, the proposed sequence is: 1. Write a test that reproduces the problem. 2. Run it and establish that it fails. 3. Make the fix. 4. Rerun the test and confirm that it passes.

OpenHands was using light planning through ordered instructions in one prompt. Given strong instruction following, one agent can follow a prescribed sequence while retaining flexibility to deviate when necessary. Explicitly coordinating reproducer, test-writing, and bug-fixing agents can make deviations harder. The presenter favors the simpler approach without ruling out more structured architectures.

Plans can also be generated and revised during execution: detect that a plan is failing, then change it. The talk introduces this through a paper whose name is transcribed as “CoAct”; the exact spoken title is uncertain. The companion links separately identify CoAct and CodeAct as distinct resources.

Repeated tasks offer another opportunity: reuse workflows instead of rediscovering them. Repairing failing GitHub Actions is one recurring example. SteP uses manually specified web-navigation workflows; Agent Workflow Memory accumulates workflows from experience.

In Agent Workflow Memory, the agent completes a task, judges whether it succeeded, and decomposes a successful task into constituent workflows. Those workflows enter the prompt for future tasks. The agent’s own success judgment controls what enters memory. The presenter reports a 22.5% increase after 40 examples on WebArena, a benchmark using real open-source website software. No baseline score is given, and the talk does not specify whether the increase is relative or measured in percentage points.

Explore first; backtrack when consequences are reversible

Before solving a task, an agent can learn more about its environment. Two examples put that understanding into its prompt: - Agentless supplies a repository map to help the agent navigate code. - BAGEL has an agent explore a website through random tasks, then incorporates what it learns into the prompt.

During execution, OpenHands follows one linear path, relying on the model to recover and try alternatives along the way. Tree Search for Language Agents instead expands multiple paths, assesses progress, and backtracks from unsuccessful branches. The practical constraint is reversibility: a code edit can be reverted, while an unwanted Amazon order is much harder to rewind. Search strategy depends on the consequences of an action.

Measure basic competence and realistic work

Evaluation serves two purposes: - Fast sanity checks: Mini World of Bits supplies simple web-navigation tasks; the Aider code-editing benchmark tests changes to individual files. - Realistic work: WebArena uses open-source website software for shops, bulletin boards, and similar services. SWE-Bench draws coding tasks from real GitHub pull requests. The presenter identifies a further gap: tasks requiring the same agent to combine coding and web navigation. An evaluation covering that versatility was under development at the time.

Charlie’s episode introduction reports OpenHands at 29% on SWE-Bench Full and 53% on SWE-Bench Verified, alongside OpenAI’s self-reported o3 result of 71.7% on Verified. Benchmark subset and reporting conditions matter: Full and Verified are different subsets, and the o3 result is explicitly self-reported.

Looking ahead from late 2024, the presenter makes three connected forecasts: - More capable, cheaper agent models: Major model trainers would focus on agents by mid-2025, increasing competition and making smaller models competitive. - Less manual workflow engineering: Better instruction following would allow more complex behavior to be specified in prompts; better error correction would reduce repetitive loops. - Harder benchmarks: As agents improved, existing evaluations would become less demanding. He already considered WebArena somewhat too easy because its tasks took humans roughly two minutes.

His account describes an alternating bottleneck: easy benchmarks motivated WebArena and SWE-Bench in 2023; weak agents were the constraint in 2024; benchmark difficulty could become the constraint again in 2025. The final step is a prediction made at the talk.

Human feedback changes what agents can finish

The presenter distinguishes benchmark success from work in his own repositories. Against roughly 53–55% on SWE-Bench Verified, he estimates that agents independently solve 30–40% of the tasks he wants done. With feedback, he estimates that 80–90% can be completed without opening an IDE. These are personal impressions, not a controlled comparison. They motivate a concrete interface goal: make correction and auditing easy for capable, imperfect agents.

That goal becomes more demanding outside software engineering. Lawyers and chemists need ways to direct programming agents without assuming knowledge of code or software architecture.

The computer-facing interface also affects success. The presenter reports substantially higher accuracy when web agents use APIs instead of website interfaces, attributing it to easier interaction, though he gives no numerical results or evaluation conditions. For GitHub tasks, he explicitly directs agents to use the API. He speculates that wider agent use could make APIs necessary for every website.

The prepared talk closes with a goal for that progress: make powerful agents affordable and broadly accessible. Agents already help build agents, which the presenter expects to accelerate development. He argues that open source and inexpensive access can expand what people can do, and invites contributions, research, and strong, inexpensive models for coding agents.

When should an agent ask for help?

An audience member asks why the personal success estimate trails SWE-Bench, and whether agents could proactively request help like a junior developer sending a Slack message. The presenter attributes some of the benchmark gap to possible training-data exposure to popular repositories and individual benchmark issues. He believes it modestly inflates accuracy but supplies no measured effect.

The team is developing a help-seeking evaluation using deliberately vague SWE-Bench issues, such as an unspecified matrix-multiplication problem. The agent must ask questions that establish how to reproduce it. Knowing when to ask is itself a capability: in the presenter’s experience, agents—including Claude—sometimes request unnecessary help and fail to ask when assistance is needed.

Represent the web for interaction or reading

A question about browsing brings out three ways to represent a website to an agent: - Screenshot and coordinates: The model sees the page and clicks a pixel location. Misclicks are a significant weakness at the time of the talk. - Text and elements: The model selects HTML elements, usually through an accessibility tree, a compact representation used by screen readers. - Hybrid representation: The model receives both a screenshot and textual information. The presenter expects this to work best, while his implementation still uses text only because visual support is under development.

Reading a page and acting on it need different representations. The browser supports full interaction with clickable elements, but converts pages to Markdown for information gathering. The presenter finds Markdown more concise and easier for the agent to handle. An audience member proposes an agent-friendly index that preprocesses pages offline; the presenter tentatively agrees that a Markdown index could make sense.

Returning to evaluation, the presenter proposes two improvements for a SWE-Bench successor: - Refresh the issues: Continuously introduce new tasks to reduce training-data leakage, following the updating approach of LiveCodeBench. - Broaden the work: Include documentation and tests that verify intended functionality, alongside issue fixing. Freshness reduces exposure to familiar problems; breadth tests more of software engineering.

Specialize with prompt additions; investigate before fixing

For most tasks, the presenter finds a strong model with general coding and browsing instructions sufficient. OpenHands specializes it with micro-agents: instructions added to the existing prompt when a trigger fires. The implementation described is rudimentary: - GitHub guidance: Detecting the word GitHub adds instructions to use its API instead of browsing. - Noninteractive `npm` use: Guidance prevents interactive terminal behavior that had left the agent waiting roughly two minutes for a timeout. He favors this approach because it is simple and maintainable, while allowing that more complex architectures may eventually help.

The Model Context Protocol (MCP), raised as a standard for agent–computer interactions, prompts a provisional objection: what does another integration add over an existing API? GitHub already has an evolving API and documentation an agent can consult. The presenter sees potential value in MCP servers for separation of concerns and security, but says the team has not yet found much additional value over direct tool access. At this point, he prefers reusing interfaces built for programmers.

The strongest failure diagnosis is acting before gathering enough information. An agent may try to fix a GitHub workflow without first reading and understanding it. The presenter considers this probably the largest failure mode of the team’s agent with Claude. Instructions to investigate first tend to help; otherwise, the user may need to redirect the agent after a failed attempt.

A monitoring-site bug shows the value of broader diagnosis. The program wrote a cache file into one directory and read it from another, while the presenter initially suspected a different part of the code. He asked the agent to generate five possible causes in decreasing order of likelihood and examine all of them. He reports that this worked: the intervention widened the investigation before committing to a fix.

Learn from experience, bound permissions, inspect the result

The closing discussion considers two routes to continuous improvement: - Consult past experience: Let a strong model access its history, possibly filtering failures so it does not overemphasize them. Model size, context cost, and retrieval remain obstacles; the presenter finds retrieval from natural-language queries to code particularly weak. - Fine-tune on successes: Identify successful examples and use them to train the model further. These are proposed approaches, not a demonstrated system with unrestricted access to its history.

A questioner suggests preserving successful workflows in a reusable skill library, drawing on Voyager’s code library and Devin’s playbooks. The presenter agrees that such reuse is possible but remains nontrivial. Agent Workflow Memory offers a concrete mechanism by adding successful experiences back into prompts. Evaluating improvement still requires care: a running average can rise simply by recovering from early failures, without the agent learning.

The final technical question concerns permissions. The presenter favors repository-specific authorization, using GitHub’s fine-grained tokens together with branch protection and approval requirements for changes to the main branch. He argues that many other APIs lack comparable controls and will need adaptation. GitHub provides a starting pattern, not a general solution.

The email-script task requested a CSV input and Jinja2 templates for the subject and body. The presenter reports that the agent wrote a script, but declines to read it to the audience. Whether that script correctly sends email remains unestablished.

Then the talk returns to the dashboard. The presenter reports a submitted pull request, opens a Vercel deployment, and confirms that pie charts are visible. A requested software change has appeared while he was presenting. That visible result closes the opening experiment, although it does not establish the correctness of every requested total, test results, or whether the pull request was merged.