How Computer-Use Agents See, Act and Recover

From screenshots to a faster feedback loop

Ari Weinstein describes a meal-prep order that took him two hours to configure manually: he says a computer-use agent later completed the task in 15 minutes. The technical question behind it is more general: how does an agent move from seeing an interface to carrying out a sequence of actions, notice a failure and try again?

At OpenAI DevDay, Weinstein, who leads product and engineering for computer-use agents, and API product lead Nikunj Handa describe two connected pieces of the stack. Weinstein discusses how an agent operates software; Handa explains the tools for coordinating long-running work and making small decisions quickly.

Weinstein says the change he notices most since his earlier work at Sky is recovery: older agents could start a task but struggled when the workflow went wrong; newer models, in his experience, are better at debugging and retrying. He attributes progress to both model capability and the surrounding harness, and says evaluation configurations vary with the safety checks used.

What the agent can see and do

A screenshot-only loop can require repeated steps: look at the visible page, act or scroll, take another screenshot, and look again. Weinstein says the agent can instead combine several representations, selecting the one useful for the task:

- Screenshots show rendered visual state. - Accessibility information exposes structured interface text and controls. - The DOM and Playwright give browser-oriented access to page structure and actions. - Generated JavaScript can execute several actions together instead of waiting for a separate model round trip after each one.

For a long results page, access to page structure can reduce repeated scrolling and visual inspection; code can then carry out multiple steps in a batch. The agent still has to determine what to do and inspect whether it worked. Weinstein presents richer observations, code execution and smaller engineering improvements as contributors to speed—not a single isolated model change.

App Shots illustrate why a picture and an interface representation carry different information. Weinstein says an app shot can bring a view and accessibility text into a conversation. A screenshot alone may show a link without its destination, or a calendar title cut short; additional accessible context can give the model information that the visible pixels omit. What is available depends on the application and its exposed information.

Even if inference gets faster, the application itself can dominate elapsed time. Weinstein gives a website loading as an example: the system should resume promptly after the operation finishes. Browser load events can help, but he says not every relevant event is available that way. Waiting for a customer-service response illustrates a different delay, with no predictable completion time.

Cloud computers and the trust boundary

Weinstein says each Dot has access to a Linux virtual computer in the cloud, able to run desktop applications and a browser. That gives delegated work an execution environment distinct from a user’s own machine. He encourages people to start with tasks they already spend time on, while the meal-order example shows what he found useful personally.

For developers, Weinstein says the Agents API exposes the computer-use implementation used in OpenAI products. He suggests that using a harness on which the model was trained might help with speed, cost or accuracy.

Operating an interface also creates authority questions. Weinstein recommends building reliability checks, asking for consent before consequential actions such as payments, and limiting access to the sites or applications a task needs.

One application Weinstein highlights is testing software the agent just built. Instead of stopping at generated code, the agent can operate the application and inspect the result. Swyx describes his visual-playtest workflow as a way to catch design problems that a code-only inspection may miss. Both describe useful feedback loops; neither establishes that agent testing replaces all human or automated review.

Let tools run while the agent continues

A slow tool need not freeze an entire interaction. Handa describes asynchronous function calling as starting a tool, continuing model execution and checking back when the result is available. Mid-turn steering lets an application add instructions while work is in progress—for example, when a tool result arrives. The implementation still has to decide what the agent can safely do before that result is known.

Handa says WebSockets provide bidirectional communication for these interactions, reducing the overhead of repeatedly going back and forth as the agent calls tools. He distinguishes this GPT-6 API discussion from GPT Live, the separate real-time voice API.

Fast decisions are a different workload

Handa says the Decisions API effort was inspired by Jev and began roughly a week before the interview. At the time he described a prototype and an intended release after a latency target was met. OpenAI’s user-operations team was interested in classifying support tickets; he also mentioned internal demonstrations combining fast tool calls with GPT Live.

The first implementation, Handa says, used existing Luna weights rather than a newly trained decision model. It constrained the output format, optimized the inference stack for fast decisions, and ran multiple questions in parallel as a batch. Weinstein describes the smaller, non-reasoning approach as less suited to sophisticated long-horizon tasks.

That division matters for computer use. Handa contrasts a stronger model writing a JavaScript sequence with a smaller model choosing an action at a time; the latter may be adequate for some tasks without matching the former’s capability. In the interview, confidence calibration and further model improvements remain open questions, rather than demonstrated properties of the first version.

Keeping long-running threads usable

Handa describes two performance concerns for long-lived agent conversations: latency and repeated context. Prompt caching can reuse prior prompt work; his team was also offering a 12-hour cache window to one user in preview, with a higher write cost. He says that longer window was not yet public. Pre-warming lets a developer prepare a known prompt before it is needed, with the discussed window of 30 minutes.

Caching does not remove the need to manage context size. Handa says the Agents API includes compaction in its harness, while Responses API users can choose a token threshold for server-side compaction or invoke compaction manually. He also mentions file-based approaches under experimentation. The practical choice is where the developer wants control over when a conversation is reduced.

The larger platform question remains open. Handa asks how much storage, memory and harness behavior an API should provide, and how much flexibility should remain with the application developer. Swyx compares this search for higher-level AI primitives to building an AI cloud.