DevDay 2025: From Tool Calls to Production Agents

Apps SDK gives developers control inside ChatGPT

A company could build a ChatGPT plugin and still have little control over how users experienced it. The plugin was essentially a tool the model could call, leaving limited room for the developer’s interaction design or brand. In the DevDay 2025 conversation, Apps SDK addresses that constraint by letting developers shape an app experience inside ChatGPT, while gaining access to its distribution.

The platform rationale is straightforward: OpenAI depends on third-party developers to reach audiences and use cases it cannot serve alone. APIs bring its technology into other products; apps bring those products into ChatGPT. Apps SDK continues the experiments with plugins and GPTs, incorporating feedback about developers’ need to steer the experience.

Underlying Apps SDK is MCP, an open protocol for connecting agents to tools. OpenAI’s adoption followed developers’ growing use of the protocol around the Agents SDK and Responses API, which launched in early March. A guest tentatively recalls MCP support arriving in the Agents SDK that month. Anthropic’s open change process and the protocol’s simplicity made adoption easier; OpenAI also participated in its steering committee.

The Canva integration makes the interface change concrete. Instead of a website containing a chatbot, ChatGPT contains a recognizable app interface. Canva can look like Canva while conversation with it retains the surrounding ChatGPT context. That gives developers a way to shape interaction beyond supplying a callable tool.

AgentKit connects prototyping to deployment

Christina’s onstage demonstration built a live chat app for the DevDay website. She introduces AgentKit as a suite for building, deploying, and optimizing agents, motivated by the difficulty of reaching production confidence: writing prompts, evaluating behavior, and iterating on the results. Its components—the Agents SDK, visual Agent Builder, Connector Registry, ChatKit, and evaluation tools—can be used independently or together. The fine-tuning API sits outside the AgentKit umbrella.

Agent Builder includes nodes for approval, transformation, and state changes. Permission to act is only one kind of human involvement: the launch approval node offered a binary decision about whether an action could proceed, similar to approving an MCP tool call. Christina contrasts that with customer workflows involving substantive human decisions over weeks. Deeper support for those interactions remained a development direction.

Christina describes two paths through the builder: - Model, test, and export. Iterate on workflows and prompts in the canvas, then export to the Agents SDK and run in your own systems, including with other models. - Collaborate and use managed deployment. Bring subject-matter experts into the canvas to provide feedback and help build datasets, then deploy through the platform without managing that infrastructure.

Code export did not yet imply round-tripping. Bringing code changes back into the canvas was a future goal, and running code inside Agent Builder was a possible addition. Visual editing already had a practical rationale: customer workflows required substantial zooming out to see the full canvas and produced long SDK files. Launch templates captured field engineers’ recurring patterns, including customer support, document discovery and comparison, data enrichment, and structured-data Q&A.

Evaluate the whole agent run

Workflow portability remained under consideration, with no definite protocol commitment. The desired outcome was to spare developers separate stateful API integrations for each model provider. A narrower capability had shipped: Evals supported non-OpenAI models. Developers could bring an OpenRouter setup, create evaluations with the dataset tool, run them across providers—including open-source models hosted on Together—and inspect results in one product.

For agents, the unit of evaluation was expanding beyond a prompt and completion. Agent Evals added grading over an entire trace—the recorded trajectory of an Agents SDK run—against what the agent was supposed to accomplish. That matters when a task takes many actions to complete.

The product direction combined two levels of assessment: - Whole-run grading: assess whether the agent accomplished its task across the full trajectory, the main launch addition. - Component-level diagnosis: break the trace into parts that can be measured and optimized individually, often with human involvement—a roadmap direction.

Correctly assessing a twenty-minute task remained a difficult problem. Multimodal evaluation also needed substantial development. Long-trace grading was an initial step toward evaluating extended agent work, with more detailed measurement and optimization still ahead.

Optimize prompts without creating model variants

The optimization discussion challenges the expectation that prompting would fade as models improved. The participants argue that it had become more entrenched, with active research making it more useful. GEPA, a prompt-optimization research approach, was cited as interesting work the team hoped to incorporate; the product was not using it at the time.

One participant calls prompt optimization zero-gradient updating: changing effective model behavior by changing prompts, without updating model weights. Its operational attraction is avoiding the burden of managing and serving many fine-tuned snapshots. The argument favors iterative prompt improvement as a practical option, without claiming a measured gain or establishing superiority over fine-tuning.

Tinker addresses the infrastructure burden for researchers who do want to fine-tune. A participant recalls John Schulman’s longstanding interest in a low-level research fine-tuning library while at OpenAI. Tinker’s appeal is described as a compact API—summarized as four functions—that lets researchers work without managing the underlying infrastructure.

Choose where deterministic control belongs

Returning to Agent Builder, a central design question is which decisions belong in deterministic control flow and which should be model-driven. The team wants developers to model existing workflows and identify common operations the available nodes cannot express. That grounds requests for new building blocks in actual tasks.

Several boundaries defined the initial release: - Explicit conditions: if-else nodes used Common Expression Language to express variables and conditional statements. Natural-language conditions were a questioner’s suggestion, not a confirmed roadmap item. - Chat-based deployment: Agent Builder initially deployed through ChatKit. Standalone workflow runs were planned, and voice was a frequently requested modality.

State storage prompted a familiar developer-tools joke: anything that stores state eventually becomes a database. A respondent discouraged using the platform for bulk storage because of the strain on the team and infrastructure. The state store was not presented as a general-purpose database, and no concrete capacity limit was given.

Connectors trade integration work for retrieval control

The connector discussion starts with the burden of authenticating Gmail, Calendar, and Drive separately. The registry’s proposed direction was to help companies manage developer access, connector configurations, and private internal connections across direct integrations and third-party MCP servers. It did not establish a shared-authentication solution.

The two connector approaches offer different tradeoffs: - First-party sync connectors keep data on OpenAI’s side. This enables retrieval work such as reranking results and indexing data in a vector store, at the cost of substantial integration and state management. A guest argues that this control can improve quality, without quantifying the benefit. - Third-party MCP connectors cover the long tail of services. Their usefulness depends on how well the creator’s API design works with the models. MCP standardizes connectivity, but leaves that design dependency in place.

A questioner imagines carrying connected MCP services into another app by signing in with ChatGPT. That remained a vision. Asked whether OpenAI would become an identity provider, a respondent said there were no plans at the time. Apple’s integration offered a limited existing example: ChatGPT account and subscription information could affect the model used when Siri handed requests to ChatGPT.

Sign-in also raises a billing question: could developers fund an initial allowance, then let users pay for further inference through personal API keys? AgentKit did not offer bring-your-own-key support out of the box. A respondent acknowledged demand and described copy-and-paste key workarounds as security hazards, without committing to a solution. Lower prices alone might not relieve the burden: the guests observed that cheaper inference could stimulate enough additional usage to keep total spending substantial.

ChatKit keeps the interface current

AgentKit serves both internal and external applications. The team repeatedly saw companies build systems for managing prompts and sharing agent templates across developers and product areas. Centralizing that infrastructure would let them focus on business-specific behavior. A participant reports that OpenAI’s own customer support already ran on AgentKit, alongside other internal uses.

ChatKit supplies the embeddable chat interface, with a design goal of a polished, responsive experience resembling ChatGPT. Its widgets visually represent actions as well as supporting ordinary chat. Feedback from customers including Ramp helped set the presentation standards for public-facing use.

ChatKit’s hosted iframe lets the platform update common interface behavior centrally. Support for new reasoning behavior or modalities can arrive without every developer rebuilding a frontend or pulling in new components. The comparison is Stripe Elements and Checkout: maintain shared interface infrastructure so businesses can concentrate on their specific behavior.

That architecture also shaped the open-source discussion. The widget language and design system were possible candidates, but the team made no commitment and emphasized the value of a continuously updated, opinionated hosted interface. Widget creation already supported AI assistance, and a playground previewed ChatKit customizations. A wider ecosystem for sharing widgets remained an idea without a specified distribution mechanism.

Codex: delegate larger tasks, retain approval

The development advice centers on giving Codex larger assignments and testing its limits. One participant credits interns with demonstrating the value of asking for entire features. Their estimate of 30–40% success in one shot was anecdotal, with failures acknowledged. They had not tried that workflow with GPT-5 Codex; better results from that model were speculation.

Codex reviewed code but could not approve pull requests, including those containing its own output. Another participant says the team relied heavily on its reviews and credits it with making the visual builder’s delivery possible after work began probably less than two months earlier. Generated code still received checking and editing.

The benefit could survive rejection of the patch. Team members reportedly started about five Codex tasks during a thirty-minute bus commute. The outputs helped them locate relevant files and understand the codebase, even when they discarded the pull request and implemented the change themselves. Others launched tasks before one-on-one meetings so work continued asynchronously. An agent run can help an engineer get oriented even when its code never merges.

Measure the service your organization receives

The closing launch is the Service Health Dashboard, which exposes health information for an organization’s own OpenAI API integration. It tracks responses, response codes, token velocity, tokens per minute, and throughput. Its practical contribution is visibility into the service that a particular integration is receiving.

A guest connects the dashboard to reliability work after a severe outage the previous December, recalled as roughly three to four hours. The team had become confident enough to expose organization-specific health through a product. Five-nines reliability remained a goal, with no measured availability figure or observation window supplied.

Asked what another nine requires, the respondent describes each improvement as dramatically more work, recalling successive reliability targets over the preceding years. When another participant cites six billion tokens per minute, the respondent says throughput had already surpassed it, without supplying an updated rate or measurement conditions. The final engineering challenge is making reliability observable as demand continues to grow.