Building AGI in Real Time: OpenAI DevDay 2024 — Part 1 of 3

A voice assistant needs access to the world

A travel assistant asked about restaurants near Fort Mason needs information beyond what its model already knows. In the opening recap of Latent Space’s DevDay 2024 recording, function calling provides that access: the model can invoke external capabilities, such as a database lookup. The Realtime API combines those capabilities with persistent WebSocket connections and voice conversations that accommodate interruptions.

Because attendees were asked not to record the keynotes, Charlie introduces a generated NotebookLM Deep Dive in their place. The opening announcement recap is therefore a secondary summary, distinct from the interviews that follow.

The recap briefly covers a reported move toward a for-profit structure and departures from senior research and technology roles. It mentions speculation connecting those events without establishing a causal relationship.

The technical announcements address different development needs: - Fine-tuning customizes a pretrained model for a job. Company communications and brand guidelines are suggested training material. Vision fine-tuning adds images; the medical-image example is hypothetical decision support, without clinical validation. - Automatic Prompt Caching discounts repeated prompts. The recap gives no matching rules, eligibility conditions, or pricing. This is not evidence of reduced fine-tuning costs. - Distillation produces a smaller, more efficient model from a larger one. It can lower resource requirements, but the recap’s suggestion of unchanged capability comes without a task scope or evaluation.

The recap presents o1 as a stronger problem solver with longer response times, describing reinforcement-learning training and demonstrations of advanced mathematics and coding that GPT-4 struggled with. Those comparisons are qualitative. At the time of this October 2024 discussion, system prompts, function calling, and structured outputs—application-friendly formats such as JSON—are described as planned additions to o1.

The recap closes by attributing to Sam Altman a desire to move beyond the term artificial general intelligence, or AGI, because it had become a poorly understood buzzword. Its emphasis on continual improvement does not establish that OpenAI had abandoned the goal.

Ordering 400 strawberries with a session and a prompt

The recorded demonstration gives the voice agent a concrete constraint: order 400 strawberries for less than $1,500, delivered to the venue. The assistant calls the seller, asks about flavors, requests a price, and agrees to order chocolate-covered strawberries after a quote of approximately $1,415.92. The exchange specifies Gateway Pavilion at Fort Mason and cash payment. The seller’s offer of delivery in roughly 37 seconds belongs to the staged dialogue; it is not a measured fulfillment result.

Ilan, introduced as the demo builder, had wanted an AI that could make phone calls for months. Experiments after the internal API launch became an internal demo and then a stage presentation. He distinguishes two engineering responsibilities: audio streams were difficult in their own right, while handling function calls felt familiar from Assistants or Chat Completions streaming.

Realtime exposed events for function-call start, argument start, incremental argument deltas, and completion. Although their names differed from Chat Completions events, their roles were similar. Ilan used the event containing the complete function call, avoiding the need to assemble arguments from incremental events.

The demonstration also had a consent boundary: Ilan had agreed to receive the AI call while playing the seller. He cautioned against calling arbitrary recipients. His suggestion that businesses might welcome calls bringing orders was tentative, and his comments about legal leeway were explicitly uncertain.

The session sustained the conversation; the prompt supplied the workflow. Ilan used neither a state machine nor a separate application loop to keep the conversation going. Speech turns triggered responses, and his implementation explicitly requested another generation after each function call.

The prompt’s list of steps acted as a soft routine: the model generally followed it, but transitions were not enforced by application code. Prompting guided both questions about missing information and function invocation. Ilan also said the system prompt could change during a conversation, although he had experimented little with that capability.

Voice input does not require spoken output

The next exchange separates Realtime from a voice-only interface. Images and video are suggested as possible future additions, based on ChatGPT demonstrations, but without a known timeline. The existing interaction choices already support different product designs: - Text in, text out: supported, though the speaker did not expect substantial latency gains. - Speech in, function calls out: suppress audio responses and require function calls for voice-driven interface commands. - Speech in, speech out: useful for conversation, but not necessarily desirable for every command.

Requiring function output narrows the model’s responsibility to issuing commands, without maintaining a spoken persona. The speaker describes this architecture as highly reliable, without quantitative evaluation. Both interlocutors favor the same interaction principle: let people speak without forcing them to listen to a reply.

A brief return to distillation offers a task-specific account: a speaker reports distilling GPT-4 into GPT-3.5 and obtaining roughly comparable function-calling performance with hundreds of functions. No dataset or accuracy figures accompany the claim, and the speaker defers details of the newly announced distillation product to its team.

The demo credits introduce Wanderlust, the application whose voice and phone-calling components built on an original project by Simon and Karolis.

Latency, automatic caching, and a pit of success

Olivier Godement explains DevDay’s return and expansion to San Francisco, Singapore, and London as mutual demand for more contact between OpenAI and developers. Developers, he argues, turn low-level capabilities into applications. He also rejects speculation that the October 1 date was connected to the o1 name.

Conversational latency shaped Realtime’s transport choice. OpenAI’s first WebSocket-based API required both a new developer experience and adaptation of safety mitigations to live audio input and output. The design worked backward from tests suggesting roughly 1,500 milliseconds for human conversational responses, then selected WebSockets after considering alternatives. That number is a design reference, not a measured API latency; no test protocol or latency breakdown is given.

Other product choices sought to remove setup work: - Caching with zero code changes: automatic prompt caching reduces the developer’s need to manage prompt prefixes and cache lifetimes. - A pit of success for model improvement: distillation and evaluation workflows aim to make a useful result reachable in few steps. This matters because fine-tuning can produce a poor model, while a badly designed evaluation can tell a misleading story about it.

Vision fine-tuning exposes the limits of simplifying the workflow. An interviewer asks whether improving stop-sign recognition might reduce other capabilities. The response reports larger gains from several weeks of vision fine-tuning alpha tests than from text fine-tuning, but supplies no numerical comparison and leaves the regression question unanswered. Text-and-image applications need additional evaluation work because correctness can be harder to judge.

An improvement loop needs trustworthy evaluations

The interviewer sees distillation and evaluations as a move from model provider to platform provider. The longer-term vision is continuous fine-tuning checked against a developer’s evaluation set: accumulated usage data would support updates, reducing manual experimentation with each new model snapshot. This remains explicitly far from realization. OpenAI’s role as a store of developer data is something it must earn through useful services.

Subsidized evaluations in exchange for shared evaluation data address two needs: - Affordable testing for developers: running o1 across thousands of samples can be expensive. - Regression checks on real applications for OpenAI: developer evaluations help test whether new snapshots damage tasks people depend on. Regression-free releases are the aspiration, not a guarantee.

The interview states that OpenAI’s default remained no training on API data without opt-in, and that sharing evaluation data was voluntary. It also describes sanitizing personally identifiable information from shared data to focus evaluation on real use cases.

The evaluation itself can be wrong. Experience with conventional software testing does not remove the difficulty of judging large language model behavior and its corner cases. The initial evaluation product prioritized ease of getting started. An API for existing test data was discussed as a possible extension, without establishing support for executing user-provided evaluation code.

The interviewer’s comparison to an AWS-like AI cloud leads to a more specific tooling boundary. Model quality alone cannot give developers confidence in an application. OpenAI’s emphasis is on fine-tuning, distillation, and evaluation close to the model, where the tools can form an improvement loop. Elaborate prompt management and more distant operational tooling offer less obvious advantage, and the company cannot build everything.

Audio as a primary interface, with controllable refusals

The voice discussion proposes audio as a primary application experience: speech can carry emotion and intent naturally. Frequent WhatsApp voice notes from friends in Europe provide a personal example of its appeal. The expectation that multimodality will move substantial application traffic toward realtime APIs remains an adoption prediction.

Excessive refusals are a product obstacle. An interviewer names them as a reason to avoid OpenAI Voice, and the response agrees that it was overpoliced. Writing a Hollywood script containing profanity illustrates a legitimate request that should not be blocked simply for vulgar language.

The proposed controls distinguish two responsibilities: - Persistent prohibitions: illegal behavior or behavior violating the terms would remain forbidden. - Developer-configurable treatment of some legal themes: applications could specify acceptable sensitive content and adjust refusal thresholds. This is a future direction, without an available API or delivery date.

Participants also raise blocked singing as a limitation, without receiving an explanation or commitment to enable it. The discussion then turns to feedback on reasoning, multimodality, and tool use. The request is concrete: build applications slightly beyond current model capabilities. Prototypes that work intermittently can expose specific gaps and give future releases useful targets.

Using o1 to plan how an application fits together

Romain Huet argues that developers invent applications and form factors OpenAI cannot build itself. He reports more than three million developers building on OpenAI at the time. His DevDay demos—an International Space Station tracker and a drone-command application—combined planning with o1 in ChatGPT and changes applied through Cursor.

For the iPhone application, Huet used o1-preview as a brainstorming and architecture partner, asking it to consolidate files before transferring the code into Xcode. He says it built on the first attempt and describes realizing the concept in roughly a minute—an account of this demo, not a general success rate. ChatGPT ran alongside Xcode because the integration was not deep. API access also allowed use through Cursor and software-engineering agents such as Cognition’s Devin.

An interviewer questions whether a beginner would know how to set up Xcode or where to paste the files. Huet’s explanation of o1’s value centers on planning dependencies together: consulting API documentation, fetching data, parsing JSON, choosing storage, and connecting components. He acknowledges that GPT-4 can do some of the same work, but characterizes it as more inclined to begin emitting code.

Huet recommends trying both o1 variants: - `o1-mini` for narrowly specified technical work: he describes it as strong in mathematics and coding. His second demo connected a frontend, backend, UDP packets, and WebSockets; his assessment that it worked perfectly applies to that task. - `o1-preview` when broader knowledge matters: he favors it for brainstorming or scientific work needing general knowledge alongside reasoning.

From Wanderlust to client and server integration

Huet returns to Wanderlust’s origin at the previous year’s DevDay. Travel naturally combined pictures, locations, and potential translation needs. The original text assistant returned in 2024 with realtime conversation and tool use, placing a phone call by connecting its AI agent to the Twilio API.

He suggests smarter customer-service centers and emergency dispatch as possible applications, not reported deployments. A concrete interaction goal is removing unnecessary phone-menu steps: a caller who starts speaking Spanish should receive an appropriate response without first selecting a language.

The closing technical question asks about streaming function calls and delivery guarantees. The response turns to integration choices without resolving exactly-once delivery. Realtime was shipping as a beta with tools and function calling. For robust client-side voice at scale, the recommendation is to consider a WebRTC client through partners such as LiveKit and Agora rather than work directly with sockets.

Sample code, reference clients, and native partner integrations provide starting points. Client-side and server-side voice applications have different integration needs, so the choice depends on where the interaction runs. The exchange does not settle whether to use open-source LiveKit directly or LiveKit Cloud. Part 1 closes with support for both client and server applications, and developer feedback positioned as critical to the beta’s evolution.