Building AGI in Real Time — Part 2 of 3

Making a cheaper model good enough

An application needs low latency and low cost while retaining useful quality. Early in this DevDay 2024 conversation, an attendee describes a distillation demonstration that moved from GPT-4o to a smaller model, transcribed as “4 mini,” with roughly a 2% performance loss and fifteenfold lower cost. The metric, evaluation dataset and cost calculation are unspecified. The practical question is how much quality a particular application can trade for cheaper inference.

Distillation is presented as a less anticipated release than speech-to-speech interaction. It fits a developer-driven roadmap: prompt caching, prompt generation in the playground and vision fine-tuning respond to application builders’ requests. The invitation for feedback extends down to endpoint details, while DevDay’s more intimate format and community talks give developers’ work greater prominence.

Voice becomes an application interface

Simon Willison offers a quick example of turning model output into a working tool: while waiting for DevDay’s first talk, he used GPT-4o to write the JavaScript for a live blog, got it online and used it throughout the day. His coding workflow still combined Copilot with copying material into Claude and ChatGPT; he had not spent enough time with Cursor for it to click.

The Realtime API introduces a different programming contract: a WebSocket connection carries streams in both directions. Described as OpenAI’s first WebSocket API, it required decisions about event types, names, timing and what to send on connection versus in later messages. An internal hackathon made the interface tangible with voice-controlled Vim: asking the model to write a file caused it to enter the necessary editor commands.

Willison’s standout demo is a conversational WebGL solar system. Its significance is speech connected to function calls and on-screen content: the user can talk, the application can act and the screen can respond. A small voice interface to a database now feels like a plausible weekend project to him. He asks for the demo’s code; another participant offers to discuss releasing it, without confirming a release.

Browser access leaves an authorization problem

Browser JavaScript could connect directly to the API, but embedding the developer’s API key exposes it to users. Hiding it required a WebSocket proxy, prompting a request for OpenAI to provide that infrastructure. Partner solutions, including LiveKit, are mentioned while the API remains in beta.

The discussion distinguishes three arrangements: - Proxy the developer’s key: protect it on a server, at the cost of additional infrastructure. - Bring your own key: keep the user’s key in browser local storage, away from the developer’s server. The user still has to trust future JavaScript updates not to steal it. - Delegate bounded spending: a proposed sign-in flow would return a token allowing the application to spend, for example, up to $4 on the user’s behalf, removing the need to copy and paste keys.

Delegated spending is a request, not an announced capability. The response attributes the gap to prioritization and limited team capacity. Sign-in with OpenAI is under consideration, without a launch commitment.

Five chickens, and why valid output can still be wrong

Vision fine-tuning is singled out as an underrated release. The respondent describes demand for extracting text from specialized formats such as government documents, and reports improvements on bounding-box tasks. The acknowledged weak baseline makes the scope important: tuning is intended to improve a particular use case. The suggestion to start with 100 images is not an accuracy guarantee.

A proposed experiment makes the task concrete: distinguish five individual chickens, including three gray birds. A developer had already supplied Claude with reference photographs, but it still confused the gray chickens. Fine-tuning is the planned next experiment, with those confusions providing clear cases to test.

That leads to evals, tests of whether a model works well enough for an application. The evals product, launched within the distillation announcement, aims to make that judgment easier. A question about function calling exposes two separate checks: - Schema validity: does the response match the expected JSON structure? Schema checking is confirmed, and structured-output guarantees depend on the model. - Functional correctness: did the model choose the right function and produce the right result? A response can satisfy the schema while failing either requirement. Product support for testing this behavior remains unconfirmed in the exchange.

Choose the audio interface for the workload

The API team’s approach is to expose basic capabilities, then add abstractions as their usefulness becomes clear. Audio illustrates the division: - Request-response audio: planned Chat Completions support would accept audio and return raw audio, fitting asynchronous work. - Interactive voice: building Advanced Voice Mode exposed the difficulty of using Chat Completions for conversation. WebSockets fit bidirectional streaming. - Client conveniences: libraries and higher-level interfaces could follow developer testing, once the team understood what they should simplify.

For a half-hour recording, a developer would rather submit an MP3 URL than slice audio into snippets and send them over a WebSocket. The respondent says that capability is planned, tentatively for the weeks following DevDay, but cannot supply the input-length limit. Support for hour-long recordings remains unresolved; the developer currently uses Whisper to transcribe YouTube videos and wants to process the recordings multimodally.

A separate idea fits streaming naturally: pipe a Unix command’s output into the API and have it spoken aloud as it arrives. This is a proposed utility, not a completed implementation.

Video is unavailable in the API under discussion. One proposal would approximate it with an image every second, conditional on image input becoming available. The message-size limit and feasibility of sending 1080p frames remain unknown. A participant reports success giving Gemini an hour of video represented as frames, though the sampling rate is unclear; whether GPT-4o could handle a similar workload is still an open question.

Let the provider handle accounting and execution

The next request is operational: track spending by application user and enforce a limit. A developer wants to query whether someone has spent 30 cents and cut them off at a dollar, without maintaining token accounting. The response acknowledges the request without promising delivery; priorities change quickly, with adoption blockers receiving particular attention.

Hosted tools expose another tension. The assistant-and-thread API provides useful capabilities but requires numerous setup requests, including creating an assistant and a thread. Its formats are still being reconsidered, and no update is announced. One participant uses it solely for Code Interpreter, preferring hosted execution because sandboxing is difficult, and requests access over WebSockets.

The discussion closes with ways participants have extended Code Interpreter: uploading pip packages, compiling C code and running custom C extensions inside Python. These are reported techniques; the package-upload route does not establish unrestricted installation by every method.

Genie’s advantage is the training signal

Alistair Pullen of Cosine turns the conversation toward Genie, its software-engineering model, and the datasets used to demonstrate desired behavior. His central claim is that domain-specific reasoning traces are Cosine’s competitive advantage. The pipeline generates human-like sequences of reasoning intended to teach the model to approach problems like a software engineer. He defends keeping those traces private because they reveal how the system works.

Pullen sees two uses for the stronger reasoning capabilities of o1: - Improve training data now: Cosine has begun replacing some traces with ones generated by, or in tandem with, o1. Pullen reports improvements without giving a magnitude or evaluation setup. - Train a stronger model later: he anticipates fine-tuning o1 on Cosine’s specialized traces. Another participant questions whether OpenAI will permit that, leaving a key dependency unresolved.

In a comparison framed around the software-engineering benchmark SWE-bench Verified, Pullen endorses the claim that Cosine’s specialized GPT-4o system outperformed out-of-the-box o1. He interprets this as evidence that custom reasoning adds useful signal beyond general reasoning ability. No scores or matched evaluation conditions accompany the comparison, and a host suggests that giving o1 more runtime could change it.

Pullen’s broader advice is to build a business that benefits when foundation models improve. He reports gains across Cosine’s transitions from GPT-3.5 to GPT-4, GPT-4 Turbo to GPT-4o, and GPT-4o to o1.

The team also wants to fine-tune a 405B model on the same dataset with the same context-window length. The experiment has not run: shipping, onboarding and DevDay took priority, and the team is seeking help with setup. An open-source model could also serve customers whose private cloud environments permit that deployment but cannot accommodate an OpenAI model.

Shipping reveals what the benchmark misses

Genie’s users complicate the benchmark story. The team finds SWE-bench a poor proxy for overall competence, particularly in React development: generated code does not establish that the intended UI appeared. Cosine had collected linked images and graphical resources alongside its original dataset but had not used them. Vision fine-tuning offers two prospective applications: - Evaluate the UI result: judge whether generated output achieved the intended interface. - Learn the image-to-code relationship: train on pairs of a desired UI and the code implementing it.

Pullen welcomes distillation tooling as a first step toward replacing internal infrastructure for model and dataset lineage. Maintaining that machinery is a burden, not Cosine’s advantage. Yet external operations products had not justified their cost for this team: a script built in a day or two and attached to an existing internal UI often sufficed. Built-in tooling can be welcome without justifying a separate purchase.

Voice receives a similar prioritization judgment. Talking through a problem with Genie could fit the ambition of an AI software-engineering colleague, especially during ideation, but remains a future nice-to-have.

The interview ends with a concrete mismatch between training data and usage. Genie’s initial mix emphasized JavaScript and Python. Early users reported worse results on a C# codebase than on Python; Pullen estimates C# represented about 3% of the data. The team correlates the weaker experience with that composition and reports improvements from incremental fine-tunes adding language- or domain-specific data, without quantitative results.

The five-person team wants its data mix to reflect what users actually do, rather than anticipated needs. Pullen also describes a future option to adapt Genie to a customer’s codebase history, offered outside the base pricing tier.

From reasoners to agents

The closing Q&A with Sam Altman and Kevin Weil turns to artificial general intelligence, or AGI. Altman prefers five capability levels to that increasingly overloaded label: chatbots, reasoners, agents, innovators and organizations. He places o1 at the reasoner level and predicts that adding agent capabilities will make systems feel substantially more capable in the near future, while still falling short of what many people would call AGI.

Accelerating scientific discovery is an important part of Altman’s conception of AGI. He is less certain about that step than about agents, while predicting steep progress over the one to two years following this 2024 discussion. He expects a blurry transition along a smooth exponential trajectory, with retrospective disagreement about when the threshold was crossed. He invokes the Turing test as a once-apparently-clear milestone that became fuzzy and lost salience as capabilities advanced.

Superior AI-research capability could be a meaningful milestone, Altman argues, even if it remains part of that continuous progression. He describes a change in OpenAI’s emphasis: scaling compute when that appeared most effective, and now pushing research again. He characterizes o1 as a breakthrough assembled from multiple approaches pursued over a long period.

His organizational distinction is discovering new paradigms versus incrementally improving known ones. He presents sustaining frontier research alongside product development as OpenAI’s strength, and predicts that a few more major breakthroughs may be enough to reach AGI.

Product roadmaps depend on uncertain capabilities

Weil explains the product consequence: teams still need to understand users and solve their problems, but the capabilities available to them change every two or three months. His hypothetical comparison is a capability reaching 90% versus 99% accuracy. The reliability threshold changes which product is feasible, yet even internal teams cannot reliably predict when it will be reached.

The ensuing exchange describes a willingness to redirect product and research priorities when results overturn forecasts. An enterprise anecdote captures the planning tension: a customer asks for 60 days’ advance notice of launches, and the speaker would like that much notice too.

Part 2 ends as the Q&A turns to alignment. The response emphasizes making capable models progressively safer, with an approach informed by experience. The speaker says the strawberry/o1 paradigm worked differently from expectations a few years earlier. The recording cuts off before the explanation develops its safety implications or describes concrete methods.