OpenAI DevDay, Part 1 of 4: Managed State, Shared Context, and the Cost of Control

Managed threads simplify requests, not necessarily bills

Alex Volkov had repeatedly built conversation management in TypeScript and Python: associate a conversation with a user, retain its messages, and supply history for the next response. OpenAI’s new Assistants API promised to take over that bookkeeping through managed threads. But sending only the latest message raised an immediate cost question. According to guidance Volkov obtained from OpenAI staff at DevDay, existing context remained billable.

That combination of enthusiasm and implementation questions runs through the Latent Space DevDay conversation, introduced by Swyx as an oral history of developers processing announcements in real time. Simon Willison expects the community to need months to absorb the releases and finds the initial products impressive but unfinished. Alessio sees an opening for founders: the announced GPT experiences center on synchronous, one-to-one chat, leaving room for applications that work in the background. That is his interpretation of the product direction, not an OpenAI roadmap commitment.

Volkov separates the model from the service around it. GPT-4 Turbo brings a stated 128K context window—the material available to the model for a response—and lower prices; he has not tested its anticipated speed improvement. Managed threads belong to the API layer. They reduce bookkeeping, but advance cost estimation becomes less direct: the staff guidance he reports is to retrieve the thread through another API call and count its tokens locally.

Persistent history still meets a finite context window. Willison asks whether long threads are summarized, leaving that unconfirmed. Swyx relays documentation saying messages exceeding the model’s context window are truncated to fit. The engineering question becomes who decides what to omit. Swyx wants control over that decision; Volkov has heard similar reservations from developers with custom retrieval and content-understanding implementations.

Document extraction needs predictable retrieval and output

Managed retrieval raises the same control problem. Retrieval-augmented generation, or RAG, uses material retrieved from uploaded documents to help answer questions. Participants object that OpenAI has not documented how it divides documents into chunks or retrieves them. Those choices matter when deciding how to build on the service, leaving room for frameworks such as LangChain and LlamaIndex that offer more customization.

The appeal is nevertheless concrete. The onstage travel demo extracts information from a United flight PDF and Airbnb material, reducing work that previously involved both parsing documents and deciding what to extract. A participant recalls a promise to open-source the travel-planning demo, including its map and JSON output, but does not confirm that the code is available.

Similar configuration screens obscure an important product boundary: - GPTs inside ChatGPT are configurable experiences created through the user interface, associated with the announced marketplace. - Assistants through the API provide application components with conversation threads, assistant runs, document uploads, and external function calls. A participant relays OpenAI staff guidance that API-created assistants cannot appear in the GPT marketplace and that the API cannot create GPTs.

For document extraction, “structured output” means JSON here, rather than citations. The newly announced JSON option addresses a need previously approached through function calling, though the participant introducing it has not tried it. Another draws the crucial distinction: valid JSON can still violate the requested schema. A response can pass a JSON parser while containing keys the application never requested. Syntax validity does not guarantee conformity to the application’s data contract.

Custom functions attached to an assistant execute in the developer’s environment and can call external systems. A participant argues that bundling instructions with a deliberate tool set should be easier to make reliable than letting users mix arbitrary plugins: the system prompt can explain how to use those specific tools, while JSON schemas describe the API calls. They interpret actions as the likely successor to plugins, without claiming that OpenAI has announced deprecation.

One upload supports retrieval and code execution

Uploaded files can serve two distinct mechanisms: - Retrieval: document contents are split into chunks and represented as vectors so retrieved material can help answer questions. - Code Interpreter: the same file is available to generated code, allowing Python to process a PDF directly. One participant also reports successfully asking questions against an uploaded SQLite database; its tested size is unclear.

Capacity and cost impose different constraints: - File count can force consolidation. An assistant is described as accepting up to 20 files of 512 MB each, potentially requiring developers to combine documentation despite the substantial total capacity. - Stored information carries a recurring charge. A participant cites $0.20 per GB per assistant per day, compared with roughly $0.02 per GB per month for raw S3 storage—about 300 times as much over 30 days. The comparison is with raw storage, not complete retrieval systems, but suggests a reason to build your own RAG when storing substantial information. At the time of the conversation, assistant usage is described as free until November 17, roughly ten days away.

API availability and built-in tool availability are separate questions. A participant reports that GPTs offer a DALL-E option, while the demonstrated Assistants interface lacks that checkbox; they leave Assistants’ vision support unconfirmed. Another reports having used the separate DALL-E 3 API the previous day.

Shared context trades explicit control for automatic tool selection

ChatGPT’s unified mode produces a disagreement about who should choose the tool. One participant finds it handles ambiguous requests reasonably well, but anticipates unwanted image generation or code execution. Another has already created a custom GPT with browsing disabled because ChatGPT keeps searching Bing when they want an answer without a search. Automatic selection can remove a choice the user wanted to make.

Reduced visibility creates a related problem for programmers: - Hidden code prevents early intervention. Code Interpreter now hides generated code behind an icon. Previously, a Python programmer could watch it appear, recognize mistakes, and cancel execution. - Opaque errors make recovery harder. Participants report seeing only Analysis failed before a retry. One resorts to inspecting response JSON in browser developer tools and pretty-printing it to debug the operation.

The strongest argument for unification is shared conversational context across tools. A supporter calls it MMIO—Multi-Modal Input and Output—and describes asking Bing for current city weather, then using that information to guide a DALL-E image. The reported result includes snow in Denver. Retrieved information can inform image generation within one conversation; the critic agrees that this is valuable but wants to opt into it.

Participants interpret the simpler interface as a bid for a broader market, leaving professional users to accept less control or build wrappers. They read the GPTs name and marketplace as a related move to bring products such as Canva GPT and Zapier GPT into ChatGPT. Meanwhile, the interface has returned to the Code Interpreter name: both the login pop-up and GPT creation controls use it instead of Advanced Data Analysis.

Chat helps creators start; actions expand what they can authorize

The GPT builder pairs a creation chat with a preview chat. Conversation fills in the name, logo, description, custom instructions, and conversation starters; the preview lets the creator try the result immediately. One participant pastes website content into the builder to produce prompts for a chatbot about their book. The stronger claim that AI is a better prompt engineer than the user receives pushback.

Another participant creates a first working GPT through chat on a phone while riding transit, then builds the next three with the form. Once the configuration becomes clear—a system prompt, tool toggles, files, and a logo—direct editing feels preferable. Conversational setup works as onboarding without becoming the permanent workflow.

External actions extend that accessibility into other systems. An attendee recalls a natural-language request calling the Hue API to change lighting. Another demo interprets a photograph of two people hugging, selects songs to match its mood, and calls Spotify to start playback. The attendee initially calls this no code, then qualifies it as low code because they suspect the APIs were connected beforehand.

Those capabilities trigger the sharpest security objection. Prompt injection here means hidden instructions in retrieved content redirecting an assistant’s behavior. A participant describes how a malicious webpage could steer a browsing-enabled assistant toward private data accessible through Zapier, then exfiltrate it or operate connected devices. These are hypothetical consequences, not a demonstrated exploit. Their criticism is that the demos encourage people to reproduce powerful integrations without explaining this exposure.

Copyright Shield addresses a separate risk: participants describe it as protection against copyright infringement suits involving API use, without establishing contractual terms or eligibility. For action security, they identify a narrower control: an action can be marked consequential, requiring user confirmation before execution. Confirmation gates execution; it is not presented as a complete prompt-injection solution.

GPTs organize recurring work and shift the usage bill

One participant proposes a division of labor: - ChatGPT for unusual tasks: requests the user may never repeat. - Custom GPTs for recurring purposes: a startup mentor has a stable purpose but cannot anticipate every question. - The Assistants API for specific applications: developers build focused experiences, whether the tasks repeat or not. In this interpretation, founders still have an opportunity to automate useful work before the user needs to initiate a chat.

Sharing adds an organizational possibility: an enterprise GPT could preserve an employee’s specialized knowledge as an assistant colleagues can question after the employee leaves. It would extend the role of documentation in tools such as Asana or Confluence.

For public side projects, the immediate attraction is moving model usage off the creator’s API budget. A shared GPT runs under the user’s ChatGPT subscription, described at the time as costing $20 per month. The creator no longer needs to fund visitors’ model calls or cut off access when they exhaust a budget. One participant had wanted OpenAI sign-in for independent applications that would let users spend their own credits. Custom GPTs offer similar economics inside ChatGPT, rather than that authentication mechanism, and the participant predicts many more side projects as a result.

Expressive speech changes how generated text feels

The speech discussion begins with a daily habit: brainstorming with ChatGPT through an AirPod while walking a dog. A participant credits expressive synthesis that changes tone with the sentence. They have also built a small command-line utility that accepts piped text and speaks it through the new speech API.

Their memorable test asks GPT-4 Turbo for a passionate speech about caring for pelicans, then sends it through speech synthesis. The text feels ordinary; the roughly two-minute spoken version feels meaningful and persuasive. They acknowledge an existing fondness for pelicans. The anecdote illustrates delivery changing the perceived force of generated text, without measuring persuasion. An actor-like pirate voice in another demo adds to the enthusiasm, but that voice is private and its release uncertain.

A participant reports that Whisper 3, a speech-recognition model, was open-sourced the previous day, though they have not examined benchmarks. Speech recognition paired with synthesis gives developers the ingredients for conversational voice interfaces. The group wants voice on the web as well as mobile. One participant proposes reusable input components that accept text, images, files, and voice, with a choice between on-device and cloud processing.

Browser inference makes local processing tangible. One participant has used Whisper demos running entirely in WebAssembly with an approximately 150 MB download, alongside browser-based language models and image captioning. They also describe Stable Diffusion running locally without a software installation. Distil Whisper enters as another example of expected efficiency gains, but the claimed size and speed improvements come without a model variant, baseline, hardware, or evaluation conditions.

Vision can interpret images; reliable clicking asks more

GPT-4 Vision’s API lets applications submit images for interpretation. A participant contrasts that with speech-to-text: converting speech into words does not itself preserve singing, intonation, or anger. Their enthusiasm includes Roboflow webcam-analysis demos and an informal impression that GPT-4 Vision is substantially ahead of the open-source vision models they have tried, without a benchmark comparison.

Frame sampling offers a way to apply image interpretation to video. One reported football demo uses FFmpeg to extract a frame every ten seconds and obtains commentary a participant considers good; that rate is not established as sufficient for arbitrary videos. Another participant relays a calculation attributed to Jim Fan: ingesting all eight Harry Potter films at one frame per second and 360p resolution would cost about $180. This is a secondhand estimate, not an actual processing run.

For software development, a participant reports a hackathon prototype inspired by Vercel’s v0, described as a tool that drafts frontends. The prototype uses vision to identify and correct differences in generated frontend output. The broader hope is that desktop and web agents could inspect rendered screens instead of parsing a page’s underlying structure.

Early accounts differ on whether that understanding is precise enough for interaction: - Reading text does not guarantee finding buttons. Div Garg of MultiOn reportedly finds the Vision API good at recognizing text in images after about a month of access, but less useful than expected for identifying buttons to click. - Returning coordinates does not establish reliable clicking. Another participant reports requesting and receiving a bounding box and recalls a demo interpreting a Discord screenshot. The accuracy needed to operate the interface remains unestablished. Participants speculate that internal and deployed model versions might explain the discrepancy, without confirming that explanation.

The technical discussion ends with image cost and context accounting still unresolved. Participants want the equivalent of a text-token counting tool for images: how much of a stated 128K text-token context does an image consume, and does that depend on resolution or file size? Predictable application budgets require an answer that the conversation does not yet have.

The builders in the room

The closing turn returns to DevDay as an in-person gathering. An attendee recalls Sam Altman spending hours meeting developers at the after-party. Another praises the organization and networking but wishes the music were quieter and attendance more international. Their standout encounter is with an invited builder working in Italy on AI education for children with special needs.

Another attendee remembers every chance conversation involving someone doing interesting, hands-on work. The venue is equally distinctive: a converted multi-story car park, ramps between floors, and a rooftop tent for lunch and socializing. Alongside the new APIs, the gathering of builders leaves a strong impression.