DevDay Recap, Part 1 of 4: Managed Threads, Custom GPTs and the Cost of Convenience

Managed threads simplify state, but still bill for context

Alex Volkov had repeatedly built conversation-history management in TypeScript and Python. The newly announced Assistants API offered a replacement: associate a thread with a user, then append messages. But if an application sends only the latest message, does it pay only for those new tokens? Volkov reported asking OpenAI staff and learning that existing thread context was still billed. Sending less data did not mean paying only for the new message.

That mix of excitement and unresolved detail runs through the DevDay recap, introduced by Swyx as an oral history of developers and founders processing the announcements in real time. Simon Willison expects the community to need months to absorb the releases; his early exploration already reveals rough edges. Volkov recalls that even some staff at the voice booth did not know Whisper 3 had been mentioned onstage, though he praises the event’s organization.

Alessio sees an opening for founders in what was absent: agents working in the background. OpenAI’s GPTs still center on synchronous, one-on-one chat, while lower model prices could benefit asynchronous applications. Swyx similarly reads the announcements as clarifying where OpenAI intends to compete. Both are offering immediate strategic interpretations, not promises about OpenAI’s roadmap.

Two technical changes deserve separate treatment: - GPT-4 Turbo expands model capacity: its reported 128k-token context window—the amount of material the model can consider at once—comes with lower prices. Volkov anticipates greater speed but has not tested it. - Managed threads reduce application plumbing: developers append messages while the service handles conversation history. That convenience leaves the cost of context in place.

For advance cost estimation, Volkov describes retrieving the thread through another API request and counting tokens locally. Willison’s separate question—whether the API returns usage counts afterward—remains unsettled. So does the exact handling of long histories: Willison raises summarization as an unconfirmed possibility, while Swyx relays documentation describing truncation when messages exceed the model’s window. A continuing thread does not preserve its entire history in model context. Swyx is skeptical about delegating that selection decision to the service.

Document extraction needs inspectable defaults

The same concern applies to retrieval-augmented generation, or RAG, which retrieves material from uploaded documents to help answer questions. The file-processing example involves splitting documents into chunks and representing them as vectors for retrieval. Participants object that OpenAI’s chunking and retrieval choices are insufficiently documented: engineers need those details to decide how to build on the service. They see continued room for LangChain, LlamaIndex and custom implementations where applications need more control.

The attraction is concrete. A travel demonstration used United flight PDFs and Airbnb material to extract information for an application. Parsing long PDFs, selecting relevant information and producing structured output had required substantial work. The demo suggested a way to make that easier. A participant recalls an announcement that its code would be open-sourced; availability is still an expectation at this point.

Similar configuration screens obscure an important product boundary: - Custom GPTs run inside ChatGPT and are configured through a user interface. - API assistants serve applications through the Assistants API. A participant reports being told by OpenAI that they would not automatically appear in the announced GPT marketplace, and that the API could not create GPTs. The Playground and GPT builder look similar, but they are separate products.

Structured output introduces another distinction: valid JSON does not guarantee the requested structure. One participant describes the newly announced JSON mode as constrained generation that prevents malformed syntax, though the precise implementation remains unclear. Function calls can still return unexpected keys in perfectly parseable JSON. Parsing a response and checking its required shape are separate obligations.

Curated tools and two ways to use files

Function calling lets an assistant request work that executes in the developer’s environment. A participant favors a known prompt paired with a curated tool set: a system prompt supplies instructions, and JSON schemas describe the external actions available. That combination should be easier to build reliably than user-selected mixtures of plugins whose compatibility is uncertain. They interpret actions as the likely successor to plugins, while acknowledging that no official deprecation has been stated.

A secondhand report that two assistants can share a thread suggests another possibility: different assistants taking different roles in one user conversation. The group finds that promising, though it is not demonstrated during the discussion.

Uploaded files support two distinct operations: - Retrieve information: a PDF can be split into chunks and converted into vector representations so its contents help answer questions. - Process the file with code: Code Interpreter, which runs generated code in a Python environment, can open the same file and operate on it directly. One participant reports successfully asking questions about an uploaded SQLite database. Files can therefore serve as inputs to programmatic analysis as well as document retrieval.

The reported limits—20 files per assistant, up to 512 MB each—may encourage consolidating documentation into fewer files. Storage pricing creates another reason to consider custom retrieval. Citing a Jeremy Howard tweet, a participant reports $0.20 per GB per assistant per day, versus roughly $0.02 per GB per month for raw S3 storage: approximately 300 times as much. That comparison is against raw storage, not a complete alternative RAG service; the decision depends on data volume and the value of the managed functionality.

Tool availability also differs across products. Participants report a built-in DALL-E 3 image-generation option for GPTs, without an equivalent checkbox for API assistants. They affirm Code Interpreter for assistants but leave assistant vision support unconfirmed. One participant is already using the separate DALL-E 3 API. An API’s availability does not by itself establish that the capability is built into every assistant interface.

Automatic tool selection makes routing part of quality

Unified ChatGPT chooses tools within the conversation, creating a failure mode beyond an incorrect answer: an ambiguous request can trigger unwanted browsing, coding or image generation. One participant finds exploratory testing reasonably successful. Another has already created a custom GPT with Bing disabled after repeated unwanted searches. A capable tool can worsen an answer when it runs at the wrong time.

The revised Code Interpreter interface removes two opportunities for intervention: - Hidden code delays correction. A Python programmer says code now stays behind an icon until processing finishes; previously, watching it appear let them cancel a mistaken approach early. - Generic errors obstruct debugging. An analysis-failed message can precede a retry without explaining the failure. One participant resorts to inspecting and pretty-printing returned JSON in browser developer tools.

The strongest case for unification is shared context across tools. An advocate calls it MMIO, or multimodal input and output: a conversation can combine images, text and different tools. In one reported experiment, Bing retrieves city weather and DALL-E generates a corresponding image, depicting snow for Denver. Another workflow starts with an uploaded image and requests a drawing of its subject. The advocate’s further claim that visual information moves between modes without intermediate text remains unverified.

The disagreement centers on whether automatic routing should be the default. Its critic accepts the value of combined tools but wants to opt in, finding a whole custom GPT excessive merely to disable search. Another participant sees a broader product choice: simplicity for the largest audience, with custom GPTs providing more control for professional users.

Conversational setup can teach the expert workflow

Participants interpret the GPTs name as a platform strategy: OpenAI adopts the naming convention of third-party wrappers and encourages products such as Canva GPT and Zapier GPT to live inside ChatGPT. They also notice Code Interpreter returning as the visible label where Advanced Data Analysis had appeared. Unified mode removes the need to select a separate analysis mode.

The GPT builder pairs a creation chat with a preview chat. The first guides users through a name, logo, description, custom instructions and conversation starters; the second lets them try the result. One participant pastes their website’s contents into the creation chat to generate configuration for a chatbot about their book. The stronger claim that AI is a better prompt engineer than the user draws explicit disagreement.

Another participant builds a first working GPT through chat on a phone, then uses the form for the next three. Once the configuration is clear—a system prompt, tool toggles, files and a logo—direct editing becomes preferable. Conversational onboarding can succeed without becoming the expert workflow. The group agrees that natural-language creation works well for someone encountering the configuration for the first time.

External actions make prompt injection consequential

A stage demonstration connects natural language to physical and digital effects. A mood-setting request invokes a Hue lighting API. A photograph of two people hugging prompts song selection and playback through Spotify’s API. The attendee is impressed by the speed, but qualifies the no-code framing: the APIs were probably connected beforehand. Their impression of a near-instant response is not a latency benchmark.

Those integrations raise prompt injection: instructions in external content that redirect the assistant. If a system can both browse webpages and access private services through Zapier, hidden webpage instructions could induce data theft or unauthorized device actions. The garage-door and lighting scenarios are hypothetical attacks. A participant criticizes the demos for leaving this risk unexplained to builders who might copy them. Grok’s advertised ingestion of real-time tweets raises the same concern: an attacker can publish instructions in content the model later reads.

User acknowledgment offers a decision point before execution. Open Interpreter, a tool for running code locally, is cited as a relevant interaction pattern. A participant later identifies a concrete GPT provision: marking an action as consequential requests user confirmation before it runs.

The conversation also turns to a different kind of protection: Copyright Shield, described as protection against copyright-infringement lawsuits arising from API use. Participants compare it with protections associated with GitHub Copilot, Adobe Firefly and Google Vertex, without detailing eligibility or coverage terms.

Choose the product by how the task repeats

One participant proposes a division based on the shape of the work: - ChatGPT for one-off requests: unusual tasks that may never recur. - Custom GPTs for recurring purposes: stable instructions and constraints, such as startup advice, with unpredictable individual questions. - The Assistants API for specific application workflows: functionality embedded in a product with a more defined job. This returns to the opening startup thesis: founders can use the bundled capabilities to automate work before users need to open a chat.

Link sharing and enterprise deployment suggest another use: interactive documentation. Participants imagine a GPT preserving an employee’s specialized knowledge after they leave, answering questions about material that might otherwise sit in Asana or Confluence. This is a proposed application, rather than a demonstrated replacement for documentation.

Sharing also changes who carries usage costs. A public GPT-4 API side project can consume its creator’s credits. One participant had wanted visitors to sign in with OpenAI and spend their own credits; shared GPTs offer a partial equivalent. At the time discussed, users need a $20-per-month OpenAI subscription, while creators can distribute a GPT without exposing their own API key or managing its API usage budget. The subscription makes small tools easier to share without funding every user’s requests.

Expressive speech changes how generated text feels

The voice discussion begins with brainstorming through an AirPod while walking a dog. A participant finds ChatGPT Voice expressive because its tone changes with each sentence. They have built OSpeak, a command-line utility installable with pip that accepts piped text and speaks it through OpenAI’s newly available speech API.

Their memorable test asks GPT-4 Turbo for a passionate speech about caring for pelicans. The roughly two-minute spoken version feels more meaningful and persuasive than the text, though the listener admits an existing fondness for pelicans. Delivery changes reception in this subjective experiment. A Pirate GPT demo makes a similar impression through theatrical speech, but its voice is private and its release uncertain.

Participants report that Whisper 3 was open-sourced at DevDay, providing speech recognition to pair with speech output; the speaker has not examined its benchmarks. They want voice conversation on the web as well as mobile. The broader frontend proposal is to make images, files and speech standard input options alongside text, with reusable components offering a choice between on-device processing and cloud processing for higher accuracy.

Local processing is already tangible in the reported experiments. One participant has used Whisper demos running entirely in browser WebAssembly with an approximately 150 MB download, as well as browser-based language models and image captioning. Another flags memory use. The estimate of a roughly 15-second download assumes a modern MacBook Pro and a fast internet connection. Browser inference looks practical under favorable conditions, while download size and device memory remain product constraints.

Vision adds feedback, but reliable clicks remain harder

GPT-4 Vision API access lets applications submit images for understanding. A participant distinguishes that capability from converting speech to text: transcription alone does not convey singing, intonation or anger as audio features. They cite Roboflow webcam-analysis demos and judge GPT-4 Vision substantially ahead of the open-source models named Llava and Bakllava, though the comparison has no stated evaluation task.

Sampled images also make video experiments possible. In a reported football demo, FFmpeg extracts one frame every ten seconds and the model generates commentary that impresses the participant. That sampling rate is specific to the example. Another participant relays Jim Fan’s estimate of about $180 to ingest all eight Harry Potter films at one frame per second and 360p resolution. The $180 figure estimates image input cost, rather than measured end-to-end video analysis.

For frontend development, a hackathon team reportedly builds a vision-enabled version of Vercel’s v0, described at the time as a tool that drafts frontends without vision. Their experiment uses visual feedback to correct differences in generated output: the model inspects the result of its coding and uses that information to revise it.

Desktop and web agents expose two different requirements: - Understand the interface: read text and identify what the screen shows. - Locate an actionable target: return coordinates precise enough to click the intended control. A participant had expected vision to avoid parsing the page’s document object model, or DOM, by letting agents inspect the screen instead. But they relay Div Garg of MultiOn’s less encouraging experience after about a month of Vision API access: good optical character recognition—reading text from images—but weaker identification of buttons to click.

Another participant reports receiving a bounding box from an uploaded image and recalls Greg Brockman’s demonstration identifying channel information in a Discord screenshot. Those successes leave the disagreement about reliable UI operation unresolved. Suggestions that different model versions or deployment constraints explain the gap remain speculative.

The technical discussion closes with an accounting question that echoes managed threads: how do images consume context and money? Developers want to know how image inputs relate to the advertised text-token window and whether resolution, file size or another measure determines cost. The conversation leaves them without a concrete method for budgeting mixed image-and-text prompts.

Meeting the builders was part of the value

The closing reflections turn to the people at DevDay. One attendee describes Sam Altman spending hours at the after-party meeting developers and reads that as personal commitment to the community. Others value encounters with people actively building products, including someone in Italy working on AI education for children with special needs. The praise comes with a request to invite more international participants.

The venue leaves its own impression: a converted multistory car park, ramps between floors, and a rooftop tent for lunch and conversation. An attendee praises the organization, including $100 in Uber credit for travel to the separate party venue, while wishing the party music had been quiet enough to talk over. For attendees eager to compare what they were building, conversation was a major reason to be there.