Structured Outputs and the Work of Making an AI API Reliable — Part 1 of 3

Valid JSON can still break an application

A model returns valid JSON, but a requested number arrives as a string or a classification uses an unexpected label. The response parses; the application still cannot use it. JSON syntax is a weaker contract than an application schema—the specification of expected keys, types, and allowed values. That gap drives Structured Outputs in Part 1 of the Latent Space conversation about OpenAI’s API.

Charlie’s introduction traces the progression from function calling in June of the previous year to JSON Mode at November’s DevDay, then constrained sampling in April 2024 and Structured Outputs in August. Community approaches included Instructor, LangChain, Outlines, and llama.cpp’s constrained grammar sampling. The new mode combined sampling constraints with training for complex schemas, especially with strict set to true.

The introduction claims 100% reliable JSON Schema adherence without specifying evaluation conditions. Its essential scope becomes clear later: schema conformance does not guarantee correct values. A model can choose an allowed classification and still be wrong about the input.

Learning reliability through production failures

Michelle Pokrass’s engineering education began with Waterloo’s six internships. Her first job involved learning Visual Basic to animate bond yield curves at a bank; Google followed. A four-month Coinbase internship in 2016 brought unusually direct responsibility: payment rails, on-call work, Git, and production databases. She distinguishes that formative internship from her later full-time stint.

After an internship at Stripe, Pokrass helped start Readwise with her roommate through Waterloo’s entrepreneurship co-op. She worked on it for about a year, credits Tristan and Dan as the real founders, and returned to Coinbase to become a stronger engineer before taking on a CTO role. She later joined Clubhouse as it was taking off, roughly its second or third backend engineer. She recalls nearly daily outages, including a major failure when Oprah appeared.

At Clubhouse, notification latency could undermine a live room. Sending all notifications could take about ten minutes when Pokrass joined: the speaker was ready, but the audience arrived too late. Follower relationships lived in Postgres, and the notification logic iterated through followers to decide whom to notify. Accelerating delivery meant improving batching, parallelism, and job queues. Database migrations followed as the existing setup struggled with its workload. Pokrass reports faster delivery without giving a resulting latency.

Debugging long-running Postgres queries became a transferable skill across Coinbase, Clubhouse, and OpenAI. Pokrass singles out zero-downtime migrations as a recurring pain point, tentatively pointing to PlanetScale’s MySQL approach as something she would like for Postgres. Running PgBouncer everywhere had not eliminated database problems.

The database discussion turns to Cosmos DB on Azure and a participant’s preference for DynamoDB when scaling. The consequential point is that moving from Postgres to DynamoDB changes how a system must be designed. Participants argue that bringing conventional database expectations to DynamoDB can cause problems, while adapting to its model can encourage more scalable designs.

ChatGPT scaled the account system, too

Useful products made the opportunity in AI concrete. One participant credits daily GitHub Copilot use with motivating the move to OpenAI, describing its productivity benefit in emphatic but personal terms. DALL-E made the company’s work understandable to family members who could see and use image generation. Another participant recalls declining an opportunity to join because image generation alone had not seemed compelling enough—a decision they later regretted.

The early API operation was small: a participant recalls at most about five people working on it within an Applied team of roughly 30–40, probably nearer 30. Access relied more on tiered rollouts. By the conversation, Applied alone was reportedly larger than the entire company had been when that participant joined.

ChatGPT’s launch stressed familiar infrastructure. Its accounts were tied to API accounts, so entering the research preview effectively created a developer account. That shared foundation brought Postgres problems and repeated authorization outages. At the same time, GPU scarcity made scaling an allocation decision: the team had to decide daily where compute should go, rather than simply increasing capacity wherever demand appeared.

Constrained decoding needs a trained model

Structured Outputs combines constrained decoding with model training. Constrained decoding masks available output tokens at each generation step so that generation fits the supplied schema. Training makes the model better at following those formats. Delivering the feature required both model work and infrastructure that could serve these constraints at scale.

JSON Mode exposed why both are needed: legal tokens can produce useless output. Developers encountered long runs of whitespace, which JSON permits. Restricting generation to valid syntax does not by itself prepare the model to complete a useful response within those constraints.

The SDK reduces integration work: developers supply a Pydantic or Zod definition and receive a parsed object through a parse helper, with serialization handled in both directions. The choice between modes depends on whether the application already knows the structure it needs: - Structured Outputs for a known contract: specify the keys, types, or function-call structure the application expects. - JSON Mode for exploratory structure: request JSON while allowing the model to propose a shape that has not been specified in advance.

Tool arguments and user replies have different jobs

Function calling grew out of internal prototypes of what became Code Interpreter. The team exposed raw tool-use capability through the API before it was ready to host Code Interpreter for everyone. Developers then repurposed function calling to obtain formatted replies, motivating a dedicated structured response format.

The distinction is what the generated object is for: - Function calling produces arguments for an action, such as querying a database or sending email. The model is trained to use tools and receive their outputs. - A structured response format produces a reply to the user, preserving a prescribed shape while retaining more of the model’s natural explanatory verbosity.

At recording time, a helper for completing the tool-calling loop was in beta in the Node SDK, with Python support forthcoming. The described workflow was to write and decorate a function, then use a tool-running method to handle the loop. Functions executed on the same machine. That was convenient for prototypes, but the local helper did not fit execution through an asynchronous job queue. A host also questioned putting it on a web request’s critical path; the respondent acknowledged that some developers would choose that starting point.

Community approaches informed the developer experience, including techniques that pre-form keys and let the model generate only values. The team built its implementation from scratch for its requirements. Developer feedback also shaped model training through collaboration among API models, post-training, and safety teams: simply publishing a research model through an API would miss needs revealed by application development.

Refusal is an explicit application branch

A schema can prescribe a response the model should decline to provide. Structured Outputs therefore includes a separate `refusal` field, preserving the ability to refuse while giving the application an explicit branch to handle and display. Developers can distinguish a refusal from broken structured content.

Returning an HTTP error was considered. One reason given against a 4xx response was that generated refusal tokens are billed, which the respondent characterized as atypical for a client error. More broadly, participants questioned whether client and server errors adequately describe model behavior. Suggested 601 and 602 codes were hypothetical; the reported implementation returned HTTP 500 when a model generated tokens invalid under its underlying language format.

ChatML is the underlying message format for chat completions. The original completions endpoint accepted and returned text; chat completions works with messages distinguished by roles such as user and assistant. Rare format failures were reported under unusual conditions, including very high temperature.

Structured Outputs also constrains tool recipients, preventing the described failure in which a model invents a tool. A valid recipient means an available tool, not a message role such as user or assistant.

Evaluations need room for progress

The respondent reports using the Berkeley Function Calling Leaderboard (BFCL) internally and collaborating on misgraded cases. Their concern is that a nearly saturated benchmark increasingly distinguishes default behaviors rather than underlying capability. The suggestion that most listed models could approach perfect scores with different prompts is an assessment, not a reported experiment. Useful evaluations need to span a broad difficulty curve.

Repository-level coding tasks test a broader combination of abilities: give a model a GitHub issue and repository, then assess whether it addresses the issue. The respondent likens this to an integration test spanning diff formatting, file search, and code writing, with a low pass rate leaving room for improvement. A host raises a qualification: task difficulty can include missing information. Human engineers can usually ask clarifying questions; a model given only the issue may lack that opportunity.

New schemas incur preprocessing latency

Parallel function calling returns multiple calls in one response, reducing latency compared with returning them individually. Newer API models supported it at recording time, but Structured Outputs did not. The obstacle was preprocessing: a new schema required building an artifact for fast sampling. Parallel calls used a combined schema across functions, so changing the function list could trigger additional preprocessing latency. Support was deferred while the team pursued a solution without that unpredictable overhead.

The overhead applied to the first request with a schema. The respondent expected development requests often to absorb that work before production use and anticipated lower latency over time, without offering measurements or a timeline. A separate schema-registration endpoint with schema IDs was rejected as extra developer complexity. Broader caching designs remained exploratory, with no commitment.

Reliable structure helps agents, but does not ensure accuracy

Chaining calls compounds failures: even a high per-call success rate can leave an application unreliable. Function calling and Structured Outputs connect natural-language intent to application actions by making those interfaces more dependable. The respondent describes a move from 95% to 100% reliability without evaluation conditions; that claim concerns structure, not end-to-end agent accuracy.

The Assistants API combined hosted tools with stored conversation state, two bets with different implementation challenges: - Hosted tools: file search and Code Interpreter provided capabilities developers could otherwise build themselves. The respondent reported strong reception for file search. - Stored threads: developers could retrieve conversations later. This first stateful API was still being refined because starting a run required several endpoint calls.

Model support reflected training differences. At recording time, the new response format was available on GPT-4o mini and the new GPT-4o, but not the older GPT-4o. Structured function calling could be enabled on all models supporting function calling because they had already been trained on those schemas. Effectively endless whitespace was a reason to withhold the new response format from poorly suited models.

Asked whether structured-output training caused regressions elsewhere, the respondent named no specific regression. They described capabilities as uneven and training outcomes as difficult to predict. Dated API versions let developers choose the model behavior that works for their application.

Design schemas around what the application consumes

Structured Outputs supports extraction from unstructured input, including vision input, and generation of interfaces. Recursive schemas represent nested trees with children, so one schema can describe many different interfaces. In enterprise classification, enums—a declared set of allowed choices—keep labels compatible with the application. They still permit a factually wrong choice from that set.

The implementation supported a subset of JSON Schema with explicit requirements: - Prohibit undeclared keys: set additionalProperties to false. Requiring this setting preserved the standard’s existing default semantics rather than silently changing them. - Require every field: explicitly list all fields as required so the requested keys appear. - Represent missing values with `null`: keep the field present and allow a union type, such as integer or null.

A math-tutoring example shows the benefit beyond validation. A response can contain an array of solution steps followed by a separate final answer. The interface can render each step individually, avoiding a custom text format and parser. Separate fields let the interface treat parts of a response differently.

Schema descriptions can prompt individual fields, an intentional and encouraged behavior. The recommended division of instructions is: - System message: when to call a function. A customer-support workflow might specify when to verify a phone number. - Function and parameter descriptions: how to call it. A date parameter’s description can specify the required format. Instruction-like property names are also discussed—for example, replacing a short topic key with a request to brainstorm topics—but as experiments with uncertain benefits.

Fewer retries and better-calibrated judgments

Application-specific evaluations make prompting changes testable. The respondent recommends them as a basis for experimenting confidently and adopting new models quickly, while the model team focuses on broad capability improvements.

One participant reports a concrete result from an AI-news workflow: replacing Instructor with native Structured Outputs removed roughly 20 lines of code and reduced measured API cost by 55%, attributed to fewer retries. This was a workflow-specific result without a detailed benchmark setup. The respondent explains that provider-integrated constraints achieve compliance in one generation, rather than through a retry loop, saving latency and cost.

Custom grammars beyond JSON Schema remained under consideration, without a near-term release commitment. The respondent wanted concrete requests for grammar classes, including programming languages such as Python, while acknowledging limits in the implementation’s expressivity.

A request for guidance on using a language model as a judge—assigning ratings to outputs—separates three problems: - Selecting a classification: the respondent recommends mapping choices to A, B, C, and D and inspecting their log probabilities to identify the most likely label, rather than relying on a sampled answer. - Calibrating a score: the requester wants meaningful differentiation across a 1–10 scale, rather than repeated scores of 7. Selecting the most likely label does not resolve that concern. - Validating a range: the requester separately worries about receiving 11 for a requested 1–10 rating. This is a hypothetical concern, not a demonstrated failure with a specified schema.

Choose models by task performance—and test the limits

The model-selection advice at recording time follows a progression: 1. Start with GPT-4o mini, described then as the cheapest model, and assess task performance. 2. Move to GPT-4o if performance is insufficient. 3. Consider fine-tuning for demanding cases that GPT-4o does not handle adequately. Roughly 100 high-quality examples, or 1,000, are suggested as enough to begin, without guaranteeing a particular performance gain.

This guidance came at a specific release moment: GPT-4o fine-tuning was described as becoming generally available the following day, after GPT-4o mini fine-tuning had been available for several weeks. A temporary offer included one million free GPT-4o training tokens per day, with September 23 tentatively given as its end date.

Fine-tuning also raised model lock-in concerns. A participant suggested retaining the original dataset to support migration. The respondent expressed a desire to make migration easier without confirming a public migration mechanism.

The closing guidance is to establish accuracy before optimizing cost and latency, using prompting, retrieval, evaluations, and fine-tuning as appropriate. Trying one model and failing does not establish the limits of language models. Part 1 ends with the harder diagnostic question still open: how can a developer distinguish a model’s capability limit from a weak prompt or an unsuitable model choice?