OpenAI Functions: From Prompted JSON to Tool-Using Software

A copilot that runs out of room

One developer’s copilot prompts consumed most of GPT-3.5’s 4K context window before the model could answer. The team used GPT-4 sparingly in production because of cost. GPT-3.5 16K addressed a concrete constraint: accommodating substantial instructions and input while leaving room for useful output.

The June 2023 OpenAI update combined that fourfold context expansion with a 75% embedding price reduction and a 25% GPT-3.5 price reduction. In the Latent Space roundtable, Alex considered function calling the substantial change; swyx initially saw a useful incremental release. Their disagreement would recur: how much does removing integration work matter when the underlying technique already exists?

Swyx qualified the excitement: fitting information does not establish that a model can use it effectively. His thought experiment involved adding two hypothetical 100-million-digit numbers. Accommodating the digits would not demonstrate the ability to compute the answer. The practical question was whether GPT-3.5’s enlarged window would retain the effectiveness of its smaller one; his proposed explanation involving attention heads was speculative.

The model requests a function; application code runs it

Function calling lets developers supply function descriptions and parameter specifications so the model can choose a function in response to a user’s request. The June API added a function message role alongside system, assistant, and user, with fields for available functions and their responses. Swyx interpreted this as bringing plugin-like capabilities into the developer API.

Simon Willison explained the goal behind many requests for reliable JSON: the model asks for a tool, and the application executes it. Prompting a model to request the right function, in the right format, at the right moment had been awkward. Native support promised to remove some of that work and provide a more dependable foundation—an expectation at this stage, rather than a measured reliability result.

Two approaches to structured output needed separating: - Fine-tuning for desired behavior can make function requests easier to obtain while still allowing malformed output or invented content. - Grammar-constrained generation restricts the syntax the model can emit. It addresses output structure, a different problem from learning when to request a tool. Participants noted that the API still warned about invalid output and hallucinations. Function specifications did not remove the need for validation.

Eric Elliott proposed comparing function-call accuracy across ordinary instructions, explicit interface definitions, and pseudocode. Pseudocode had helped him obtain better constraint adherence in earlier prompting work, but he had not extensively tested the release. The experiment would test which way of specifying a task actually improved the behavior an application needed.

Who controls the workflow?

Sean said the release eliminated work planned for smol developer, but left a larger architectural choice unresolved. Where should the large language model (LLM) sit? - LLM core, code shell: the model directs execution, requests application functions, and receives their results before deciding what happens next. - Code core, LLM shell: application code controls the workflow, calling intelligent components for bounded interpretation, planning, or selected decisions. Sean favored the second arrangement for control, security, and privacy. He cited Voyager as an example of using an LLM to draft code, then reusing working code instead of repeatedly returning to the model.

At a Fixie hackathon, Stef had observed two onboarding problems: - Newcomers using natural language struggled to constrain and verify outputs. - Programmers mixing prompts with code struggled to decide which should handle a given task. Riley expected function calling to remove one recurring lesson—how to coax JSON out of a chatbot—although he had only read the documentation at that point.

Function schemas offered a familiar foothold: specify that an argument should be a number, text, or an object, much as engineers express expectations with TypeScript types. A familiar specification does not guarantee adherence, but it makes the expected behavior easier to describe. Whether the model followed that specification better remained a question to test.

Cheap embeddings leave data-location choices intact

Roie was more struck by the embedding price cut than by function calling. Its cause remained unknown. Sean suggested infrastructure optimization and different training–inference trade-offs, then offered a commercial explanation: a corpus embedded in one provider’s representation depends on that representation for querying and retrieval. Lower prices could therefore encourage dependence on the provider’s ecosystem.

Xenova described [Transformers.js](https://github.com/xenova/transformers.js) as a project that began with running models locally in the browser. Generating embeddings locally could keep sensitive data on the device and avoid repeated remote requests. Lower API prices leave those privacy and latency motivations intact.

The practical boundary was disputed. Xenova favored hosted APIs for industrial applications and local embeddings for smaller projects. Another participant reported using Transformers.js successfully in Node and argued that avoiding third-party calls had broader value. Xenova compared OpenAI’s roughly 1,536-dimensional embeddings with local models of around 768 dimensions he had encountered; this was neither a browser limit nor a quality benchmark.

Choose the tool catalog, then build the loop

Large catalogs introduced unanswered questions: how would selection scale, and what would happen when descriptions overlapped? No function-count limit was established. Simon emphasized that developers controlled the exposed functions, unlike a plugin environment assembled by users from unfamiliar third-party components.

Simon’s hypothesis was that a few expressive tools could work better than many narrow ones. His SQL plugin accepted a query and returned a result. A short instruction requesting SQLite syntax worked well in his experience because the model already knew the language; he pointed to a similar example in the OpenAI function-calling cookbook. Extending this approach to JavaScript, Python, or an eval function was an exploratory suggestion, not a tested deployment recommendation.

For larger catalogs, another proposal was two-stage routing: select relevant functions first, then expose only that subset to the next model call. The set could change as application state evolved. The June API’s execution loop kept responsibilities explicit: 1. Supply the user request and available function specifications. 2. Let the model choose a function, or require a particular function choice. 3. Execute the requested function in application code. 4. Return its output and identity in a function message so the model can process or summarize the result. Retrieval from a vector database could occupy the execution step, with the retrieved material returned for a subsequent answer.

A proposed code agent needed three capabilities: generate code, test code, and call existing code. Single-shot generation in smol developer sometimes needed repeated repair; a loop could provide additional attempts. A reusable skills library would then preserve successful work: draft code, validate it, save it, and invoke it again. Participants credited Voyager with this approach and considered it a direction for smol developer V2, without presenting results for the new design.

Function descriptions become routing instructions. Similar descriptions could compete, and shared libraries would enlarge the search space. Automatic selection would require experimentation with those descriptions. Applications that already knew the required operation could request that function’s output directly.

Larger context can remove splitting work

Mayo challenged the larger-window enthusiasm from a question-answering perspective. If retrieval remained part of the workflow and embeddings were cheaper, what made more context transformative? He saw possible uses such as named entity recognition, but regarded function calling as an extension of existing LangChain tool-use abstractions. The meaningful shift, he suggested, might be replacing some prompt engineering with fine-tuning for tool use.

A transcript-translation workflow supplied a concrete answer. One participant used Whisper for translation, then GPT-3.5 or GPT-4 to refine the translated transcript. Long videos required token counting and splitting, which worked poorly in that application. A larger window could allow more transcripts to be processed whole. Per-token costs and effective use of the longer input still mattered.

Context capacity and output length remained separate questions. One participant had received roughly 7,000-token responses from GPT-4 32K and, separately, supplied roughly 30,000 input tokens. These observations established neither a combined request budget nor a formal output limit. Generating 16,000 tokens of JSON remained a proposed test.

The conversation returned to observability and repeatability of action selection. A participant objected to hiding how an agent reached an action. The code-core alternative kept intelligent components bounded, with BabyAGI’s component design cited as inspiration. Native function calling simplified the connection without deciding who should own control flow.

A conversational API can power more than chat

Riley described a trade-off in moving from document completion to message-based APIs. Conversation became easier, but developers lost some ability to establish a fictional document and have the model continue it. An assistant might instead refuse or add unwanted prose. The familiar structured-output failure was JSON preceded by a conversational introduction that application code had to strip away.

Stefania argued for two changes: - Smaller, more efficient models could eventually support on-device deployment with better privacy and affordability. Compute scarcity and latency strengthened her case, though these were anticipated benefits rather than deployment results. - Interfaces beyond chat would avoid making every calendar or email task a conversation. Riley separated the API from the product: a message-based API can supply structured outputs that drive other interactions, with the model operating as a reasoning component behind the interface.

Riley traced this distinction back to in-context learning in GPT-3: examples inside a prompt establish a task such as translation. A conversational assistant was a subsequently engineered behavior, and choosing a polite persona that answers questions and follows commands was a design decision. Earlier completion prompts could steer the model by establishing a fictional skilled translator. In Riley’s view, later releases increasingly put control of that persona with the provider. Function calling gave structured programming tasks a more natural place within the conversational interface.

Valid function calls can still cause harm

Simon argued that easier integration could increase exposure to prompt injection: untrusted content redirecting the model away from its intended task. His hypothetical email assistant was asked to summarize a message, but the message instructed it to delete other emails. Connecting the model to functions made that redirection consequential.

His conservative threat model treated the author of a webpage being summarized as potentially able to control the model’s output. Better adherence to system instructions could make attacks harder, but detection alone was not a sufficient security boundary: adaptive attackers could search for inputs that bypassed the filter.

Simon’s design advice assumed an attack could succeed: - Limit available actions each round. Control which functions the model can request for the current task. - Make changes reversible. An unwanted deletion should have a recovery path. - Require human approval for irreversible actions. Sending an email to a boss was his example of an action that could not simply be undone. Approval also needed care: users could become habituated to accepting prompts, much as they click through cookie banners.

Nisten emphasized the product opportunity in a more predictable response interface and proposed functions for cleaning sensitive data before sending it onward. Simon wanted conversational data-cleaning tools with undo as a core capability. Another participant proposed classifying smol developer’s functions as reversible or requiring human input.

Stef introduced Garak, a tool for probing LLM security with automated adversarial testing, and raised the disclosure dilemma: publishing attacks educates developers and attackers alike. Simon favored public education because, at the time, he saw no known general fix for prompt injection. Developers sometimes needed to decline features they could not build securely.

Constrained syntax does not establish safe intent

[JSONformer](https://github.com/1rgs/jsonformer) sharpened the distinction. Simon described it as adding logic at next-token prediction to constrain structured output. Xenova initially suggested this eliminated prompt injection, then accepted Simon’s counterexample: a malicious webpage could induce a perfectly valid call to an unwanted function. Correct syntax does not establish that an action is appropriate.

Grammar-based constraints could extend the technique beyond JSON to other specified output languages. Simon also welcomed a proposal to upload a WebAssembly program that would control next-token selection at the API. This was a desired capability, not an announced feature. Participants did not know whether OpenAI used constrained decoding internally.

What developers wanted next

The closing requests exposed several development priorities: - Retrieval designed into training: Stef wanted better retrieval and knowledge graphs without necessarily depending on vector databases. - Interactive widgets: Simon wanted plugins that could display maps and selectable options. - Models developers could experiment with: Xenova wanted a roughly 30-billion-parameter model and would accept a Llama-style license. Mayo wanted the function-use fine-tuning approach applied to open-source models. - Specialized model routing: Shawn planned to explore smaller models for particular tasks and recursion in his developer agent. - Cheaper long-context inference: one participant wanted GPT-4 32K at a 75–80% lower cost—a request, not an announced reduction.

Riley returned to the economics that opened the conversation. Lower prices could make secondary prompts that check earlier work affordable, allowing developers to add redundancy where a system needed it. He considered cheaper inference and larger contexts potentially more consequential than function calling itself, without claiming measured reliability gains from those checks.

The June 2023 closing announcements included Roie’s virtual hackathon for June 19–26, with 100K in prizes; Riley’s planned prompt-engineering talk at a July 15 Scale hackathon; and Shawn’s preview of a George Hotz interview about Tiny Corp, including an email exchange with Lisa Su. Riley believed hackathon applications had closed and expected, without confirming, a later webinar version of his talk.

The final technical request was for rapid access to a Vision API, opening more uses for GPT-4’s visual understanding. It came with another version of the same security question: could text presented visually redirect the model?