From API Stability to o1’s Reasoning Budget — Part 2 of 3

Prompting starts with specifying the result

A developer should not need a story about a sick grandmother to get a useful model response. This part of the Latent Space conversation opens with that prompting problem: experimenting with ChatGPT builds intuition for API use, but contrived emotional appeals are friction the interviewee wants models to remove. For now, creativity and persistence still help.

Clearly specifying what to build remains engineering work. The interviewee expects that skill to retain its value as models improve: developers still have to determine requirements and explain the intended result, even when better tools handle more of the implementation.

Model selection introduces another kind of contract: - ChatGPT-latest tracks a rolling model. Its weights are not pinned; it exposes the model used in ChatGPT for chat-oriented applications. - Pinned API models keep weights stable. Developers can tune applications against a specific version. These models emphasize developer needs such as function calling and structured outputs, with a stated commitment to long-term support. The rationale for offering both is application choice. ChatGPT and API models had often been identical in earlier releases, though they sometimes diverged.

Stable versions do not solve the problem of understanding an upgrade. The interviewee acknowledges that release notes were less detailed than desired: characterizing what changed between models and how their capabilities differed remained difficult.

File search removes an early retrieval bottleneck

The API strategy pairs developer breadth with first-party depth. Developers can reach niches and communities that OpenAI’s own products cannot all serve. The API predates ChatGPT as OpenAI’s first product and initial commercialization idea; writing and deploying software remains central to turning model capabilities into applications.

The Assistants API illustrates the delivery pressure behind that ambition. A very small team worked toward a fixed DevDay deadline, prioritizing day-one access. Roughly two hours before the live demos, an outage prompted a scramble. Service was restored and the demonstrations—including the free-credit giveaway—succeeded. The interviewee did not know the backup plan.

The subsequent file-search redesign changed more than capacity. The earlier implementation supported approximately 20 files per assistant and chose a file to search based on its filename. The newer offering supported 10,000 files and searched semantically across all files at once. That removed the need to select a document before searching its contents. Customers reportedly saw good performance, without a quantified benchmark in the discussion.

The goal was RAG that worked out of the box, with optional controls for power users. File search already included re-ranking. More re-ranking controls were expected soon, while chunking options also offered customization. The intended balance was useful defaults for most developers and optional controls for power users.

Behavior controls are not guarantees

The smaller API controls reveal several limits: - `seed` provides best-effort repeatability. The implementation was more deterministic in the first few tokens. Improving determinism involved tradeoffs with reliability and uptime. - Token biasing favors outputs. Biasing valid classification labels makes matching outputs more likely. Some power users also bias punctuation to encourage succinct writing. - Word suppression depends on tokenization. A proposed attempt to suppress “delve” raises the need to handle possible token permutations; no successful implementation is confirmed.

Access rules were becoming more transparent. Developers could see their API tier in the dashboard and consult published criteria for moving between tiers. Tiers also supported gated feature rollouts; the interviewee believed tier two and above had full access to the fine-tuning launch under discussion.

For developers building around OpenAI, the larger uncertainty was the platform’s scope. The interviewee offers an 80/20 prioritization principle: build capabilities capturing most of the value, potentially leaving the long tail to others. Feedback, integration effort saved, and impact would guide decisions. There was no firm boundary around what OpenAI would build.

Model fallbacks make that uncertainty concrete. A host argues that outages force application developers to implement fallback logic even when it adds little differentiated value. The response emphasizes reliability investment and reports substantial uptime improvement over the preceding year, without figures. Hiding integration complexity was the platform’s broader goal, but the fallback requirement remained unresolved.

Batch requires deadline tolerance and independent requests

At the time of the interview, the Batch API offered a 50% discount for a 24-hour turnaround, including support for GPT-4o mini. Suggested applications included daily or milestone-based user-activation emails and offline evaluations that did not need immediate results.

The promised turnaround must fit the production deadline. A host’s daily workflow required results in two to four hours, making the 24-hour contract unusable despite the discount. Shorter windows, and much longer windows for low-urgency work, were suggestions rather than announced offerings.

Vision brings a different advantage to extraction: image input preserves spatial relationships that can be difficult to express in text. The interviewee describes it working with Batch, Chat Completions, and structured outputs. Extending that to video still left developers managing frame sampling; an API that accepted video and sampled frames for them was floated as a possibility, without a release commitment.

Video exposes a second Batch constraint: requests stand alone and do not chain off one another. A batch of images from a long video could support independent frame analysis. It would not provide the dependency needed when processing the next frame requires an earlier result. Serial batch execution was proposed during the exchange, not described as an existing feature.

Speech needs speaker handling and vocabulary guidance

The podcast workflow makes the next gap tangible: a useful Whisper transcript still needs diarization, the assignment of speech to speakers. A participant uses a Python notebook for that work, but overlapping speech and similar voices cause errors requiring manual review. Praise for the team’s simple Whisper-based transcription tool, including a favorable comparison with Descript, is anecdotal.

The interviewee, who says they shipped the Whisper API, attributes the decision not to expose Whisper V3 through it to performance tradeoffs and competing priorities. V2 reportedly performed better on some unspecified tasks. No benchmarks or firm V3 release commitment accompany that explanation.

Technical vocabulary can be handled at two points: - Guide transcription with a prompt. The Whisper API accepts unfamiliar acronyms and vocabulary as context. The recommendation carries no deterministic accuracy guarantee. - Repair recurring errors afterward. The podcast workflow keeps a dictionary of observed misspellings and applies string replacements, including inconsistent renderings and capitalization of LLaMA and LangChain.

Speech-to-speech interaction required a broader interface rethink. OpenAI was exploring a new API shape because ordinary request-response interaction was expected to be inadequate; design details were not ready to share.

Enterprise administration had more concrete additions: recently shipped admin and audit-log APIs enabled programmatic key management, project management, and audit-log retrieval. The interviewee also reported improvements to single sign-on, following difficulties tracking an organization’s API keys through the dashboard.

Building habits and reconsidering ownership

Turning to engineering formation, the interviewee credits Waterloo’s co-op program and hacker culture. Six personal internships, alongside an environment shaped by Hack the North and startup incubators, provided extensive practice. In this account, six internships let students graduate with roughly two years of experience.

The Making of the Prince of Persia, a Stripe Press book, is recommended for its journal account of the effort behind an exceptional product. A host’s memory of playing The Sands of Time as a first PlayStation 2 game leads to a related observation: a striking product raises expectations for other products. Advanced voice technology prompts a similar reaction.

Richard Thaler’s Misbehaving offers a way to inspect decisions. Suppose you win concert tickets worth $10,000—a hypothetical value. Would you spend that amount in cash to buy them, or prefer the money? The heuristic is to reconsider ownership as a fresh choice, rather than treating how something was acquired as a reason to keep it.

Loss aversion—reacting more strongly to losing something than gaining it—appears in two examples: - Promotion expectations: missing a promotion brings disappointment, while receiving it may bring only relief. - Onboarding incentives: a platform grants points, then threatens to remove them unless the user acts within the first few days.

The interviewee borrows the book’s term “econs”, rational decision-makers contrasted with humans, as a personal heuristic: what would an econ do? Asked whether large language models are more rational than people, the interviewee narrows the analogy to their optimization functions being clearer.

The interview closes with hiring as it stood then: applied engineering, research, and a newly announced model behavior role. The interviewee emphasizes low ego, drive, and attention to users, alongside willingness to do necessary work. Hires came from varied backgrounds; extensive AI-specific experience was not presented as a prerequisite.

o1 changes the integration contract and token budget

The recording moves to an emergency San Francisco meetup. Latent Space had delayed the interview’s publication after learning that Strawberry, released as o1, was imminent. In this launch-day discussion, o1-preview and o1-mini were available; full o1 remained unreleased. Preview was described as an intermediate checkpoint toward the broader model.

The presenter reports that o1 solved a film’s board problem, without supplying the exact prompt or verification procedure. The integration failure is more specific: existing API code could not be ported unchanged. The launch API lacked system-role messages, temperature configuration, tool calling, and streaming. The missing system role broke the presenter’s Instructor-based code.

The presenter distinguishes the model’s hidden chain of thought, its intermediate reasoning, from ChatGPT’s displayed summary. Their understanding was that the summary was unavailable through the API and reasoning tokens were billable. In a multi-problem task, reasoning consumed the available token budget: the model produced the first solution but failed to complete the next three. Reasoning could exhaust the budget before all requested answers appeared. The precise limit and token-control contract remain unclear in the exchange.

The larger development was test-time compute: computation spent answering a question, beyond computation spent training the model. The presenter’s reading of the launch material was that performance appeared to scale predictably in a log-linear fashion with both training-time and test-time compute. Additional inference computation could therefore become an investment target.

That raises a practical boundary: how much reasoning improvement can developers obtain at inference time, and how much requires model training? The presenter interprets the available hints as chain-of-thought behavior learned through reinforcement learning and substantially stronger than prompting alone. Developers did not have access to that training process.

Stronger specialist scores do not establish general superiority. The recap attributes o1-mini’s stronger science, technology, engineering, and mathematics scores to targeted training, while cautioning against treating it as better than o1-preview across tasks. An OpenAI representative encourages testing previously difficult agents and applications, acknowledging that broader access and higher rate limits still required work.

Use reasoning selectively and retry capability limits

Lindsey explains that the o1 naming reset signals a shift toward reasoning. The “o” stands for OpenAI, unlike “omni” in GPT-4o. Investment in the GPT series and multimodality would continue. The exchange distinguishes continued modeling work from its eventual name, without establishing a definitive GPT-5 roadmap.

The proposed application architecture combines the model families: - GPT-4o and GPT-4o mini remain workhorses for tasks already working well, including summarization and labeling. - o1 handles difficult, multistep problems where additional reasoning may help.

Speed and compute allocation remained open engineering questions: - Speedups were being investigated. A respondent says conventional model-speedup techniques still apply and mentions alternative attention mechanisms, without a specific implementation or measured gain. - The serving system contributes to reasoning behavior. The answer does not explain the division between model and orchestration. - Developer control over compute was unresolved. A respondent tentatively says the model and question determine allocation, without identifying a concrete control.

The most actionable recommendation is to retry a project that hit a GPT-4-class capability limit. Suggested experiments include refactoring a large codebase or troubleshooting an issue across it, with o1-mini recommended for coding. These are invitations to test, not reported refactoring successes. The API was also an early release, with more features and tools planned.

Visible summaries, retained context, and disclosure

A model that may spend around 30 seconds reasoning creates a new interface problem. Initial API integration prioritized Chat Completions; more information and controls around thinking were possibilities. A respondent points to a quantum physicist following equations step by step as an example of how visible reasoning could help users understand a solution. A useful explanation is not necessarily a full account of internal computation.

The multi-turn ChatGPT discussion separates two kinds of state: - Visible conversation content accumulates, including the displayed thinking summary and answer. - Hidden thinking tokens do not accumulate across turns. Each new turn requires thinking, but previous turns’ hidden tokens are not all carried forward.

One participant reports a conversation that stalled and stopped accepting messages instead of truncating history to allow continued use. The exchange does not identify a reproducible trigger or precise token limit.

The closing exchange turns from retained reasoning to disclosed reasoning. A speaker reports that Cosine scored slightly above o1 in a coding-benchmark comparison using o1’s self-reported results, without numerical scores or evaluation conditions. The benchmark identity and the scope of the comparison between its variants remain unclear.

According to the same speaker, Cosine could not join the leaderboard because submissions required thinking trajectories that it withheld as intellectual property. The speaker draws a parallel with OpenAI withholding its own trajectories. The part ends on that tension: protecting reasoning trajectories can conflict with disclosure requirements for evaluating results.