Beyond the Model API: Tools, Labor, and Expressive Voice — Part 2 of 3

A model that can write code needs somewhere to run it

Code execution makes a large language model (LLM) useful beyond conversation, but it requires infrastructure around the model. This part of the Q2 2024 recap opens with a product heuristic: each capability in GPT Creator could support a standalone business. Search, retrieval, code interpretation, and collaboration with other agents all extend what a model can do.

E2B provides code execution as a service, with a software development kit for adding a code interpreter to different models. A speaker who discloses an investment describes its partnership with Mistral and estimates growth from roughly 10,000 to one million cloud containers spun up in about four months. The counting period is unspecified: these could be cumulative totals or counts within recurring periods.

The first disagreement is about priorities. From an early-stage investment perspective, one speaker favors adding capabilities—analysis, code execution, and search—as the clearest source of customer value. The other calls operations their biggest current problem when building AI applications. A model API leaves substantial production work unfinished, raising a practical purchasing question: what requires an AI-specific operations layer, and what can existing tools such as Datadog handle?

Prompt management is more than storing a string

An unnamed vendor’s $49 monthly offer makes the operations debate concrete: unlimited prompt-template storage and variable substitution at inference time. The developer questions the price while acknowledging the value of helping product managers and engineers collaborate. Grab reportedly takes another approach, treating prompts as code and building its own import mechanism. Either way, prompt text, model selection, and configuration need to travel together; storing the string solves only part of the problem.

That leads to a sequencing dispute. One speaker argues that frameworks generalize practices already known to work, favoring tools built inside a company and later spun out. The reply is that early-stage experimentation must begin before success is established. Julius illustrates the value of focus: despite overlapping with ChatGPT’s Code Interpreter, the speakers attribute its appeal to dedicated attention to a particular problem.

The operations discussion separates three overlapping categories: - Frameworks: application helpers, including prompt management and model calls. - Gateways: one endpoint that proxies multiple model endpoints and normalizes their APIs, commonly toward OpenAI’s interface. - Monitoring and tracing: records of model interactions, latency, and execution-step counts. LangSmith and Langfuse are named examples.

For a small team, specialization can mean integrating several products to cover one workflow. Building a replacement risks adding another competing standard. Beyond integration, one speaker wants observability that helps judge output quality and choose models. They suggest users might accept another 50 milliseconds for a better answer—a proposed tradeoff, not a measured preference. Knowing a hosted model is slow offers limited help when its output is unusable and the developer cannot control its performance.

The other speaker points to Apple’s internal tooling for profiling individual transformer layers and comparing model variants as an example of actionable performance analysis. They want comparable clarity for AI application engineering: tools that help explain what to change.

Conversation memory needs more than similarity search

The proposed LLM operating system is a surrounding set of building primitives that goes beyond prompt-call helpers and observability. Memory is one candidate. One speaker considers vector databases too low-level out of the box: similarity matching and retrieval leave much of the application unfinished. They predict convergence as database and operations vendors add framework capabilities, while frameworks add operations tooling.

The motivating query is what did I say to this person yesterday, last year, or three years ago? Recovering that conversation differs from finding a relevant factual document. A personal wearable illustrates the intended workflow: record conversations, then retrieve material to augment recall. The other speaker argues that semantic similarity alone is insufficient because the same words can carry different meanings in speech. Memory layers above vector databases could address this need for agents, assistants, and chatbots.

GraphRAG is described as combining knowledge graphs with RAG. It had attracted attention at the speakers’ conference. One speaker remains skeptical of recurring enthusiasm for graphs, while allowing that this cycle could differ. Multimodality receives a more incremental assessment: the speaker reports Character AI’s use of LanceDB for multimodal embeddings but sees it as an extension of vector databases’ existing role.

Internal agent coordination and external interoperability

The conversation broadens to the agent ecosystem. One speaker reports that Open Devin was being broadened into All Hands AI, identifying agent coordination and supporting tools as an opportunity their earlier “four wars” framing underemphasized. Browser infrastructure is another distinct capability. BrowserBase, in which the speaker discloses an investment, illustrates the opportunity around browsing separately from search.

An open ecosystem also needs institutional support. One speaker says the AI Engineer Foundation was wound down for lack of interest; its first project was Agent Protocol, donated by E2B. They wanted an open standard but believed companies were prioritizing ownership, venture backing, working products, and revenue. The other speaker reports little interest among their agent investments in communicating with outside agents.

The exchange clarifies two different integration needs: - Internal coordination: several agents divide and perform work within one system. AutoGen and CrewAI are suggested frameworks to examine, using a mixture of natural-language instructions and structured output. - External interoperability: an agent integrates with another vendor’s agent already deployed at a customer. The speakers describe this as a future need with little current demand.

For external integration, the proposed starting point is existing API interfaces—OpenAPI and REST—with workflows supporting longer-running operations. One speaker questions whether every tool ultimately needs rebuilding for AI, but argues that agents currently need specialized interfaces. English alone is too underspecified and unstructured to serve as a reliable protocol.

How much future cost reduction should a product assume?

A comparison between March–April 2024 and July shifts the discussion to model economics. One speaker had plotted the best available combinations of price and capability, with Haiku leading the earlier comparison. They estimate a roughly tenfold cost decline at a fixed intelligence level every four months, revising an earlier twelve-month estimate. Neither the intelligence measure nor the chart methodology is specified, making this a thesis about that period rather than a dependable budgeting rule.

The other speaker challenges the extrapolation: could more capable, more expensive releases change the pattern, and might the apparent acceleration reflect a slower cadence of frontier releases? The proponent has no answer. Their broader development sequence is capability, then generalization, then efficiency, with GPT-4o, GPT-4o Mini, and Claude 3.5 Sonnet cited as signs of efficiency becoming more prominent during 2024.

Larger models can help produce cheaper ones through synthetic training data and distillation: generating training material and transferring capability to smaller models. Comparing Llama 3.0 and 3.1 at the 8B and 70B sizes, the speaker reports the greatest uplift for 8B across their chart’s benchmarks, without naming the benchmarks or quantifying the gains. The engineering question remains open: how should a product plan around rapidly declining base costs?

Two business implications follow: - Build ahead of viable economics: one speaker proposes using venture funding to build useful but currently uneconomic products, accumulating user data and attention while waiting for model costs to fall. - Higher customer value could sustain higher prices: another argues that providers face price pressure when downstream applications capture too little value. A substantially more useful model might resist that pressure.

The second speaker reports annual savings of $10–20 million at some Fortune 500 groups through routine applications such as document translation, contrasting them with the absence of $100–150 million outcomes in their account. The companies and measurement methods are unspecified.

Sell completed research, investigations, and reports

One speaker first argues for more creative applications where invented output is useful, citing Web Sim as a reason for optimism. The discussion then turns to service as software: selling completed labor. Its proponent argues that employee productivity gains can become saved time without increasing company revenue, while startups can benefit by operating with smaller teams.

Two products illustrate purchasing against an existing labor budget: - Brightwave provides research. Financial firms already spend heavily on research, which the speaker says supports higher pricing than buyers might expect for an AI tool. The objection is that paid research can go unread; the reply emphasizes operating-cost pressure and replacing in-house work. - Dropzone investigates security alerts. An agent assesses suspected phishing emails forwarded by an enterprise. A speaker estimates about $35 per alert for human analysis versus about $6 for Dropzone, without specifying a measurement method or establishing equal investigation quality.

Starting a consulting service and gradually automating its work is one suggested route, though its fundability remains dependent on the intended scale. The more immediate advice is to define the specific labor and reduce customer setup. Adept serves as a caution about requiring customers to help build their agents. One speaker also credits Jason Liu with prioritizing finished reports over chatbot-style RAG: reports give people something to consume without requiring them to drive an extended conversation.

General benchmarks guide selection; product evaluations test the job

The evaluation agenda moves beyond reliance on the general benchmark MMLU toward a capability profile: reasoning, mathematics, instruction following, coding, context use, function calling, vision, and multilinguality. The speakers suggest several evaluation directions: - Reasoning: MuSR for multistep reasoning, alongside mathematics and BigBench Hard. - Coding: MBPP as an option after HumanEval, which one speaker considers saturated. - Context utilization: Ruler and Infinite Bench, in a call for tests more demanding than finding a single fact in a long context. - Function calling: Gorilla, API Bank, and Nexus.

General capability signals cannot replace product-specific evaluations. One speaker acknowledges overlooking product evaluations in their original AI engineer post. Academic benchmarks cannot encode an individual application’s requirements; creating those tests is the engineer’s responsibility. The proposed sequence is to look for benchmark signals relevant to the use case, then evaluate its particular requirements.

The recap closes with AI News, an application that grew beyond its original purpose. Its creator started it to summarize Discord discussions, describes it as becoming a media company, and reports roughly 1,000 new signups per month.

Advanced voice responds to performance direction

A bonus session follows, recorded as ChatGPT advanced voice mode began reaching alpha testers shortly before publication. Ethan frames prompting as directing an actor: give guidance, listen, and adjust. In a reading of Rudyard Kipling’s If, broad requests for an expressive performance fail to satisfy the director. The instructions become more specific: lower intonation, an older narrator’s raspiness, and a sense of betrayal in the lines about lying.

The same passage is tried in Scottish, Irish, Australian, New Zealand, Valley Girl, New York, and Boston styles. A listener is particularly impressed by the perceived Australian–New Zealand distinction. Style direction also changes wording: requests for filler words and regional attitude lead ChatGPT to rewrite parts of the poem. Controlling delivery and preserving text are separate concerns in this demonstration.

After repeated changes, the participants say the accents have become mixed. Ethan hypothesizes that accumulated audio tokens in the conversation context condition subsequent output, making an established style harder to change. He suggests a fresh thread for cleaner results. Style persistence from audio context is an untested explanation here, as is the proposed remedy.

Accent guessing succeeds after feedback

The next demonstration reverses direction, asking ChatGPT to infer where a speaker is from. The two attempts follow different paths: - A broad regional match: ChatGPT guesses the United States, then the Midwest near the Great Lakes. The participant identifies St. Louis as home and accepts the answer as close. - A revision with additional context: ChatGPT first guesses the Philippines for another participant. After rejection, more speech, and disclosure of a mixed accent from living in the US, it revises to Singaporean English, which the participant praises.

The Singaporean English result is a successful revision, not a correct first guess. The participant says an earlier attempt had guessed Malaysia. Producing Singlish also takes corrections: the participants repeatedly find the delivery too generic. A later response uses specific coffee-order terms and cautions against adding particles such as la or lor indiscriminately, a distinction the participant strongly endorses.

Expressive range still depends on instruction following

Laughter exposes the gap between describing a performance and delivering it. A prompt about hearing an exceptionally funny joke initially elicits a verbal reaction. The direction becomes explicit: make nonverbal vocalizations, then sustain a belly laugh. Both participants respond positively to the final attempt. They continue with weeping, uncontrollable sobbing, and breathless speech after an out-of-shape marathon finish, praising laughter and breathlessness more confidently than crying. The transcript preserves these reactions but does not capture the nonverbal sounds well enough to assess their quality or duration.

One participant considers these performances beyond their conception of text-to-speech, without offering a settled technical definition. The next task exposes a different limit: the system repeatedly narrates directions instead of performing them. Asked for pure bedtime dialogue between a high-pitched character and a deep-voiced character, ChatGPT keeps speaking character labels and stage directions. After several corrections and vocal demonstrations, it finally omits them; the text alone does not establish whether the pitch contrast succeeded.

Asked to clone a participant’s voice for one character, ChatGPT states that it cannot clone voices. The final experiment shifts from producing expression to interpreting it. Asked about energy and tone, the system describes the speaker as engaged, focused, and direct. Asked explicitly about emotion, it tentatively suggests curiosity and introspection before the response cuts off at this part’s boundary. An emotion description does not establish recognition accuracy: there is no ground truth here, and the exchange does not separate vocal cues from the words spoken.