Production AI Engineering Starts with Evals

Good technology still needs a usable package

Infer a database schema from the first N rows, then receive a row with a different shape. The schema must change for the preceding rows too. Early in this Latent Space conversation, Ankur Goyal uses that problem to explain Snowflake’s `VARIANT` type, which stores and queries semi-structured data while accommodating different shapes. In his experience, its efficiency approached that of an explicitly specified schema.

Goyal wanted that capability for observability—capturing and inspecting application behavior—but Snowflake’s query latency, licensing, and deployment constraints stood in the way. He could not take the implementation and run it on a laptop or in a customer’s environment. Packaging determines who can use a technical capability. A more accessible implementation might accelerate a new observability company, he argues, while acknowledging that Honeycomb could have specialized requirements he did not understand.

That tension runs through his career. A Microsoft internship offered practical impact but little creative latitude; academic research offered creativity but little use of what he built. He found the combination he wanted at the three-person MemSQL, joined as employee number two, and stayed almost six years, eventually running engineering. He later ran Impira for six years, spent eight months at Figma after its acquisition, and reached Braintrust’s first anniversary.

Looking back at SingleStore, Goyal describes sophisticated database technology whose hardware and software costs made it a substantial commitment, even though it was cheaper than Oracle Exadata or SAP HANA during his tenure. It was hard to imagine starting a weekend project with it. His assessment is explicitly historical: he had been away for eight years, during which the company had continued to change.

Impira: feasibility does not guarantee priority

Impira aimed to make unstructured data as easy to use as structured data, with machine learning supplying the connection. AlexNet inspired the idea, but pre-LLM models lacked sufficient capability, and customers struggled to assemble enough training data. The product eventually focused on extracting information from PDFs.

The founder’s account separates two commercial lessons: - Technical enthusiasm is only part of a sale. Successful demonstrations at MemSQL had hidden the work of obtaining meetings, negotiating contracts, closing deals, and retaining revenue. An early attempt to close Stitch Fix exposed mistakes an experienced account executive could handle. - Customer understanding shapes product judgment. Selling to line-of-business buyers at Impira made six- or seven-figure contracts easier to secure, but those customers were harder for the founder to understand intuitively than technical users. That weakened anticipation of their needs and made segment focus harder.

A hypothetical insurer illustrates a deeper obstacle. Its executives could choose between two investments: - Automate existing paperwork: extract information from documents more cheaply, improving a process that already has a manual workaround. - Replace the paperwork experience: build a conversational phone application that completes an underwriting survey and potentially reaches a new customer segment. The argument is that executives would favor the second project even if it cost ten times as much. A defensible return on investment does not guarantee organizational priority. An operational improvement still competes with opportunities to create a new product.

Model progress then challenged Impira’s technical advantage. The team had reportedly reduced the examples needed for a document type from thousands to perhaps two or three, using computer vision and other machine-learning techniques. Documents contain both visual and textual signals; as text models improved, visual extraction became less defensible. In an informal experiment, a BERT question-answering model on Hugging Face extracted invoice numbers surprisingly well. The account gives no measured success rate.

The team reworked Impira around Microsoft’s Layout LM, contributing document-model support and fine-tuned models to Hugging Face. This reportedly removed the need for the more complex extraction technology in almost all cases. Another informal experiment with OpenAI text generation appeared stronger still, using PDF text without visual structure. Its exact interface and timing are unclear, and no quantitative comparison is given. But the strategic realization was clear: dismissing the new approach because it was expensive repeated the defensive response to the previous model transition.

Acquisition, release constraints, and a stalled model decision

Impira pursued acquisition after weighing rapid model progress, commercial traction, team exhaustion, and the prospect of raising more capital for another five to fifteen years on the same problem. Figma acquired it in December 2022. The combination of language and visual information was familiar territory, and the team liked the people it would join.

Winding down Impira remained painful. The team offered generous refunds and wrote code to help customers transition, though the respondent could not say every customer was adequately served. The experience left a difficult founder lesson: making a necessary decision and handling its emotional consequences can be separate tasks. Discomfort can delay a decision without making it less necessary.

Figma brought different constraints. The guest describes an annual release cadence and a high reliability bar that made rapid AI iteration difficult. Figma-style vector representations also consumed substantially more tokens than HTML or JSX in this account, making them awkward inputs for language models; no token measurements are given. AI-generated code looked promising as a way to help designers collaborate with UI engineers. Personally, however, the guest wanted a faster loop from user complaint to shipped improvement.

Braintrust’s origin lay in an earlier Impira stalemate: promising examples were easy to demonstrate, but approaches were hard to compare. The team debated models for two or three months. After building an evaluation prototype with Snowflake and scripts, it reportedly shipped a new model about two weeks later. Some cases regressed, but the aggregate improvement supported a decision.

Evals are repeatable comparisons of outputs against explicit scoring criteria. They turn disagreements into concrete engineering work. In the guest’s illustrative migration, Layout LM handles ten invoices the old computer-vision system could not process but regresses on three others. Those three become targets for improvement and remeasurement. The counts illustrate the decision process rather than a benchmark result.

An interviewer points out that production machine learning already used evaluations. The response locates the opportunity in a change of users: BERT, GPT-3, and GPT-4 let traditional software engineers participate directly in developing AI behavior, while existing tools still reflected ML specialists’ workflows. Braintrust’s developer platform makes evaluation the loop for improving AI products: measure each change, then use the results to choose the next one.

Use the same function in production and evaluation

Once teams can run evals, they need useful examples. Converting application logs into evaluation datasets can become a separate data-transformation project. Braintrust’s shortcut starts in the application: evaluate the same function abstraction that production runs. Capture its runtime information consistently, and the logs arrive in the same trace-and-span format used during evaluation.

That shared boundary supports two entry points: - Start with evals: reuse their instrumentation to obtain application logs. - Start with logging: collect production examples into datasets and reuse the application function for evaluation. The benefit depends on aligning the function boundary and captured information. Arbitrary logs do not automatically become useful evaluation inputs.

Aggregate gains still require case-level debugging. A higher score leaves unanswered which outputs improved, which regressed, and whether one metric rose while another fell. Those questions pushed Braintrust from a dashboard toward a debugger. Users then wanted to act on what they found: edit a prompt, change a model, and run the comparison again.

The playground consequently added automatic saving, sharing, real-time collaboration, side-by-side prompt and model comparisons, and saved prompts deployable into code. A Brex engineer’s request to make it an integrated development environment helped clarify that direction. The ambition was to bring software-engineering practices to AI development; forking VS Code remained an idea, not a commitment.

The demo: a perfect score still misses a defect

The press-release summarization demo shows the loop in practice. Its LLM judge—a language model used to score another model’s output—uses a prompt with several possible choices, each mapped to a score. The demonstrator follows three steps: 1. Write the evaluator first. Establish the desired summary behavior before changing the generation prompt. 2. Run the baseline. Generate summaries and score them against that evaluator. 3. Change the prompt and rerun. Add a brevity requirement already represented in the evaluator. The reported summary-quality score rises from 54% to 100%. This is a toy demonstration under a particular rubric, not a general measure of summarization accuracy.

A 100% score did not mean the output was finished. One summary still had an unwanted introduction. The demonstrator added a narrow Python evaluator checking for the word sentence, then edited and tested it interactively in a sandbox. An additional embedding-similarity score compared generated summaries with a reference summary. The shorter version reportedly improved that score too, though no numerical similarity results are given.

Full experiments provided more inspection space than the playground, allowing comparisons across runs and granular differences between outputs. A preview then extended the workflow to tools. TypeScript wrappers around external APIs, including Exa, supplied type definitions for tool-call schemas and metadata for naming and storage. The demonstrated npx braintrust push command bundled files and dependencies for use in Braintrust. Tool support was presented as an upcoming iteration expected by publication.

A one-row example asked for the premier conference for AI engineers, first without tools and then with Exa search enabled. Search ran inside the playground, but a participant still described the answer as imperfect. The broader workflow joined reusable tools, prompts callable through REST API endpoints, and custom evaluation scores in one interactive development process.

Beyond a spreadsheet: history, execution, and accounting

An interviewer offers a practical alternative: Claude in Sheets already supported spreadsheet formulas, model calls, and Claude evaluating Claude. HumanLoop offered persistent playgrounds and evaluations too. The response concedes that an initial eval script is easy. The harder product is a durable team workflow with historical runs, shareable results, and reliable parallel execution. At Impira, evaluation screenshots in Slack had supplied the experimental memory needed to revisit abandoned ideas.

Braintrust’s declarative interface describes an evaluation through three components: - Data: examples to evaluate. - Task: a function mapping an input to an output, through a model call, an agent, or other computation. - Scorers: one or more functions assessing the result. Braintrust executes the specification, handling caching and parallelism through matching TypeScript and Python interfaces. The declaration also makes the eval portable: developers can bundle task and scoring functions for interactive debugging in the UI, or invoke a saved evaluation through an API to try another model.

Customer requests exposed details a basic results table would miss. To correlate latency with token counts, for example, separate task tokens from judge tokens. When a language model scores another model, both consume tokens. Mixing them confuses application work with evaluation overhead and obscures the relationship between task usage and latency.

Three adoption choices shaped the product: - Keep data in the customer’s cloud. Early users including Zapier, Coda, and Airtable reportedly needed this to adopt Braintrust. Hybrid deployment was designed in from the outset; serverless helped contain failures without server or Linux-process restarts. - Meet product builders in TypeScript. An early request from Olmo prompted a weekend SDK implementation, followed by feedback that shaped its ergonomics. - Solve the hardest workflow first. At Figma, existing tools covered observability and the team could build a prompt playground. Evals remained harder, making them the starting point for expansion into adjacent capabilities.

Explaining improvement over time exposed defects that daily use missed. Preparing a joint conference talk required a customer to reconstruct a longer history of experiments. That uncovered rough analysis workflows and brought the customer into direct conversations with engineers. By the recording, the experiments page supported dynamic filtering, grouping, and scatter plots.

Choose durable problems before technologies

Specialized tools and vertical integration each have benefits. The strategic risk is building deeply around a capability customers stop needing. The guest recalls advice that a startup gets only a few technology bets and should choose them carefully. Even Braintrust’s influence on code structure invites disagreement: an interviewer sees something framework-like, while the response treats avoiding log-to-eval transformation as a reusable engineering lesson independent of the product.

Vector databases make the tradeoff concrete. At Figma, the difficult part of deploying vector search was described as the adjacent application data: who could access which design files and components. Search must respect permissions and connect with existing data. The guest therefore favored adding vector search to systems already encoding that complexity, rather than rebuilding the application around a new store.

The database thesis is that a new storage model is especially compelling when it demands a new execution model. Databases compile and execute queries as well as store data. Snowflake’s separation of storage and compute, and Databricks’ treatment of arbitrary code, are offered as problems requiring coordinated work across both layers. The guest argues that vector search’s constrained nearest-neighbor query pattern did not require the same breadth of redesign at the time. The interviewer mostly agrees but wants to hear the opposing case.

Fine-tuning receives a similar treatment: the valuable outcome is better performance from data; fine-tuning is one method. The alternatives discussed are: - Fine-tuning: adapt a model to the use case. - DSPy-style optimization: optimize prompts automatically. - Manual refinement: improve wording and examples supplied in the prompt, then measure with evals. A business tied to one method is vulnerable when larger context windows or newer models change which approach works best. The guest regards a quality improvement as a stronger proposition than merely matching another approach at lower cost or latency.

On the September 25 recording date, the guest reports little fine-tuning among customers and fewer production deployments of fine-tuned models than six months earlier. That observation is limited to the customers visible to the speaker and explicitly open to change. Demand for automatic optimization remains. The usability argument separates DSPy’s optimization goal from its programming interface: a PyTorch-like style may suit ML engineers while adding friction for TypeScript and Next.js developers. The guest favors simpler code or English-based interfaces and invites collaboration with optimization builders.

Model choice includes the endpoint

Within the customer segment visible to the guest, usage before Claude 3 was almost entirely OpenAI. Haiku offered an attractive price, speed, and capability tradeoff, giving Anthropic a foothold. By the recording, Sonnet was popular for prototypes and side projects, and developers were evaluating both providers for new projects. OpenAI still served an overwhelming majority of observed customers. These are historical impressions, not measured market-wide shares.

Tool compatibility lowered switching costs. Tool calling connected prompts to application code, and Anthropic adopted the JSON Schema format already used by OpenAI for tool definitions. Developers could reuse those definitions without reworking all their calls. The guest also credited OpenAI’s availability, rate limits, and reliability as an operational advantage alongside model quality.

Large companies did not universally use model labs through their public-cloud partners. Early Azure deployments could require load balancing across endpoints with different credentials and model availability, while capacity and access to newer models could lag direct access. Braintrust’s gateway helped present a simpler endpoint over complicated deployments. For research and development, endpoint management competes with experimentation time, making direct lab APIs particularly attractive.

Reasoning models: absorbed complexity versus explicit control

The arrival of o1 prompts a tentative prediction: reasoning logic expressed through elaborate agent graphs and interacting roles may move into models. The guest sees partial validation of this direction in o1, drawing on Impira’s experience of model progress overturning engineering assumptions. Today’s model weaknesses are an unstable foundation for a product bet. The prediction remains explicitly uncertain.

The disagreement exposes two engineering priorities: - Let models absorb reasoning complexity. Better reasoning may eliminate external orchestration built to compensate for weaker models. - Keep explicit control over execution. Businesses may prefer chains of model calls when hidden reasoning is expensive, opaque, or moving in the wrong direction. The reply accepts the transparency problem but questions whether it is permanent. Future open reasoning models might offer more control; that remedy is speculative too.

Two adoption claims need to remain separate. The earlier decline concerned fine-tuning in production, not open-source models generally. Open-source production usage was separately described as small within the observed customer base. Availability remained contested: one participant pointed to Together and Fireworks as ways to serve open weights; another reported bugs in personal benchmark runs and judged OpenAI more reliable. These were operational experiences, not a controlled provider comparison.

The inference-business discussion distinguishes software providers from hardware bets. The respondent reports high margins at some software inference companies but leaves their durability open. Early traction and customer relationships could support additional products if initial margins erode. The enduring opportunity is solving inference problems; whether a standalone inference API captures that value remains unsettled.

Keep a code core and make small AI features easy

A Slack bug report becoming a Linear ticket with an automatically generated title is useful even when its user does not know how it works. Loom’s suggested recording titles help even when they need editing. These small features lead to a practical development constraint: if adding an AI call requires editing five files, developers will add fewer of them.

The guest distinguishes three workload shapes: - Single-prompt transformations: tasks such as many summaries, estimated at roughly half the use cases visible to the speaker. - Simple agents: a prompt with tools, operating as an enhanced chatbot. - Advanced agents: longer-running work, loops, or behavior beyond that simple prompt-and-tools pattern.

Across collaborators, the reported trend over the preceding six months was away from orchestration that was difficult to debug and run, toward familiar application code. The preferred workflow used Next.js or FastAPI, conventional logic between model calls, prompt refinement, and evals in continuous integration. Designing around the user experience creates ordinary code between model calls: interface code engages the user and shapes the interaction while the model works.

An interviewer names the distinction code core versus LLM core: ground the application in an established codebase and introduce model calls where they help. Generating and executing code can fit this approach. It differs from the Voyager-style pattern of storing generated code for future reuse, which the guest did not report seeing among known customers. The preference is for familiar systems that engineers can inspect and operate as model capabilities change.

Stay close to the problem and the people

Asked what else they would build, the respondent returns to accessible variant-type handling for observability. Useful technical ingredients may exist while vendors’ target customers, use cases, and budgets leave other needs underserved. High user frustration can coexist with low commercial value for an incumbent. Asked specifically about another AI project, the answer is model training, motivated by a desire to work closer to the technological frontier.

That ambition comes with a revised view of expertise. Deep database knowledge once made some successful data-tooling companies look unnecessary to the respondent. Experience changed the judgment: proximity to a customer’s problem can be as valuable as mastery of the underlying system. In AI, the respondent now feels closer to users’ difficulties than to the model technology itself.

A hypothetical fair-market-value acquisition offer from OpenAI receives a no, without ruling it out forever. The reason is the current working environment: a problem worth caring about, customers worth serving, and colleagues worth working with. Independence matters, but the respondent places greater weight on enjoying that combination.

Asked about Alana’s effectiveness, the respondent—her spouse—credits sustained care and relationships formed before someone becomes an obvious investment prospect. Alana and Elad stayed in touch during the period at Figma. Her technical work offers a complementary lesson: with a computer-science education and product-management experience, she built her own software stack and combined product, design, and engineering skills to solve recurring inefficiencies.

The episode closes with a then-current hiring call for software engineers, salespeople, a developer-relations hire, and another designer, with a strong preference for San Francisco and some exceptions. AI software builders are invited to try Braintrust, and former users to revisit it or get in touch; the company reports capacity to work with more customers.