Braintrust

Braintrust builds a developer platform for evaluating, debugging, and improving AI products. Introduced in September 2023 by Ankur Goyal, its initial stack combined evaluations, versioned datasets, and a collaborative prompt playground. It subsequently added production logging, deployable Functions, the Brainstore database, and the Loop AI assistant. By September 2026, its latest expansion connected investigations of recurring agent behavior with diagnoses of individual runs, in public preview.[2][4][6][8][9]

The platform's evaluation-centered approach grew out of earlier work: Goyal recalled building internal evaluation tools at both Impira and Figma. His account of Impira explains the practical problem those tools addressed—deciding between models when isolated examples showed different strengths and weaknesses.[5][1]

Before Braintrust, Goyal ran engineering at MemSQL, later called SingleStore, and founded Impira to make unstructured data easier to use. Impira eventually focused on extracting information from PDF documents. In his October 2024 interview, Goyal recalled that text and document models challenged its largely computer-vision-based approach. The team retooled the product around LayoutLM, while experiments with GPT-3 further changed his assessment of its technology. Impira was acquired into Figma in December 2022, where Goyal went on to lead AI work.[1]

Goyal recalled that Impira spent two or three months debating model choices before prototyping an evaluation system with Snowflake and scripts, then shipping a new model roughly two weeks later. Evaluations made gains and regressions visible together: the team could identify newly workable invoice cases while targeting cases that had become worse. He described this as an organizational benefit as well as a technical one, replacing competing anecdotes with a shared basis for decisions.[1]

Braintrust's September 2023 launch introduced tools for scoring AI outputs, comparing experiments, investigating failures, and tracking performance over time. Goyal started the company with assistance from Elad Gil, who helped incubate the initial evaluation product and team. The launch named Coleen Baik as founding designer and Manu Goyal as founding engineer; David Song, who had built evaluation solutions at mem.ai and was part of Gil's team, also helped.[2]

The evaluation interface described by Goyal in 2024 used a declarative specification: data, a task function that turns input into output, and one or more scoring functions. Braintrust handled execution, caching, and parallelization, with matching interfaces in TypeScript and Python. The UI let teams examine individual improvements and regressions rather than relying only on an aggregate score. Automatically versioned datasets preserved the examples on which evaluations depended, while experiment history gave teams reference points for discussing earlier results.[1][2]

Goyal credited Brian from Zapier, Braintrust's first user, with repeatedly challenging early demos until the developer experience was useful enough to show his team. Separately, Olmo's request for a TypeScript SDK prompted Goyal to build one over a weekend. Olmo became its first user, and Goyal credited him with shaping its ergonomics. Preparing their AI Engineer World's Fair talk also brought Olmo into Braintrust's office, where he worked directly with engineers on bugs and usability problems. In the 2024 interview, Goyal estimated that more than 75% of users running evaluations used the TypeScript SDK.[1]

Braintrust supported deployment within a customer's cloud from its launch. Goyal said the company engineered for this hybrid model from the start because early users—including Zapier, Coda, and Airtable—needed their data to remain in their own clouds. The launch also offered a playground where technical and non-technical colleagues could test prompts against datasets or prior experiments.[2][1]

Production logging extended the evaluation workflow. Goyal explained that using the same function abstraction for application execution and evaluation allowed logs to capture information in the same trace-and-span format used by evaluations. This reduced the work of transforming production logs into test datasets. The playground likewise expanded from prompt comparison into collaborative editing, custom scoring, experiment execution, and saving prompts for deployment into code.[1]

On 8 October 2024, Braintrust introduced Functions: tools, prompts, and scorers authored in a codebase and uploaded with braintrust push. Braintrust bundled, versioned, deployed, and sandboxed that code in its own cloud or the customer's, making the functions available through its UI and API. This connected code-based development with interactive experimentation and deployment.[4]

Braintrust introduced Brainstore on 3 March 2025 to handle AI logs and their full-text and semi-structured queries. The Rust database uses object storage for data, Postgres for metadata, and Redis for distributed locks, with stateless nodes and no durable local disk state. Its search architecture combines Tantivy's inverted index and columnstore with a custom object-storage write-ahead log. Reads include newly written log entries alongside indexed data; background compaction moves those entries into Tantivy indexes. This makes writes immediately available to reads with strong consistency.[6]

Goyal credited Manu, Austin, and Deeks (Mike) with engineering and operational contributions to Brainstore. Manu brought experience from BigQuery, Dropbox storage, and Nuro infrastructure; Austin had worked on model execution and incremental query processing at Impira; Deeks had previously used Braintrust at Instacart. Friends from Turbopuffer, Neon, and Warpstream helped work out the design.[6]

In a company-run March 2025 benchmark over 3,925,153 documents of 25 KB each, a full-text search for “out of memory” took 240 ms in Brainstore, 78,963 ms in an unnamed open-source warehouse, and 20,789 ms in an unnamed observability competitor using that warehouse. Brainstore and the warehouse ran on identical AWS c7gd.8xlarge hardware with NVME disks, but Brainstore cached locally and persisted to object storage while the warehouse stored data locally. In a separate write test using a batch of 100 documents of 100 KB each, the warehouse acknowledged writes faster—331 ms versus 1,780 ms—while making them visible to reads more slowly, at 2,678 ms versus 1,780 ms. These results describe the tested workloads rather than a universal speed advantage.[6]

On 31 March 2025, Brainstore became the default for Braintrust's hosted UI and API. Self-hosted deployment defaults were unchanged; operators could enable it themselves. The migration also made a consequential query distinction explicit: returning only matching spans versus returning every span from a matching trace. Brainstore defaulted to spans, with a temporary compatibility period retaining the old trace behavior for requests that did not explicitly select Brainstore.[7]

By November 2025, Loop was available to all Braintrust users as an integrated AI assistant. It supported semantic trace search, BTQL query generation, datasets and scorers derived from production traces, experiment analysis, prompt suggestions, and custom charts. In Braintrust's customer account, Retool used Loop to find recurring errors and prioritize issues by impact. The company reported that it helped Retool identify multi-page support as its leading user request and prioritize fixes for context-window overflow; Allen Kleiner, Retool's AI Engineering Lead, described using it weekly to synthesize log data.[8]

On 3 September 2026, Braintrust added Patterns, Debugger, and an enhanced Loop experience in public preview. Topics classified broad themes across traces; Patterns used scheduled Loop investigations to identify recurring failures or opportunities with supporting traces and recommended next steps. Debugger analyzed an individual run's spans, tool calls, results, and model outputs to propose likely failure modes. Loop added persistent investigations and scheduled analyses, with approval required to create or edit resources such as prompts, scorers, datasets, and dashboards. Developers and domain experts retained responsibility for validating evaluators, approving changes, and deciding what shipped.[9]

The launch named Elad Gil and Alana Goyal of Base Case Capital as leaders of Braintrust's initial funding round. On 13 December 2023, Braintrust announced a $5.1 million seed round led by Saam Motamedi at Greylock, reporting $8.3 million in total capital raised. It named Zapier, Coda, Airtable, and Instacart as customers and reported doubling its team and customer base in the six weeks since a previous round announcement.[2][3]

On 8 October 2024, Braintrust announced a $36 million Series A led by Martin Casado at Andreessen Horowitz and reported $45 million in total funding. Participants included existing investors Elad Gil, Greylock, and Basecase, alongside Datadog, Databricks Ventures, and individual investors including Guillermo Rauch, Simon Last, Bryan Helmig, Greg Brockman, and Arthur Mensch. The company also reported that the average team using Braintrust ran more than ten experiments a day.[4]

On 17 February 2026, Braintrust announced an $80 million Series B led by ICONIQ, with returning investors including Andreessen Horowitz, Greylock, Elad Gil, and basecase capital. It named Notion, Replit, Cloudflare, Ramp, and Dropbox as organizations using the platform. Goyal described the round as supporting Braintrust's expansion into production AI observability, with plans to grow engineering and go-to-market teams, open additional offices, and develop new products.[5]