Olmo Maldonado

Olmo Maldonado is a staff engineer at Braintrust, known online as ibolmo. At Zapier, he helped develop AI Zap Builder and Copilot and described how his team used evaluations to define correct behavior, detect regressions, and decide when to adopt a different model.[2][5][4]

On November 7, 2024, Maldonado announced his departure after more than seven years at Zapier and his intention to join Braintrust to help companies build AI products. The move followed close collaboration between the teams. He later helped host the Evals track at the AI Engineer World’s Fair 2025.[5][1]

Before joining Braintrust, Maldonado presented Zapier’s evaluation workflow with Ankur Goyal at the AI Engineer World’s Fair 2024. His central point was practical: “evals are the way that we make decisions.” He recommended involving product managers because deciding whether a generated automation works is both an engineering and a product problem.[4]

For AI Zap Builder, which generates workflows from prompts, the team translated product priorities into evaluation criteria: choose the right triggers and actions, put steps in the correct positions, support the top 25 apps, and handle Zapier’s Paths and Filters. Their in-house framework, built with Braintrust’s help, seeded coverage with synthetic data from corporate accounts. It loaded Braintrust-hosted datasets, exercised AI providers through a custom runner in CI and manual runs, and graded outputs with both logic-based and LLM-based checks.[4]

Maldonado reported that the team expanded from seven manually run unit tests to more than 800 tests run with merge requests and continuously, with alerts for regressions. He reported nearly 300% improvement in accuracy, explicitly cautioning that this did not mean 100% accuracy. He did not specify the baseline or calculation behind that improvement.[4]

Copilot extended the single-shot Builder into an interactive chat inside the editor, including testing steps and configuring fields. Multiple tool calls made it harder to identify the critical path and understand what needed improvement. Maldonado explained how Braintrust tracing let developers inspect individual chat completions’ inputs, outputs, token usage, and response time rather than relying on the final answer alone.[4]

Maldonado’s team initially used GPT-3.5 Turbo to get Copilot to market quickly, then selected GPT-4 Turbo for its message router despite slower responses because of its routing accuracy. Switching to GPT-4o exposed regressions: he reported that scores previously at 80% or better fell below that level. The faster model needed changes to the application’s prompts and API settings before adoption.[4]

Maldonado credited colleague Maggie Cody for the investigation he demonstrated. The comparison he demonstrated contained 22 regressions involving unwanted answers or apparent failures to follow system instructions. His diagnosis was that the prompts had become too tailored to GPT-3.5 Turbo. Relaxing their elaborate instructions helped restore earlier score levels.[4]

The revised setup used tool_choice: "auto"; requiring tool calls remained a planned experiment. After prompt and API-setting changes, the team adopted GPT-4o. Maldonado reported that streaming Copilot latency fell from around 14 seconds to three seconds, without specifying whether that interval measured first output or a complete response.[4]

In an October 2024 interview, Goyal recalled that Maldonado asked, “where’s the TypeScript SDK?” when Braintrust offered only Python. Goyal built the SDK over a weekend, and Maldonado became its first user. Goyal credited his feedback with helping shape the product’s ergonomics.[6]

Goyal also recalled Maldonado working from Braintrust’s office while preparing their joint talk, finding bugs and usability problems and discussing them directly with the responsible engineers. Reviewing the team’s usage over time exposed limitations in the experiment-analysis interface that were less apparent during everyday use.[6]

Maldonado’s AI work followed broader backend engineering at Zapier. In his departure announcement, he recalled contributing to its first public API, embedding the platform in partner products, developing distributed systems and service APIs, and building a system to manage millions of Zaps. These were team contributions alongside his work on workflow reliability and scalability.[5]

Earlier, Maldonado founded Code RGV and created GitHub Spy there to review students’ public GitHub activity. The Node.js script ran periodically and published activity to Keen.IO for analysis. His personal site lists the accompanying post in October 2016; the repository was archived in December 2019. This work provides concrete context for his experience building developer tools and teaching software development.[3][8]