Pratik Bhavsar

Pratik Bhavsar created Galileo’s Agent Leaderboard, an evaluation framework that distinguishes an AI agent’s ability to select tools from its ability to complete a user’s goals. His work includes multi-turn agent benchmarks, technical guides, and evaluation workflows for coding agents. On his personal site, he reports working at Cisco on tokenomics and agent evaluations since May 2026, following a June 2023–May 2026 role leading open-source evaluations and developer relations at Galileo.[4][3][2]

In his July 17, 2025 Agent Leaderboard v2 launch article, Bhavsar explained why Galileo moved beyond basic tool-calling tests. Scores were clustering above 90%, making models harder to distinguish; static scenarios offered little context carry-over or ambiguity; and aggregate results did not isolate performance by business domain. He also raised possible benchmark leakage as a concern. His framing was direct: “What we measure shapes AI.”[3]

V2 launched with 500 synthetic support scenarios across banking, healthcare, investment, telecom, and insurance. Each conversation contains five to eight interconnected goals, with prior context, changing requirements, and missing or irrelevant tools. The simulation pairs the model being tested with a user simulator that adapts the conversation and a tool simulator that generates responses from predefined schemas. Agents do not interact with real APIs or sensitive customer data.[3]

One banking scenario asks an agent to report a lost credit card, verify a mortgage payment, arrange automatic bill payments, locate a branch near a Paris hotel, obtain exchange rates, and configure travel alerts before departure. Completing it requires keeping track of separate requests, coordinating dependencies, and confirming each outcome.[3]

The benchmark separates Action Completion—whether every user goal receives a complete answer or confirmation—from Tool Selection Quality, which evaluates tool choice, required arguments, and unnecessary or erroneous calls using a dedicated Claude evaluator. Galileo’s reported results on July 17, 2025 illustrated the difference: GPT-4.1 averaged 62% Action Completion, while Gemini-2.5-flash scored 94% on tool selection but 38% on completion. These were results from the synthetic support benchmark.[3]

Bhavsar discussed agent evaluation on Latent Space in “Ranking Agentic LLMs”. In the v2 launch article, he connected that interview to the limitations of v1 and lessons from building the leaderboard.[1][3]

On May 19, 2026, Bhavsar and Paul Lacey coauthored the introduction to Eval Engineer, an alpha skill bundle for Claude Code and OpenAI Codex. It turns Galileo evidence into three reviewable artifacts: a diagnosis, a bounded fix plan, and a verification plan. Their billing-support example narrows an incorrect escalation to an overly broad tool description and a missing routing rule, then checks the proposed edits against local tests and fresh canary evidence.[5]

The coauthors reported that compact debug packets made Eval Engineer more reliable than raw traces by providing stable fields for metrics, failing cases, expected and actual behavior, and trace links. They kept product decisions, merges, and broad architecture changes with humans. They also cautioned: “A production failure can become a candidate eval case, but it should not silently become permanent ground truth.”[5]

In his May 15, 2026 article on AI brittleness, Bhavsar argued that controlling output randomness does not establish reliability across differently phrased requests: “Brittleness across paraphrased inputs cannot be fixed with temperature.” He recommended testing five to ten semantically equivalent paraphrases per evaluation prompt, preferably with rule-based or human verification where wording carries domain-specific meaning. He acknowledged residual inference variance even at temperature zero and argued that production thresholds should depend on failure cost, recoverability, and the domain.[6]

Before joining Galileo in June 2023, Bhavsar worked on search and applied language systems. In his career account, he describes building NLP extraction, sentiment, and quantitative machine-learning workflows at Morningstar, and semantic search and recommendations at TaskHuman within 2017–2020. He then contributed to Jina AI’s open-source neural-search framework from September to December 2020. At Enterpret, from February 2021 to May 2023, he reports working as a founding engineer on semantic search, reranking, text generation, MLOps, and customer-feedback NLP pipelines.[2]

Bhavsar also founded Maxpool in 2019 as an AI practitioner community. His personal site lists five GenAI books covering agents, multi-agent systems, retrieval-augmented generation, LLM-as-a-judge, and evaluation engineering. He describes his approach to agent development as a feedback loop: “I build agent systems where evaluation is not the ending. It is the feedback signal for the next version.”[2]