Galileo Labs

Galileo Labs is the organization behind Galileo’s work on evaluating generative AI applications and agents. Its products combine model evaluation, production monitoring, and real-time guardrails; its Agent Leaderboard separates an agent’s ability to call tools correctly from its ability to complete a user’s goals. Pratik Bhavsar, identified as working on evaluations and leaderboards at Galileo Labs, led the published explanation of the benchmark’s second generation.[1][7]

On July 16, 2025, Galileo announced its Agent Reliability Platform, available to start using for free. Its Graph Engine maps agent paths and multi-agent workflows, with graph, timeline, and conversation views. The Insights Engine analyzes logs and metrics using evaluation reasoning models, tying failure diagnoses and recommendations to particular spans, traces, or components. Luna-2, a family of small language models, supplies real-time guardrails. Supported frameworks included CrewAI, LangGraph, OpenAI’s Agent SDK, and LlamaIndex.[7]

The platform launch also introduced Agent Leaderboard v2, which Bhavsar explained in a July 17 article. Galileo’s first version had tested more than 30 LLMs across 14 datasets. Bhavsar described scores clustering above 90%, limited scenario complexity, static interactions, and insufficient domain isolation as reasons to change the benchmark. He also raised possible contamination of public benchmarks by model training as a concern.[7][1]

Version 2 launched with 100 synthetic scenarios in each of five domains: banking, healthcare, investment, telecom, and insurance. Each conversation contains five to eight interconnected goals, with prior context, changing requirements, ambiguous requests, and missing or irrelevant tools. A generative user simulator adapts to the tested agent’s responses, while a tool simulator supplies outputs. The agent interacts with simulated tools rather than real APIs or sensitive customer data.[1]

One banking scenario combines reporting a lost credit card, checking a mortgage payment, arranging automatic utility payments, locating a branch near a Paris hotel, obtaining exchange rates, and configuring travel alerts. Completing it requires maintaining context, coordinating tool dependencies, and confirming each outcome.[1]

Action Completion measures whether the agent fulfills and clearly answers or confirms every user goal. Tool Selection Quality measures appropriate tool choice, correct parameters, and avoidance of unnecessary or erroneous calls; Galileo uses Claude to evaluate individual tool calls. In Galileo’s simulated results as of July 17, 2025, GPT-4.1 led with 62% average Action Completion. Gemini-2.5-flash scored 94% on Tool Selection Quality but 38% on Action Completion—a concrete illustration of why the benchmark measures both.[1]

The agent platform followed Galileo’s earlier expansion from NLP and computer vision tooling into LLM application development, and its development of specialized models for production evaluation.[3][4][7]

Galileo dates its inception to early 2021. Its co-founders are Vikram Chatterji, Yash Sheth, and Atindriyo Sanyal. Battery Ventures records an investment in 2022 and describes Galileo’s earlier offering as a machine learning data intelligence platform for data scientists working with unstructured data.[4][2][8]

Galileo announced LLM Studio as generally available on September 18, 2023. Chatterji described the preceding focus as NLP and computer vision. In his account, conversations with financial-services, healthcare, and AI-native customers exposed the need for a toolchain covering prompt experimentation, comparisons among LLMs, retrieval-augmented generation (RAG), and fine-tuning.[3]

LLM Studio had three modules. Prompt compared prompt templates, models, and parameters, with automatic version controls for collaboration. Fine-Tune identified problematic training data, including incorrect ground truth, low coverage, and poor-quality examples. Monitor connected production measures such as engagement, cost, and latency with evaluation metrics such as Uncertainty, Factuality, and Groundedness, alongside tracing and alerts. A shared metrics store supported evaluation across all three modules.[3]

Galileo’s 2023 Chainpoll method used chain-of-thought prompting for LLM-based hallucination evaluation. Chatterji later described its production cost and latency as reasons to pursue specialized evaluation models. After nearly a year of research and development, Galileo launched Luna on June 5, 2024, making it available by default to customers at no additional cost at launch. Luna powered automatic evaluation metrics in Evaluate and Observe and supported Protect, which intercepted application inputs and responses in real time.[4]

The Luna paper, first submitted June 3, 2024, was authored by Masha Belyi, Robert Friel, Shuai Shao, and Atindriyo Sanyal. It introduced a 440-million-parameter DeBERTA-large encoder fine-tuned to detect hallucinations in RAG responses: claims unsupported by the retrieved context.[6]

In his technical explanation, Bhavsar described generating responses to open-book question-answering inputs with GPT-3.5 and Claude-3-Haiku. GPT-4-turbo annotated which context sentences supported response claims, with unsupported tokens labeled as hallucinations; re-annotation and manual inspection checked the labels. Luna was trained on QA RAG examples, while evaluation on RAGTruth also tested generalization to summarization and data-to-text tasks.[5]

Luna separately chunks context and response so response sentences can be checked against the available context. Galileo described serving an ONNX-traced model through NVIDIA Triton with TensorRT and Business Logic Scripting, reporting processing of up to 16,000 input tokens in under one second on an NVIDIA L4 GPU. In separate launch tests, the company reported evaluation of production traffic at 97% lower cost than GPT-3.5.[5][4]

On October 15, 2024, Galileo announced a $45 million Series B led by Scale Venture Partners, with participation from Databricks Ventures, Premji Invest, Amex Ventures, Citi Ventures, ServiceNow, and SentinelOne. Existing investors Battery Ventures, Walden Capital, and Factory continued their support. The announcement presented an Evaluation Intelligence Platform spanning fine-tuning, development, testing, monitoring, and security, with Luna evaluation models and Galileo Wizard, an inference engine built for evaluation.[2]

At that announcement, Galileo reported 834% revenue growth since the beginning of 2024, a quadrupling of enterprise customers, and the addition of six Fortune 50 companies. It named Twilio, Comcast, HP, and ServiceTitan collectively as customers and partners. In July 2025, Galileo reported that Luna-2 was helping large enterprises protect hundreds of agents and process hundreds of millions of queries daily through dozens of guardrails.[2][7]

Battery Ventures lists Galileo as acquired by Cisco.[8]