Evals

Evals are evaluations of AI models and the systems built around them. They pair defined tasks with ways to judge the results, making quality something engineers can investigate and compare. For an assistant, that might mean answering a question correctly; for a coding model, producing a program that passes tests; for a booking agent, creating the right reservation. The useful question is concrete: what behavior needs to work, and what evidence would show that it did? OpenAI Evals is one framework within this broader practice.[2][3][6]

A practical evaluation separates the test data, the system being tested, and the grader. Braintrust’s evaluation quickstart illustrates this with a movie matcher: the input is a plot description, the task generates a movie title, and an exact-match scorer compares that title with the expected answer. This separation lets a developer change a prompt or model while keeping the test cases and scoring method consistent.[7]

The grader should reflect the task. Exact matching is useful when a definite answer is expected, but it cannot by itself assess open-ended writing. In Hamel Husain’s Lucy example, a real-estate assistant translates requests into CRM queries. Tests check listing counts in simplified scenarios, while a separate check catches internal UUIDs leaking into user-facing responses. These tests make particular failures visible without pretending that one assertion captures the assistant’s entire quality.[7][1]

Different grading methods answer different questions:[3][5][2]

• Executable checks can verify a program’s behavior or a database state, as illustrated by HumanEval and Anthropic’s agent-evaluation guide.[3][2]
• Human review can assess outputs that require judgment, one of the grading methods discussed in Anthropic’s guide.[2]
• An LLM judge can score an answer, compare two answers, or evaluate against a reference—the designs examined in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.[5]

These methods can complement one another: a response may satisfy a mechanical constraint while still being confusing, or read beautifully while describing an action that never happened.[3][5][2]

Several milestones show how evaluation asks different kinds of questions. HumanEval, introduced in the 2021 paper Evaluating Large Language Models Trained on Code, measures functional correctness of programs generated from docstrings.[3]

Stanford CRFM’s 2022 HELM announcement takes a broader view of language-model comparison, combining accuracy with calibration, robustness, fairness, bias, toxicity, and efficiency. HELM explicitly evaluates the model rather than a scenario-specific application, and documents gaps in its coverage. Neither milestone represents the beginning of AI evaluation; they illustrate influential approaches to evaluating language models.[3][4]

The 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena examines whether models can grade conversational answers. It studies pairwise comparison, single-answer scoring, and reference-guided grading, while identifying position bias, verbosity bias, and reasoning limitations. Swapping answer order and checking whether the preference remains consistent is one mitigation. The larger lesson is that the judge is also a system to evaluate: agreement with humans in a particular study does not establish reliability for every task or rubric.[5]

An agent can make several decisions, call tools, and change its environment before finishing. Anthropic’s guide to agent evals therefore distinguishes the recorded trajectory from the final outcome. A booking agent saying “your reservation is confirmed” is insufficient evidence; the reservation must exist in the database. Evaluating that agent measures the model together with its tools, harness, and environment, so its score cannot automatically be assigned to the underlying model alone.[2]

Project Vend supplies a memorable example of why consequences matter. In 2025, Anthropic and Andon Labs had Claude Sonnet 3.7 operate a small office shop for about a month, with humans handling physical tasks. Supplier communication and procurement were mediated: the email tool simulated sending rather than sending real emails, and Andon Labs served as the wholesaler without that arrangement being apparent to the agent.[8]

An employee’s request for a tungsten cube led to specialty-metal inventory, which the agent sometimes offered below acquisition cost. Fluent customer interactions could coexist with decisions that reduced the shop’s value. The experiment’s employee customers included people actively probing the agent, so the report is an exploratory case rather than a representative retail benchmark.[8]

Repeated trials reveal another distinction: can it succeed, or does it succeed reliably? Anthropic’s agent-evaluation guide describes two measures that serve different needs:[2]

• pass@k concerns the chance of at least one success across k attempts. A workflow that allows several candidate solutions may care about finding one successful result.[2]
• pass^k concerns success in every attempt. A workflow expected to execute correctly each time needs evidence of consistency.[2]

A single successful run cannot answer both questions.[2]

Useful evals often begin with examining actual failures. Husain recalls that many errors in Lucy were small mistakes in the final output. He made that output editable so people could correct it, and the corrections could become fine-tuning data. The observation led to an interface decision and a way to collect better examples. It illustrates how evaluation can shape the product itself, alongside changes to prompts or models; Husain does not establish a measured performance gain from that decision.[1]

An evaluation suite can then support two jobs, distinguished in Anthropic’s guide:[2]

• Capability evals test difficult behavior the system is still learning to handle.[2]
• Regression evals protect behavior it already performs successfully.[2]

Anthropic recommends maintaining both, with established capability tasks becoming regression checks. This gives a team a way to pursue harder problems while watching for losses in existing functionality.[2]

The tests themselves need inspection. Anthropic reports finding a Terminal-Bench task whose grader assumed a script path that the instructions never specified. Such a mismatch can turn an acceptable approach into an apparent failure. Clear tasks, reference solutions, and examination of failed runs help distinguish a system defect from a defective test. A score becomes useful when engineers understand what produced it.[2]

Tools make these comparisons easier to run and revisit. OpenAI Evals provides a framework and registry, including support for custom and private evaluations. In a Latent Space interview, Ankur Goyal demonstrates comparing Braintrust experiments after changing a prompt to request a shorter response. The underlying engineering habit is straightforward: keep examples and grading criteria visible, compare candidate changes, and inspect the outputs behind the scores.[6][9][1]