Benchmarks 201: What a Leaderboard Score Actually Measures

A leaderboard begins with a failed comparison

Hugging Face’s reinforcement-learning team wanted to compare its results with published papers but could not make the comparison work. Its response was practical: manually evaluate models under a shared procedure and publish a small leaderboard. Community interest turned that project into the [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard). Roughly six weeks later, Clémentine Fourrier and Nathan Habib took responsibility for making it a production service.

Fourrier came to evaluation through geology. After graduating as a geology engineer in 2015, she moved into computer science, worked on brain-illness prediction and natural-language processing, and completed an INRIA-funded PhD after working there as an engineer. Geology shaped her view of machine learning as an experimental science, whose claims require scrutiny through measurement.

Her first Hugging Face project explored foundational graph transformers. The team stopped after six months because of resource constraints, leaving reusable graph machine-learning artifacts. Fourrier then spent a month reading evaluation papers and helped establish the internal evaluation suite that became LightEval. The leaderboard began separately before joining that work.

By the interview, Fourrier reported that leaderboard V1 had evaluated 7,400 models, mostly community submissions, and accumulated roughly 800 support and discussion threads. Some startups had written to credit their rankings with helping secure funding. Those were the startups’ accounts, but they show the stakes attached to the scores. Swyx frames the scientific contribution as independently running reproducible comparisons, rather than relying on each developer’s published evaluations.

When a benchmark stops separating models

The initial benchmark choices followed research practice. Edward Beeching and Lewis Tunstall looked for evaluations that recurred in major reinforcement-learning papers. Fourrier names MMLU, ARC Challenge, and GSM8K as common standards, while noting that GSM8K may have joined later. Subsequent additions drew on community requests for missing capabilities; collaboration with the reinforcement-learning team continued into V2.

A dataset need not be flawless to provide useful signal. Many older evaluations relied on underpaid crowd work, and manually checking thousands of examples is difficult. Even relatively strong starting points such as MMLU and ARC Challenge need closer inspection when performance saturates—when high scores leave little room to distinguish models.

One danger is contamination: exposure to benchmark material can let a model learn the answer key, including its mistakes. The MMLU critique is that some human answers marked wrong are actually correct, so a model exceeding a human score may have learned erroneous answers rather than demonstrated superior knowledge. A high score alone does not establish contamination, however. Genuine capability gains can also make a benchmark too easy.

When a participant worries that an MMLU successor might last only a year, the response is to accept another leaderboard revision. Benchmarks have useful lifetimes. The archived V1 preserves performance curves over time; replacing its tasks allows the comparison to keep distinguishing stronger models.

Correctness, preference, and the judge’s taste

Automated benchmarks give every model the same evaluation procedure, but their conclusions remain bounded by the tasks. Multiple-choice accuracy cannot establish whether a model writes good poetry. Three approaches extend that view: direct application testing, preference comparisons in systems such as [LMSYS Chatbot Arena](https://chat.lmsys.org/), and paid expert annotation with an explicit rubric. Experts can check specified criteria; their expense creates demand for large language model (LLM) judges, which evaluate other models’ responses.

The critique of model judges centers on biases in what they reward: - Family preference: favoring outputs from their own model family. - Position bias: favoring the answer presented first. - Verbosity preference: rewarding longer answers. - Unreliable numerical grading: struggling to assign meaningful scores across a continuous range.

For applications that still require a judge, the recommendation is a small model such as [Prometheus](https://prometheus-eval.github.io/prometheus/) or Judge LM, used for relative rankings rather than numerical grades. The speaker advises against GPT-4 because its closed-source status limits reproducibility. Beyond individual grading errors lies another concern: developers optimizing against the same judges could make otherwise different models converge on the judges’ preferred behavior.

Human preference has a related measurement problem. An arena reveals what users ask and which responses they like, but an underspecified judgment of a “good” answer does not isolate factual accuracy. Crowdsourcing becomes more useful with clear criteria. Red teaming—testing for failures such as private-information disclosure—is offered as a case where the judgment can be more definite.

Sycophancy, or agreeing with users to win approval, makes that distinction consequential. A participant cites Anthropic’s sycophancy paper as finding that people prefer models that agree with them. Another argues that a positive disposition toward humans can be desirable; both recognize the need for truthful disagreement. A separate Cohere–University of Edinburgh paper is recalled as finding that evaluators can prefer an assertive false answer over a less assertive correct one, though the exchange gives no conditions or effect sizes. Optimizing production models for popularity could therefore reward behavior that undermines factual accuracy.

The practical advice is to test your own use case. A general leaderboard helps compare candidates; it cannot settle application fit. RavenWolf’s detailed qualitative evaluations are offered as an example of inspecting which questions models get right or wrong, despite a scope estimated at roughly 15 prompts. Multi-turn interaction remains a shared gap: one participant describes the available arena sample as mostly single-turn, and another acknowledges that automated benchmarks also tend to omit it.

V2: harder questions and explicit checks

MMLU Pro expands multiple-choice questions from four options to ten and incorporates expert review. IFEval takes a different approach: unit tests for language. Its prompts impose strict formatting instructions, and explicit checks determine whether the model followed them. This tests instruction following without requiring a semantic-similarity metric to decide whether two pieces of prose mean the same thing.

GPQA tests difficult science questions written by PhD experts, intended to be substantially more answerable by specialists than nonspecialists. Its access gate primarily blocks automated web collection: people can obtain the answers by creating an account and accepting the gate. Protecting a test set can also require refusing direct requests. A contributor to GAIA, an agent benchmark, recounts turning down institutions seeking its test-set answers for internal benchmarking, even when they invoked safety benefits.

The prompt and scorer are part of the experiment

The leaderboard adopted the MMLU prompt format used by [EleutherAI’s evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness), the library that runs its evaluations: Question:, the question text, Choices:, lettered options, and Answer:. Experiments removing labels and changing how choices appeared reportedly produced a 30-point spread on a 100-point scale. The tested models and full protocol are unspecified, so the magnitude applies to those experiments rather than every MMLU evaluation.

The smallest prompt contained only the question. The scorer then compared candidate answers’ log probabilities, selecting the answer the model assigned the highest likelihood. A more elaborate prompt explicitly listed the choices; the variant with letters in parentheses reportedly scored best across most tested models. Both approaches were intended to probe the same knowledge, yet the results depended heavily on how that knowledge was elicited. The team also collaborated with Outlines on an explanation of structured generation for evaluation.

Scoring adds another choice. HELM is described as asking the model to generate an answer letter and counting outputs outside the allowed set as wrong. Returning a word when the permitted answers are A through D fails the output contract. The leaderboard chose log-likelihood scoring for multiple-choice tasks because it was much cheaper for most tasks; inspecting a single token also made batching straightforward. The same questions can therefore impose different demands depending on the scorer.

Compute limits and implementation errors

Compute constraints shaped the task mix. The leaderboard used only the hardest, level-five MATH questions to distinguish stronger models without paying to run the full dataset. At the time, an evaluation used a node with eight 80 GB H100 GPUs and took approximately two hours for a model with 7 billion parameters, or twenty hours for one with 70 billion. The cited price of roughly $100 per node-hour was a public instance-price comparison, not Hugging Face’s internal bill.

Alongside MATH, MuSR, or Multistep Soft Reasoning, was described as one of the hardest evaluations. It tests reasoning over long inputs through tasks such as identifying the culprit in rule-generated murder mysteries. Few models reportedly exceeded random performance at the time. Generative tasks add a separate difficulty for maintainers: scoring must correctly recognize the answer within the model’s output.

DROP’s decimal-point bug shows how evaluation code can misrepresent capability. During V1 work, the team found that its implementation treated a period as an end-of-sentence token, truncating floating-point answers and marking them wrong. The team discarded the evaluation. V2 checks across varied models took roughly a month and covered prompt formatting, stray literal \n sequences, and tokenizer problems affecting generative tasks. Selecting relevant, stable benchmarks was only the beginning; fairness also required inspecting their execution.

The free community service ran on spare capacity in Hugging Face’s shared research cluster. Leaderboard jobs had the lowest priority and could be terminated when other research needed the resources. This kept incremental costs low but could halt the queue. User-funded inference endpoints were under consideration: LightEval already supported them, but the team needed to port that functionality into the harness. The additional engineering kept the proposal lower on the priority list.

Moving between inference providers introduces further variables. Custom GPU kernels, reduced numerical precision, and attention shortcuts can change performance despite identical starting weights. The leaderboard recorded model precision explicitly, sometimes listing a model at several precisions, while acknowledging small differences caused by hardware and batch size. Reproducibility includes execution conditions, not just the dataset.

Long context and agents need meaningful tasks

Useful long-context tasks require applying information across a substantial input. One example supplies a linguist’s grammar book for Kalamang, then asks about grammar and correct language forms. Another, attributed to Allen AI, uses novels published the previous year, with summaries and adversarial event descriptions written by people who read them. Its questions are intended to require understanding the whole book. Using recent novels was hoped to reduce contamination; their exclusion from training data was not established.

GAIA began with expected assistant work: browsing the web, extracting structured information, using tools for different modalities, and reasoning across information and tools. A contributor describes its design as a response to the artificial, closed environments encountered during GAIA’s development. Its proxy tasks stand in for practical assistant work. The team shared its construction recipe so others could contribute or build similar datasets, while recognizing that new tools can make existing tasks substantially easier.

Chollet’s ARC AGI challenge is distinct from the ARC Challenge subset of Allen AI’s dataset. Its appeal here is its logic problems and the prospect of learning patterns that generalize. The respondent rejects solving it as proof of artificial general intelligence, while valuing success as a potentially meaningful advance in reasoning. The useful question is what progress on the tasks demonstrates.

How to evaluate an evaluation

A practical benchmark review starts with inexpensive inspection, then moves to running models: 1. Inspect construction. Check whether examples came from humans or models, how annotators were paid, and whether their language background fits the task. Decide whether the questions plausibly measure the claimed capability. 2. Read examples, prompts, and code. Try solving questions. Look for obvious errors, inconsistent formatting, and implementation problems. 3. Run contrasting models. Test a capable small model and a strong large model. Inspect generated answers for quality and truncation instead of stopping at aggregate scores.

The historical model suggestions were Phi-3, offered cautiously as a small-model candidate, and Qwen2 or Command R Plus as larger examples. The point is to examine the evaluation across different models and read what they actually produce.

Exact-match scoring compares an extracted answer with the expected answer, so extraction must work reliably. The speaker praises GSM8K’s constrained output format and criticizes DROP’s described bag-of-words approach to judging correctness. Together with the decimal-point bug, this exposes a central debugging question: did the model fail the task, fail the output contract, or encounter faulty scoring machinery?

What to measure next: confidence, robustness, disagreement

Model calibration heads the closing wishlist: how well an answer’s probability score corresponds to its correctness. Multiple-choice datasets offer a starting point for measuring that relationship. Better calibration could make confidence scores useful, helping distinguish reliable answers from uncertain ones. A participant also argues that instruction-tuned models are more overconfident than base models, though no measured comparison accompanies that claim.

Prompt robustness returns to the formatting problem: ten wordings with the same substantive content should not produce ten different substantive answers. Repeated evaluation costs more, which the speaker sees as a reason this property receives too little attention. A third priority is sycophancy—whether models reinforce users’ thought bubbles or make factual corrections when disagreement is warranted.

At the July 2024 close, work on the next leaderboard version was already underway. The direction was more reasoning and mathematics, possibly code, with long-context evaluation still offering substantial room to explore. The speaker expected rapid gains on IFEval and eventual contamination of GPQA; a sycophancy evaluation remained a distant ambition. The next version’s scope would depend on how quickly models changed and which measurements still revealed meaningful differences.