Benchmarks 101: What a Model’s Score Actually Tells You

A correct answer can hide the important question

A child and several fish emojis are supposed to identify a movie. Swyx answers Finding Nemo, while acknowledging that he prepared the question. A superhero couple with three children yields The Incredibles. The opening quiz exposes a central problem in evaluation: a correct answer does not establish how it was obtained.

The quiz then moves through physics to mathematics. Which set of points in the complex plane satisfies z² = |z|²: two points, a circle, a half-line or a line? Alessio chooses a line, and Swyx confirms it. Similarly short answers can demand very different capabilities; understanding those demands is the first step toward interpreting a score.

Benchmarks provide tasks and measures for comparing models. In AI Fundamentals: Benchmarks 101, Alessio argues that release claims about beating benchmark X or Y mean little without knowing what each tests. Swyx places benchmarks alongside data, neural networks and compute as influential ingredients in development: they determine which capabilities become visible and how progress is judged.

They also shape incentives. Alessio argues that pressure to publish better scores steers research toward the tasks a benchmark rewards. Swyx adds that selecting which results to publish is a form of marketing. The evaluation target influences development; the reported subset influences the reader’s impression.

Read the metric and the setup together

A Spotify playlist makes precision and recall concrete: - Precision: what fraction of the songs in the playlist do you like? - Recall: what fraction of all the songs you like made it into the playlist? A conservative selection can increase precision while missing more relevant songs. Choosing a metric means deciding how to value the quality of selected results against coverage of everything relevant.

F1 combines precision and recall through their harmonic mean: F1 = 2 × precision × recall / (precision + recall). Broader evaluation considers more than answer correctness. Stanford’s HELM evaluation work, associated here with Percy Liang’s lab, includes calibration, robustness, fairness, efficiency, bias and toxicity.

The setup matters as much as the metric: - Zero-shot: provide no task examples before testing. - Few-shot: supply examples before the test question; K = 5 means five examples. - Fine-tuned: adapt the model using task-specific data before evaluating it. These require different amounts of preparation. Fine-tuning consumes additional data and compute, with improved performance as an expectation rather than a guarantee. A benchmark name and score alone leave these conditions unspecified.

From word relationships to image recognition

WordNet provides an early example of turning language relationships into something measurable. Its manually constructed database connects words through category membership, whole–part relationships and entailment: canine is a broader category containing dog; a building contains windows; snoring and sleeping illustrate an action implying another. Distance through the graph can serve as a measure of semantic similarity and a target for a model to predict.

The historical tour continues through Penn Treebank, dated to 1989 and built through manual text collection and labeling, to MNIST. MNIST makes the evaluation loop tangible: train on handwritten digit images, predict numerical labels and measure recognition error. Its 60,000 training images became a familiar introductory machine-learning exercise.

The Enron email dataset, released in 2004 after the company’s collapse, offers naturally occurring workplace correspondence for classification, summarization, entity recognition and language modeling. But realism is domain-dependent: recognizing entities in one energy company’s emails may poorly predict performance elsewhere. Even MNIST’s familiar task is narrow. The 2017 datasets EMNIST and Fashion-MNIST broaden recognition to handwritten letters and clothing, respectively, without making the evaluation universal.

ImageNet connects lexical organization to computer vision. Fei-Fei Li began the work in 2006 and released it in 2009; a meeting with a WordNet co-author forms part of that lineage. ImageNet used Amazon Mechanical Turk for image classification, prompting a concern about whether payment per image rewards speed over careful labeling. That is an incentive question, not a measured quality comparison.

The pivotal result is AlexNet’s 2012 ImageNet performance, attributed to a deep convolutional neural network: roughly 15% error versus 25% for the runner-up, a gap of about ten percentage points. The episode presents this as a turning point for deep learning, though it does not specify the error metric or evaluation protocol.

CIFAR-10, released in 2009, offers another compact recognition task: 60,000 color images at 32 × 32 resolution, covering categories such as airplanes, automobiles and animals. Its ten classes contrast with CIFAR-100’s hundred. Requiring finer distinctions is one way to renew a challenge as models improve—a pattern that becomes especially visible in language evaluation.

Language understanding becomes a collection of tasks

GLUE, General Language Understanding Evaluation, combines nine tasks across single-sentence classification, similarity and paraphrase, and inference. Introduced in 2018, it includes distinct demands: predict sentiment from a movie-review sentence in the Stanford Sentiment Treebank, or decide whether two news-derived sentences in the Microsoft Research Paraphrase Corpus mean the same thing.

The paraphrase corpus makes metric choice consequential. The discussion reports that 68% of its examples are positive and cites that imbalance as a reason to report both accuracy and F1. The distribution of answers matters when interpreting how well a classifier performs.

Question answering can also be reduced to a more specific evaluation task. Starting with SQuAD material—questions paired with Wikipedia paragraphs containing their answers—the described transformation pairs each question with individual context sentences, filters out pairs sharing few words, and asks whether a sentence contains the answer. That construction determines which part of answering a question gets measured.

SuperGLUE follows in 2019 as a response to models rapidly outgrowing GLUE. The discussion emphasizes multi-sentence context and inference over conversational text, including embedded clauses and disfluencies. Following hesitant, clause-filled speech demands more than classifying an isolated, cleanly written sentence.

The COPA causal-choice task exposes why word matching is insufficient. A person casts a shadow over grass: was the cause sunrise or grass cutting? Sunrise is the intended answer, although the other option repeats “grass.” The hosts interpret success as using learned world knowledge beyond the sentence; a correct choice alone does not reveal the model’s internal reasoning.

Plausible continuations, broad knowledge and code

SWAG, Situations with Adversarial Generations, asks for the most plausible continuation of a partial description. A woman sits at a piano onstage; putting her fingers on the keys is the intended next action. The task tests contextual plausibility, beyond grammatical compatibility. It also exposes ambiguity: the most likely continuation is not the only possible event.

Generation helps scale benchmark construction: SWAG is described as containing 100,000 multiple-choice questions, while HellaSwag draws on sources including wikiHow and ActivityNet. The episode reports HellaSwag scores of 85 for GPT-3.5 and 95 for GPT-4, against a random-guessing baseline of 25%. Prompting, model variants and evaluation settings are unspecified. The hosts interpret the near-ceiling result as benchmark saturation: the test needs harder challenges to distinguish further progress.

MMLU, Measuring Massive Multitask Language Understanding, spans 57 tasks, including mathematics, US history, computer science and law. Material ranges from elementary questions to professional examinations. The opening complex-plane problem returns alongside examples involving thyroid cancer and microeconomics, combining breadth of knowledge with problem-solving.

Reported MMLU scores are 70 for GPT-3.5 and 86.4 for GPT-4, against a random baseline of 25 out of 100; evaluation settings are not given. The discussion also relays MMLU author Dan Hendrycks’s prediction that an incremental successor to GPT-4 could reach expert human performance across its subjects. That remains a forecast in this account.

HumanEval shifts the target to code generation. Introduced in 2021 alongside OpenAI’s Codex, the code-focused GPT model powering GitHub Copilot, it is explained as completing partially supplied functions and receiving success-or-failure outcomes across 164 evaluations. The reported GPT-4 score is 67%, without execution-test or sampling conditions. An observation that older coding questions seem easier for models than newer ones introduces the next concern: whether test material appeared during training.

Whose language—and what kind of knowledge?

XTREME, Cross-lingual Transfer Evaluation of Multilingual Encoders, evaluates across languages. Its deliberate diversity includes Tamil, Telugu, Malayalam, Swahili and Yoruba. This addresses a concern that optimizing around English and its abundant training material can leave other language communities behind.

Being understood and sounding natural are different targets. An Italian-speaking participant says GPT-3 produces Italian but would struggle to pass as someone from a particular locality. A Chinese-speaking participant describes a language-tutor application whose output was understandable yet unidiomatic. These personal observations highlight local dialect and colloquial usage; a hypothetical movie-advice example adds that English-only evaluation may also miss differences in audience preferences across cultures.

BIG-Bench, presented as a 2022 benchmark, expands evaluation to 204 tasks contributed by 442 authors across 132 institutions. Topics range from linguistics and mathematics to social bias and software development. The collection includes challenges intended to resist memorization, while the discussion distinguishes finding checkmate from recalling the element with atomic number six—a factual lookup models are expected to memorize.

An anachronism, something placed in the wrong historical period, makes the knowledge-and-inference distinction concrete. Compare someone using a radio during the bombardment of Iwo Jima with someone using an iPhone there. Detecting the mismatch requires connecting the event and technology to their periods, then comparing them.

BIG-Bench’s absence from GPT-4’s reported results initially prompts speculation about poor performance. An editing-room correction supplies a different explanation: GPT-4 reportedly knew its canary identifier, a random string intended to reveal exposure to benchmark data. The correction attributes the omission to contamination.

When a score stops measuring unfamiliar problems

Uneven mathematical results complicate intuitive rankings of difficulty. The hosts report GPT-4 scores of 30 on AMC 10 and 60 on AMC 12, despite describing the latter competition as harder. They suggest training exposure as a possible explanation, without establishing it. Their scoring discussion also does not justify the claim that GPT-4 answered everything incorrectly.

Benchmark contamination has a straightforward route: published questions and solutions enter later web-scraped training corpora, allowing a model to reproduce familiar answers during evaluation. High scores may then reflect memorization rather than generalization, the ability to solve unfamiliar problems. Public tests become vulnerable to the preparation problem exposed by the opening movie quiz.

A secondhand Codeforces anecdote reports that GPT-4 solved all tested pre-2021 problems and none from 2022 onward. The speaker interprets the split as memorization associated with a reported training cutoff. Without sample sizes, difficulty matching, prompts or execution conditions, the anecdote raises the concern without isolating its cause.

Other failures originate in the dataset. A speaker alleges that ImageNet labels white businessmen without an ethnic qualifier while adding one for Asian businessmen, potentially reinforcing stereotypes. An Iris dataset example illustrates the danger of fitting recording errors, though the alleged erroneous records and biological constraint are not established. Task specificity creates another limitation: answering questions from single sentences may poorly represent a product’s actual work.

Reproducibility includes the surrounding code. Unversioned, untested preprocessing and postprocessing can make nominally identical benchmark runs disagree. Verification also requires money and access: GPT-3’s API-only availability and restricted GPT-4 API access were barriers at the time of the discussion.

Confidence calibration asks whether a model’s confidence corresponds to how often it is correct. The discussion reports that a training change documented in the GPT-4 paper made confidence exceed correctness. This is a separate evaluation target: even when a system makes mistakes, can its confidence reliably indicate when to trust it?

The benchmark your product needs

Production evaluation must place answer quality alongside operational measures: - Latency: how long the user waits. - Cost per call: whether each interaction fits the business model. - Throughput: whether the system can handle many requests at once. The discussion uses illustrative prices of one cent versus 0.1 cent per thousand tokens to make budget sensitivity concrete. A research score alone cannot resolve that product tradeoff.

Judge latency against the work the response replaces. Copilot suggestions need to arrive within a developer’s flow of writing, even when imperfect. By contrast, one participant would wait an hour for a model to create an entire application. Model selection therefore depends on the task’s tolerance for delay, alongside the capabilities required.

Two development strategies follow. One prototypes with a large, capable model, then moves toward a smaller domain-specific model as production demands lower cost and latency. The other keeps large models and gives agents more time when they can save substantial human effort. These are proposed directions for development, not measured outcomes.

The second strategy has a concrete desired workflow: screenshot a table embedded in an image and ask an AI browser to turn it into a chart. Waiting would be worthwhile if the model removed the intervening manual steps. Likewise, roughly ten seconds for Bing could be acceptable if it performed several searches and returned the right result. The screenshot-to-chart workflow is an aspiration here; its value lies in work avoided, not response speed alone.

The episode closes by introducing its fundamentals format: prepared topical discussions that deepen understanding beyond the news cycle. Readers can explore the accompanying benchmark repository and suggest future fundamentals topics in the episode comments.