Benchmarks, Self-RAG, and Step-by-Step Verification — Part 3 of 5

When every flight is full

An agent planning a trip to Vienna can produce a reasonable itinerary and still fail when every flight is full. It needs backtracking at the system level: revisiting a decision and adapting its plan, much as a search traverses alternative paths through a graph. The capability belongs to the surrounding agent system rather than the language model’s token-level decoding.

The opening exchange identifies two complementary routes to better agents: - Improve the language model: scaling the underlying large language model, or LLM, is expected to improve agent performance. - Improve the surrounding system: give the agent better tools, planning, and ways to recover when actions fail. The scaling claim is a prediction here. The interviewee also describes an open benchmark and leaderboard developed with Meta and Hugging Face, inviting others to submit results and push performance forward.

The next talk in the ICLR 2024 recording examines that collective activity. Moritz Hardt treats benchmarking as a scientific practice: how does a community competing on shared tests produce dependable progress?

A public benchmark breaks the holdout’s isolation

Hardt argues that machine learning benefits from unrestricted inspiration—brains, child development, physics—while benchmarks give that freedom an empirical constraint. Researchers agree on test cases and a measure of success, compete, and rank their models. He connects this practice to philosopher Michael Strevens’s iron rule: disputes ultimately face competitive empirical testing.

His historical sketch follows four changes in how that competition is organized: - DARPA in the 1980s: benchmarks helped compare funded research against proposed objectives. - The MNIST era: public datasets and leaderboards brought benchmarking into the academic mainstream. - The ImageNet era: developing, testing, and ranking models on ImageNet became integral to the deep-learning developments of the 2010s. - The polymorphic era: Hardt’s provisional name for the proliferation of differently structured benchmarks, including multitask evaluations and datasets that evolve over time.

A foundational mistake, Hardt says, is equating this community practice with the holdout method. In that setup, researchers freely use training data but reserve test data for evaluation at the end. His vault assumption requires that the test set remain isolated from model development. Under the guarantees he discusses, the number of supported model comparisons grows exponentially with dataset size, though he does not specify the error tolerances or confidence levels.

Public benchmark use is adaptive: test results influence the next model. Researchers evaluate, learn from the result, change their methods, and evaluate again. That feedback breaks the isolation behind the holdout guarantees. In worst-case constructions, the number of supported evaluations can fall to linear in dataset size.

At the time of the talk, Hardt reports roughly five million Hugging Face downloads in the preceding month across two versions of MMLU, a language-model benchmark with about 14,000 questions. Extrapolating that rate and assuming at least one evaluation per download suggests tens of millions of annual evaluations—an estimate of potential use, not a measured evaluation count.

Why heavily reused tests can still rank models

Practical benchmarks appeared to survive much longer than worst-case theory suggested. ImageNet’s 2012 benchmark continued supporting improvements after roughly a decade of intensive use. More tellingly, researchers carefully recreated its test-set construction and found that model rankings persisted on fresh data. Hardt reports similar stability for MNIST and several Kaggle competitions.

He calls this internal validity: improvements replicate under closely recreated testing conditions. Years of feedback into model development had not made those rankings useless.

A theoretical result offers one explanation. Suppose researchers attend only to results that beat the previous best and ignore non-improvements. Under that simplifying behavioral assumption, exponential benchmark longevity can be recovered. Limiting which feedback drives future work can protect the benchmark. An evaluation system could enforce limited feedback, but Hardt prefers the descriptive interpretation: the way a community organizes competition can itself provide protection.

Other influences include code sharing and collaboration, researchers’ cognitive and behavioral limitations, and dataset choices such as the number and identity of classes. Hardt connects the shared-code ecosystem to Dave Donoho’s phrase frictionless reproducibility. Benchmark longevity depends partly on how people work with the data.

ImageNot: different data, the same architecture rankings

ImageNot asks whether architectures developed on ImageNet retain their ordering after retraining from scratch on substantially different data. Hardt calls it an anti-replication: preserve comparable size, scale, and diversity while changing the dataset’s construction as much as possible.

ImageNot draws from LAION, a resource of image-caption pairs derived from Common Crawl. Its construction replaces careful human annotation with caption-based selection: 1. Choose 1,000 arbitrary classes, excluding branches of the WordNet hierarchy that contain ImageNet’s ILSVRC 2012 classes. 2. Use the RoBERTa text model to compare class names with captions and retrieve matching images. 3. Apply additional safety filters, with no annotators and minimal human intervention. Selection never inspects image content. A caption match determines inclusion without visual confirmation that the named object is present.

The resulting categories expose two consequences of that choice: - Objects need not be visible: the cleats category includes football players whose cleats cannot be seen, as well as shoes with bicycle cleats. - Words can mix unrelated meanings: batter includes baseball cartoons, food-related material, and a cosmetic product called Batter Up.

Yet the architectures studied retained exactly the same ranking, with roughly the same relative improvements over AlexNet. Hardt also reports preserved comparison patterns under fine-tuning and similar relative utility for transfer learning. These are retrospective results for the architectures examined, rather than guarantees for arbitrary models or environments.

The experiment motivates an external-validity conjecture: improvements under sufficiently rich testing conditions may replicate elsewhere. What counts as sufficiently rich remains undefined. ImageNot suggests that some dataset artifacts and annotation choices may matter less for model ranking than assumed, without establishing that any dataset of similar size will do.

When more examples beat cleaner labels

Work with Florian Dorner makes the annotation question precise. Two binary classifiers, F and G, must be compared by accuracy. Unlabeled examples are free, each requested label costs one euro, and each label is wrong with probability p < 1/2. The objective is to spend N euros to maximize the probability of identifying the better classifier.

Two strategies trade cleaner individual labels against more examples: - Repeat and vote: sample N/K examples, buy K independently drawn noisy labels for each, and reduce label error through majority vote. - Label once: sample N examples and buy one noisy label per example. Within the presented model, `K = 1` is optimal for identifying the better classifier. Cleaning each label sacrifices examples that contribute more to the ranking decision.

The proof uses Cramér’s theorem from large-deviation theory to characterize the asymptotic probability of choosing the wrong classifier. A union bound extends the analysis to multiple comparisons. Hardt contrasts this with the less precise upper bound obtained from Hoeffding’s inequality. Dorner also built a sample-size calculator for noisy-label dataset design. The result is specific to this noise model and ranking objective; it is not a universal annotation policy.

Multitask leaderboards inherit voting-system problems

Benchmarks such as SuperGLUE, MMLU, Big-Bench, and HELM respond to the concern that one task is too narrow for language and multimodal models. Hardt examines their aggregation through social choice: tasks act as voters, models as candidates, and the benchmark as a voting rule. Each task supplies a judgment; the benchmark must turn those judgments into one ranking.

Two forms of aggregation use different information: - Cardinal benchmarks aggregate numerical scores. Big-Bench and the OpenLLM Leaderboard are presented as examples that average task accuracies. - Ordinal benchmarks aggregate rankings. HELM, an effort to evaluate language models holistically, uses a winning rate: how often a model beats others across tasks. This can be computed from task rankings alone, without numerical score differences.

Hardt presents Arrow’s impossibility result as a constraint on ordinal aggregation. A rule that accepts all rankings, respects unanimous preferences, and does not simply copy one task’s ranking cannot also avoid sensitivity to irrelevant alternatives. Adding another contender can change the relative order of existing contenders. The theorem alone neither quantifies the disruption nor covers cardinal score aggregation.

The empirical work measures two properties: - Diversity: how much task rankings differ, measured using Kendall’s W coefficient of concordance. - Sensitivity: how much the aggregate ranking changes under transformations the researchers regard as irrelevant. These include adding weak models to ordinal benchmarks and applying monotone linear transformations to cardinal task scores—changes that preserve each task’s model ordering.

Across the benchmarks examined, greater diversity accompanies greater ranking sensitivity. Tasks that disagree provide different information, but their aggregate is more susceptible to these transformations. As a sanity check, the researchers randomly partitioned ImageNet classes into 20 nominal tasks. Their measures still identified it as effectively a single-task benchmark: multiplying task labels had not produced diverse judgments about models.

The movements can be substantial. Under the tested metric transformations, a model could move across 80% of ranks on Big-Bench Hard and close to all ranks on MMLU. HELM and HEIM subcategories showed a similar, less clean diversity–sensitivity pattern. These measurements are computational lower bounds on sensitivity: the exact maximum is difficult to find and may be higher.

The `benchbench` package lets practitioners load benchmarks and compute diversity and sensitivity; the installation command is pip install benchbench. Hardt invites benchmark submissions for the ongoing measurement effort. He also credits the engineering behind evaluation infrastructure, including the Eleuther Evaluation Harness underlying the OpenLLM Leaderboard.

Dynamic benchmarks can stall after a few rounds

Dynamic benchmarks evolve by incorporating newly discovered failures. Hardt describes the approach pursued by DynaBench as a repeating procedure: 1. Start with a dataset and let developers build models. 2. Find examples on which those models fail through adversarial data collection. 3. Add the failures to the dataset and repeat. The intended benefit is continued improvement on increasingly challenging examples.

Work with Ali Shirali and Rediet Abebe represents a dynamic benchmark as a directed acyclic graph: operations can branch and merge without forming cycles. Its nodes build models, collect or ensemble models, collect adversarial examples, or pool data. This permits structures beyond the standard path that simply alternates model building and adversarial collection. Hardt reports a theorem showing that the standard path can stall after only a few rounds, with some experiments exhibiting a similar plateau.

Hierarchical dynamic benchmarks run parallel threads and merge them in particular ways. They offer stronger theoretical progress guarantees, but implementation is substantially harder and practical feasibility remains uncertain. An evolving test set alone does not guarantee sustained progress.

Hardt closes by delimiting the noisy-data argument: accuracy ranking is a narrower objective than fairness or safety assessment. Bias analysis, red-teaming, alignment, and related evaluations depend substantively on the content of individual test cases. His broader call is for theoretical and empirical foundations for benchmarking itself, because intuitions about which collective practices produce scientific progress often fail.

Self-RAG decides when evidence helps

How did US states get their names? The question requires combining evidence across documents, and a plausible answer can still contain factual errors. It motivates Self-RAG, presented by Akari from the University of Washington. Standard retrieval-augmented generation, or RAG, retrieves documents for a query and places them alongside the question in the language model’s input. Self-RAG trains the model to decide when to retrieve, generate, and evaluate its output.

The state-name example exposes two failures when retrieved passages are concatenated: - Irrelevant evidence distracts: a paragraph about Michigan’s history influences the answer despite not explaining the state’s name. - Relevant evidence goes unused: useful context elsewhere does not prevent unsupported claims.

Retrieval can also be unnecessary. A request for a summer-vacation essay may retrieve definitions or material about a movie with that title, adding latency and potentially hurting output quality. Self-RAG makes retrieval conditional: when external evidence is unnecessary, it can proceed as an ordinary language model.

When evidence is needed, the model makes intermediate judgments explicit: 1. Trigger retrieval: emit a special retrieval token. 2. Process passages separately: use batch decoding to handle documents in parallel. 3. Assess relevance and generate: for each paragraph, predict relevance and generate a continuation. 4. Assess support: predict whether that paragraph supports the continuation. 5. Select sentence-level candidates: retain the best K outputs through beam search, ranking them with a weighted sum of normalized probabilities for desirable critic tokens such as relevant and supported.

Relevance and support are separate judgments. In the example, a paragraph discusses Utah but the continuation mentions Utah and Alabama, so Self-RAG labels the output partially supported. The Michigan-history paragraph is labeled irrelevant. An adjustable retrieval threshold controls how often retrieval occurs, trading efficiency against final performance.

Training reflection into the generator

Reflection tokens extend the model’s vocabulary with retrieval decisions and critique judgments. A separate seven-billion-parameter critic teaches the generator when to emit them. The critic receives an evaluation instruction together with inputs such as a question, retrieved passage, and proposed output.

Because sentence-level human annotations for retrieval necessity and multiple critique dimensions are expensive, the pipeline prompts GPT-4 with instructions and demonstrations to produce synthetic critic-training data. The trained critic and a retrieval model then augment existing instruction pairs, inserting passages and reflection tokens to mimic inference. The presentation reports 150,000 instruction-tuning examples produced this way.

The generator learns ordinary text and reflection tokens using the standard language-model objective over the expanded vocabulary. This makes the pipeline adaptable to other models and codebases. At inference time, retrieval thresholds and critique weights allow behavior to be adjusted without another training run.

The evaluation uses Llama 2 at seven and thirteen billion parameters, trained on four GPUs, across six tasks spanning closed-set tasks and short- and long-form generation. Standard RAG helps on PopQA, where an answer can be extracted from one document, but gives limited improvement on tasks presented as requiring composition across documents, including PubHealth and ASQA.

Self-RAG reportedly outperforms the evaluated baselines at the same model scale and matches or exceeds ChatGPT on five of six tasks; numerical scores and the ChatGPT version are not specified. The presentation also reports higher citation precision than recall. It closes with released code and checkpoints, integrations into LangChain and LlamaIndex, and research extending the method to biomedical applications.

Step-level verification matters more as search grows

The narrator connects Self-RAG’s intermediate critique to Let’s Verify Step by Step, introducing an interview about mathematical reward models—models trained to grade candidate solutions and help select a correct one. The study compares two supervision strategies: - Outcome supervision: label whether the final answer is correct. - Process supervision: label the correctness of each individual solution step. Process supervision makes credit assignment more direct: the labels identify where a solution goes wrong, rather than leaving the model to infer that from the final outcome.

Contractors labeled individual steps to train the process-supervised reward model, while the outcome-supervised model learned to predict final-answer correctness. The interviewee reports significantly greater reliability from process supervision in this math-specific comparison. Math, STEM, and code are offered as intuitive examples of reasoning, but the demonstrated comparison remains mathematical.

The generator stays fixed throughout the experiment. Reward models select among candidate solutions at test time; their feedback does not update the generator. Both can be fooled by plausible but incorrect work. The interviewee describes outcome supervision as more likely to overlook subtle mistakes, with process supervision better at detecting them—not a perfect verifier.

The difference becomes more apparent when selecting among many thousands of solutions, compared with roughly a hundred. This tests the prospect of solving difficult problems through extensive sampling, provided the system can reliably select a good solution. More generation compute makes the quality of the selector increasingly consequential.

The closing discussion turns to using strong judges to bootstrap smaller models. The researchers used GPT-4 to label smaller-model samples for ablations comparing outcome and process supervision, enabling experiments the interviewee says would not have been feasible with human labeling alone. They also released the dataset so others could train process-supervised reward models and investigate findings the original team might have missed.