Chatbot Arena: What a Preference Leaderboard Measures

Vicuna needed more than a convincing demo

The team had released Vicuna, but its demo left a harder question unanswered: how could they establish whether it was better than another chatbot, or measure its gap to the newly announced GPT-4 and Claude 1? New fine-tuned models were appearing every week. An appealing conversation was not enough to make a defensible comparison.

Wei-Lin traces Vicuna to experiments fine-tuning Llama 1, inspired by Stanford’s Alpaca project. The team collected ShareGPT conversations between users and ChatGPT and used them as dialogue training data. In his assessment, Vicuna showed that an open-weight model could offer conversational capabilities similar to ChatGPT; he gives no quantitative comparison for that claim.

The demo became the starting point for Chatbot Arena. The team added other open models and models accessed through APIs, then a side-by-side interface. Anonymous pairwise comparison followed: hide the model identities and let users vote for the better response. The community could make the comparative judgments instead of the researchers arguing for their own model. Wei-Lin places the launch around April or May 2023, with weekly leaderboard updates early on.

Why compare answers instead of grading them?

An email draft or a story may have no single correct answer. Conversations spanning several turns make evaluation broader still. Wei-Lin’s case for human feedback starts here: people can judge which response better serves their purpose even without a fixed reference answer. Choosing between two responses is also easier, in his account, than assigning an absolute quality rating.

Anastasios adds that changing a generative model changes the outputs needing evaluation; those outputs cannot all be annotated in advance. A fixed benchmark can isolate historical knowledge or another capability, but even many such tests struggle to capture the breadth and subjectivity of conversational usefulness.

Live human evaluation is slower and more expensive than offline testing, so developers still need static benchmarks for rapid iteration. Arena measures a different kind of performance rather than being uniformly better. The team continued exploring automated evaluation, including ArenaHard: select high-quality questions collected through Arena, then automate judging to produce faster benchmark signals.

The voters define the measurement

Arena encouraged organic use by offering free direct chat without requiring visitors to enter battles. Some visitors would become interested enough to compare answers and vote. Anastasios argues that this broad funnel mattered: reserving free use for battle participants would not have worked.

Voters were a self-selecting audience of enthusiasts and early adopters, with substantial interest in coding. The leaderboard represents the people who come and vote, not the general population. Its aggregate score needs to be read with that audience in mind.

Participation was also fragile. After the launch, user numbers declined enough that the project’s prospects looked unpromising. Continuing votes were essential to maintaining the leaderboard. The team emphasized trust and fair treatment as priorities for a public, community-driven project.

What style control actually adjusts

Millions of conversations and votes made more specific analyses possible. Style control asks which model would perform better if response length or formatting were held constant. That is a conditional comparison, not a declaration that presentation is unimportant.

Arena fits a logistic regression, a statistical model that predicts human preferences from the identities of the competing models. Each model receives a fitted coefficient representing its contribution to predicting votes; a larger coefficient indicates stronger preference performance.

Length control adds another predictor: the difference in length between the paired responses. The regression can then assign length-related predictive power to a separate coefficient, adjusting the coefficients associated with model identity. Arena’s style adjustment uses roughly five or six components, including Markdown headers and bulleted lists.

Statistical adjustment does not by itself establish causation. Its interpretation depends on conditions that are not always verifiable; estimating causal effects requires an appropriate account of how the factors influence one another. The work was described as a first step. Cost and parameter count were suggested as further adjustment dimensions for asking which models perform unusually well for their resource requirements.

Choosing a default score also means choosing which preferences count. Making style control the default remained under consideration: a guest suggested a community vote rather than imposing that judgment. The host countered that keeping the existing default was itself a judgment, with no neutral choice.

Categories, expert judgment and executable evidence

Community demand pushed Arena toward domain-specific leaderboards because models could perform differently on coding and other technical questions. An automated classification pipeline divided conversations into coding, mathematics, instruction following and harder problems. These categories aimed to provide more specific signals and clarify what each comparison measured.

The approach remained to collect organic interactions and identify useful subsets afterward, with open-source cleaning and filtering pipelines. But filtering raised a separate question about expertise: could aggregate preferences miss distinctions that highly skilled programmers or red teamers would notice? A questioner raised their possible absence as a sampling concern, not an established finding about Arena’s audience.

For tasks with less subjective outcomes, preference alone is insufficient. Executing code inside Arena was a desired improvement: it would give voters evidence about what a program does, beyond the answer’s apparent quality.

Red Team Arena explored another approach: human attempts to break a model’s constraints could become games with explicit wins and losses. Extensions to Bradley–Terry modeling, a method for assigning strength from pairwise outcomes, could jointly rate human players and models while accounting for the target jailbreak tasks. This was an active research direction with initial ideas.

One proposed game would hide a secret in a system prompt and challenge players to extract it. The broader aim was to build tools skilled red teamers enjoyed using and give them credit for their work, with the expectation that serving that community would produce useful evaluations. The platform and proposed games were still at an early stage.

o1: stronger answers, longer waits

o1 made Arena visibly slower. A guest reported response times of 30–60 seconds, sometimes longer, alongside substantial improvements in mathematics and coding categories. These were early-release observations, before what a questioner called the full model had been seen; the exact evaluated version was not identified.

Another guest described using o1 to prove theorems they said had never been public—a personal account, without the theorem statements or proofs. They interpreted its reported leaderboard lead as evidence that Arena had not saturated when earlier models clustered together. Elo is a comparative rating with no maximum score: a stronger entrant can distinguish itself by beating existing competitors.

A questioner raised the fairness objection that other models might improve with similar reasoning time. The response accepted the concern but pointed out that Arena already compared models with different resources, including models with 8 billion and 405 billion parameters and different latencies. Latency control was discussed as a way to examine that trade-off, though its implementation status was unclear.

Private testing and the winner’s curse

Private testing served Arena’s ambition to help model builders evaluate candidates as well as offer public comparisons. It also introduced a statistical concern: if a lab tests several candidates and releases the highest scorer, favorable evaluation noise partly determines the winner. Its measured performance can be overstated. This selection bias is the winner’s curse.

At the time, Arena’s confidence intervals—its ranges expressing uncertainty in estimated performance—were not adjusted for multiple comparisons. A guest proposed the Bonferroni correction: for five candidates, construct each interval at confidence level 1 − α/5 rather than 1 − α, where α is the error allowance. The guest also reported simulations in which selection bias from five candidates was small relative to ordinary voter variability, without giving a setup or numerical effect size.

Arena’s second defense was fresh votes. Evaluations collected after selection could pull an initially inflated score down. This is a long-run argument: it depends on substantially more fresh data accumulating than was used to choose the candidate.

Score movement also requires careful model identification. A participant’s informal tracking of GPT-4o August and RekaCore left uncertainty about data entry and whether the comparisons concerned different checkpoints or changes within a fixed endpoint. The Arena response was that fixed endpoint versions usually showed small changes after release. Expected rating variation could be calculated, and larger movements warranted investigation.

RouteLLM: choosing a model for each question

RouteLLM explored whether preference data could train a router—a system that chooses which model receives a question. Different strengths across mathematics, coding and other tasks might make that choice useful. Its initial release provided an open-source framework for training and evaluating routers; the reported usefulness was modest, with work ongoing.

Does a router need to be as intelligent as the strongest model it selects? The response separated two objectives: - Cost–performance routing: send easier questions to a smaller model and harder ones to a larger model, using signals such as code, mathematics or prompt length. - Specialist routing: estimate which model performs best for each type of question and select accordingly. A simple difficulty rule can be a useful starting point for the first. The second requires estimates accurate enough that routing introduces less error and variability than it saves compared with simply using the best model.

The team identified the lack of a good router benchmark during development as a significant gap. Proposed next steps included more preference data, models trained to predict preferences, better routers and benchmarks for evaluating the resulting choices.

Building beyond the leaderboard

Arena had grown into a distinct effort within LMSys, a student-driven collection of research projects. Separating their identities was intended to keep LMSys from becoming synonymous with one leaderboard and leave room for other work, including SGLang.

The closing invitation returned to concrete infrastructure: red teaming, vision, coding copilots and especially an environment for executing code inside Arena. A guest singled out code execution as a potentially major improvement for coding evaluation, while describing the academic lab as short of the people needed to support that scope.

Contribution needs ranged from backend and full-stack development to new evaluation ideas and foundational statistical proofs. The speakers invited pull requests and promised contributor credit: the ambition remained a community project whose work spans both the mathematics of comparison and the tools people use to judge models.