Wei-Lin Chiang

Wei-Lin Chiang builds systems for evaluating AI models, including Chatbot Arena, where users compare anonymous model responses and vote for the stronger answer. He is an equal contributor with Lianmin Zheng to the Chatbot Arena research report. Felicis lists him as Arena’s CTO and co-founder, alongside CEO and co-founder Anastasios Angelopoulos and co-founder Ion Stoica.[2][3]

In his November 2024 Latent Space interview, Chiang explained why the team turned to human comparisons: conversational tasks such as drafting an email or writing a story may have no single ground truth, and people can find it difficult to assign an absolute score. Choosing between two answers offers a simpler judgment. He recalled that the team’s own difficulty evaluating a chatbot led to a side-by-side comparison interface, then anonymous voting: “So the community decides which one is better”.[1]

Chiang coauthored the December 7, 2023 ranking update with Tim Li, Joseph E. Gonzalez and Ion Stoica. The team replaced online Elo updates with maximum-likelihood Bradley–Terry ratings, fitted centrally from the comparison history. Unlike online Elo, the method assumes fixed model performance and does not depend on game order. Bootstrapping the fitted ratings supplied confidence intervals; the team reported that these better captured uncertainty than its previous shuffled-Elo procedure.[4]

That assumption also changed how Arena tracked proprietary models. The team separated GPT-4 API versions whose votes had previously been pooled under one name and tried to pin specific API versions. Hosted models could still change without notice. When the team observed different ratings for two GPT-4 versions, it considered several explanations—including changes in users or prompts and missing comparisons against newer models—and brought the older version back to collect fresh votes.[4]

Chiang argued for category-specific leaderboards because models perform differently across domains and a single ranking cannot answer every community question. In November 2024, he described the team’s work on automated classification and filtering to provide clearer signals for coding, math and harder prompts, with the cleaning and filtering pipelines released as open source. During the discussion of whether style-controlled rankings should become the default, he observed: “There's no neutral choice here.” Angelopoulos supplied the interview’s detailed statistical explanation of style control.[1]

Chiang also emphasized the cost of live human evaluation: developers need cheaper automated benchmarks to track progress and iterate quickly. His work includes the MT-Bench and Chatbot Arena paper, published in the NeurIPS 2023 Datasets and Benchmarks Track, whose author list marks Lianmin Zheng, Chiang and Sheng Ying as equal contributors. In the 2024 interview, he described LLM judging and Arena-Hard as experiments in turning Arena’s collected questions into faster evaluation signals.[2][1]

The April 19, 2024 Arena-Hard announcement, coauthored by Tianle Li, Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Gonzalez and Stoica, described a pipeline starting with 200,000 Arena queries. It grouped prompts by topic and used LLMs to assess seven criteria, including specificity, domain knowledge and problem-solving. Arena-Hard-Auto-v0.1 sampled two prompts from each of 250 selected clusters, producing 500 prompts. An LLM judge compared answers against a baseline twice, swapping their positions to reduce position bias, for 1,000 judgments per model.[5]

For the specified overlapping set of 20 top models as of April 13, 2024, the team reported 89.1% agreement with English-only Chatbot Arena and 87.4% separability, using ranking metrics that account for confidence intervals. These describe agreement and distinguishability between model rankings, rather than answer accuracy. The team acknowledged possible judge self-bias, LLM-driven prompt-selection bias and verbosity effects; this version also did not evaluate multi-turn interactions.[5]

Chiang described RouteLLM in November 2024 as another use for preference data: learning to route a question to a suitable model, potentially exploiting differences in coding or math performance. The team released an open-source framework so others could train and evaluate their own routers. He presented larger preference datasets, better routers and a routing benchmark as future directions.[1]

At the time of that interview, Chiang introduced himself as a fifth-year Berkeley PhD student. He supported working with frontier labs to make evaluation useful to model builders as well as the public. He and Angelopoulos also discussed separating Arena from the student-driven LMSYS group, allowing LMSYS to support other research projects. Chiang’s personal biography identifies his Berkeley SkyLab research relationship with Ion Stoica.[1][2]

Before his language-model evaluation work, Chiang researched efficient machine-learning training. His December 2018–March 2019 Google Research internship focused on algorithms for large and deep graph convolutional networks, providing earlier systems-research context for his career.[2]

Chiang coauthored Cluster-GCN with Xuanqing Liu, Si Si, Yang Li, Samy Bengio and Cho-Jui Hsieh. First submitted in May 2019 and published at KDD that year, the algorithm samples blocks of nodes belonging to dense subgraphs identified through graph clustering, then restricts neighborhood searches to those subgraphs. This reduces the computational and memory demands of graph convolutional network training.[6]

The authors created Amazon2M, a graph with two million nodes and 61 million edges, to test scalability. For a three-layer GCN on that dataset, they reported training in 1,523 seconds using 2.2 GB of memory, compared with VR-GCN’s 1,961 seconds and 11.2 GB. The comparison illustrates the efficiency gains in that particular experiment.[6]