Arena’s evaluation problem: earning the usage that makes a benchmark useful

Who pays for millions of real questions?

What does an evaluation platform need $100 million for? Anastasios Angelopoulos starts with an operating expense: Arena pays for all inference on its platform, with standard enterprise discounts. Free access still creates a substantial bill. His off-the-cuff estimate puts cumulative conversations at roughly 250 million, with monthly conversations in the mid-tens of millions.

That expense follows from LMArena’s central evaluation bet: people bring their own tasks. The platform measures AI capabilities through real-world usage and user feedback. Angelopoulos says the team considered remaining an academic project or becoming a nonprofit, but concluded that a company could provide the resources, distribution, and platform quality the mission required.

The transition began before the researchers committed to a business. Angelopoulos credits an early backer, called Anj in the recording, with providing resources and forming an entity while leaving the team free to walk away; Sequoia also provided a grant. The companion article identifies Anj as Anjney Midha. The shorter Arena brand expresses the ambition to broaden beyond the project’s language-model origins.

Angelopoulos treats the $100 million raise as room for successive bets, rather than a spending target. A later January 2026 update in the companion article reports a $150 million Series A at a $1.7 billion valuation and links to The Information’s funding coverage. The written update also reports $30 million in annualized consumption revenue, or $2.5 million monthly, following the September evaluation-product launch.

The audience becomes part of the benchmark

When users supply the tasks, their backgrounds matter to the evaluation. Angelopoulos reports that about a quarter of Arena’s users work in software and about half are logged in. Surveys, login information, and incoming prompt distributions help characterize that audience. Expert Arena is part of the effort to understand which experts use the platform. These estimates remain subject to survey response bias, which Angelopoulos explicitly acknowledges.

The participants describe Artificial Analysis as aggregating public benchmarks, independently rerunning them, and producing analytics and reports. Arena’s representative argues that questions submitted for users’ own purposes give Arena a different kind of realism. Their video comparison exposes the tradeoff: - Pregenerated comparisons avoid the generation wait. Artificial Analysis’s video arena is described as presenting existing videos without letting voters enter their own prompts. - Organic requests test the user’s own task. Arena’s approach lets people bring video requests, with the accompanying generation delay.

The host adds another reason to visit: other people’s prompts reveal capabilities. Seeing a prompt and its result can teach users what to ask for, even when the output does not address their immediate task. Both participants recognize this educational value. Personal task relevance and capability discovery give an arena different ways to attract users.

From Gradio to React: reducing development friction

Arena’s move from Gradio to React and Next.js reflects the demands of building a consumer interface. Asked whether performance forced the migration, the representative emphasizes development friction, access to React tooling, and developer familiarity. Custom video-loading indicators and notifications illustrate the interface features the team wanted.

The decision turns on the cost of building and staffing an interface, beyond whether a framework can support a feature. The representative acknowledges that the desired features might be possible in Gradio, but hiring developers familiar with the replacement stack reduces retraining. Inference for free usage remains the primary stated use of funding, with hiring another expense.

Preview testing attracts users—and raises fairness questions

Leaderboard Delusion, a paper criticizing Arena, challenges its prerelease testing practices. As summarized by Arena’s representative, the allegation is that undisclosed private testing creates leaderboard inequities: providers submit experimental models for exposure on the platform before release. The representative disputes that characterization and says Arena’s response identified factual errors, only some of which were subsequently corrected.

Two distinct issues emerge: - Which models receive evaluation traffic? The representative disputes the paper’s open- versus closed-source sampling figures and offers a roughly 60/40 split. Without identifying which category received 60%, the denominator, or the period, that figure cannot support a quantitative comparison. - What does preview access do to fairness? The representative defends prerelease testing as longstanding and popular with the community. That explains its product appeal, but does not establish whether it introduces evaluation bias.

Nano Banana’s Arena debut supplies the memorable example. The representative says it became a global sensation after appearing on the platform. The companion article describes the attraction of previews: early access, secret codenames, and a chance to cast the first votes on unfamiliar capabilities. The representative also credits Nano Banana with changing Google’s market share and moving billions of dollars in stock value, without measurements establishing that causal attribution.

Image generation becomes a tool for understanding

Nano Banana leads both participants to reconsider image generation. The host had seen it as peripheral to reasoning and a source of reputational problems, but now expresses enthusiasm for Nano Banana Pro. Arena’s representative likewise says they underestimated the benefits and predicts substantial economic value from multimodal models—here, capabilities extending beyond text into images and video—particularly in marketing and design.

A personal reading workflow makes the value concrete. The host fed explanations of reinforcement-learning environments from the DeepSeek V3.2 paper into Nano Banana Pro and used the resulting diagram to understand the paper. A generated diagram can help during comprehension, before it becomes a publication asset. The host calls the result paper-quality and compares it with a month of a PhD student’s manual work—an anecdotal estimate, not a measured productivity gain.

The discussion extends to explainers, infographics, YouTube thumbnails, and future AI-generated research-paper figures. These uses give substance to Arena’s broader scope: evaluating the visual outputs people want alongside language-model responses.

Fresh tasks and clear publication rules

Arena’s stated mechanism for keeping its benchmark fresh is a continuing supply of real user tasks. New data arrives as models and use cases change. The goal is to resist overfitting and give developers a relevant target. This renewal is a design rationale, not a guarantee. The representative also says Arena continues releasing millions of real-user conversations for community research and improvement.

The public leaderboard is described as a loss leader that does not meaningfully generate revenue. Its stated publication rules draw a consequential boundary: - Payment cannot buy entry or removal. Providers cannot pay to enter the public leaderboard or remove a poorly scoring model. - Unreleased previews may remain unpublished. The representative distinguishes experiments that never launch from released models whose scores appear publicly.

The representative argues that aggregating votes from millions of people worldwide makes displayed scores statistically sound and reflective of real-world capabilities, but does not explain the statistical procedure or provide validation results. Publication policy and score validity are separate questions: prohibiting paid removal does not by itself establish what a ranking measures.

Expansion needs focus; retention needs daily value

Occupational comparisons aim to show how models perform for particular groups. The representative describes categories spanning medicine, law, business, finance, accounting, creative work, and marketing, arguing that even single-digit percentages of Arena’s large audience provide substantial groups within these fields. Code Arena and Expert Arena are named as recent offerings. Further multimodal evaluation is planned, with the on-site video launch described only as later that year or early the next, relative to the conversation.

An application programming interface (API) is under consideration, with acknowledged demand but no commitment or timeline. The decision rule is do arenas well before expanding into adjacent offerings. Even a useful addition competes for a startup’s attention.

The retention discussion is more concrete. Arena’s representative credits community manager Greg’s work on Discord and the platform, while acknowledging that a large audience does not mean the consumer-product problem is solved. Nano Banana brought a sharp increase in users; those arrivals could leave just as quickly. Keeping them requires learning how they use the platform and giving them reasons to return.

Persistent conversation history gave users a reason to sign in. The representative reports sign-in as a major retention driver, without specifying a measured lift or experimental comparison. Users could preserve the conversations they came to have. At the time, Arena was also seeking specialists across consumer product, machine learning, business-to-business work, go-to-market, and marketing.

The next evaluation boundary is the agent’s harness

The closing partnership discussion turns to Cognition’s Devin. Arena’s representative proposes evaluating it as a possible collaboration. The host raises the key distinction: evaluating a model differs from evaluating an agent’s harness. Arenas, the host argues, tend to emphasize the model.

The representative identifies Code Arena as agent evaluation and suggests evolving it to support a full-featured harness. That changes the object of comparison to the fuller agent system users rely on. It remains a proposed direction, without an announced Cognition integration or reported agent benchmark result.

The final pitch is that a central evaluation platform could make an agent’s strengths visible amid negative public narratives. The suggestion that the discussed agent might rank among the best is promotional, not a finding. Arena’s closing ambition is to extend its evaluation approach to fuller systems and show what those systems do well.