Artificial Analysis: What a Model Benchmark Has to Measure

Every application stage becomes a benchmarking problem

In 2023, one of Artificial Analysis’s founders was building a legal AI research assistant with a multistage workflow. Each stage raised the same question: what is the minimum viable model for this part of the job? Answering it meant balancing accuracy, performance and cost at each step.

Artificial Analysis began as a side project to expose those trade-offs across models and hosting providers. The founders continued their other work, bought a domain and connected it to an existing Vercel deployment. They describe Mixtral 8x7B as an early turning point: its arrival encouraged developers to consider alternative providers for serverless inference, making differences in serving speed and price more consequential.

At the time of the conversation, the company reported just over 20 staff. Its customers included enterprises choosing AI technologies and companies building across the AI stack that needed private benchmarking. Paid work ranged from standardized deployment reports—comparing serverless inference, managed deployments and leased chips—to custom evaluations. The team’s stated boundary was clear: public website data remained free, nobody paid to appear, and customers could not buy better results.

A score depends on prompting, parsing and repeats

A question-and-answer benchmark looks simple: ask questions and check answers. But when models compete over a few points, prompting differences can change the comparison. Artificial Analysis chose to run evaluations itself under consistent conditions rather than assemble lab-reported scores, and to interpret capability alongside cost and performance.

Answer parsing is part of the measurement. A response can contain the correct answer and still receive zero because it violates the expected format. Whether that is appropriate depends on the evaluation’s purpose. If the target is reasoning alone, a large language model can extract the answer independently of its presentation. If output-format compliance matters, a format failure may legitimately count. The team says clear instructions and examples increasingly make extraction with a regular expression sufficient. Randomizing multiple-choice options can also mitigate answer-position bias.

Repeated runs introduce another concern: the team reports substantial score variation for reasoning models at lab-recommended temperature settings, particularly on small question sets. During evaluation development and index updates, it runs many trials to choose repetition counts. Its target is a 95% confidence interval of approximately ±1 point or tighter for the aggregate Intelligence Index. That precision target applies to the combined index, not necessarily to each component benchmark.

Repeats also change the economics. The founders initially spent roughly hundreds of dollars across their evaluations: there were few models, simpler tasks and often direct answers without intermediate reasoning. Repeated trials multiply spending. At recording time, the website’s cost-to-run figure for the Intelligence Index assumed one repeat; Artificial Analysis’s actual spending was substantially higher.

Independence requires testing the public product

Consistent prompts cannot establish that a private evaluation endpoint serves the same model or configuration as the public API. Artificial Analysis’s mystery shopper policy addresses that risk: it registers accounts outside its own domain and runs intelligence and performance tests without identifying itself as the evaluator. The team says it discloses the policy to labs and has encountered no objections, partly because each lab wants assurance that competitors cannot manipulate the comparison.

A harder problem survives honest execution. Once a benchmark becomes influential, researchers optimize for it. Scores can improve without equivalent gains in broader usefulness, even without training on the test set. Competition mathematics illustrates the gap: models can excel at those problems without the gains transferring one-to-one to coding-agent work. The team’s response is to keep developing evaluations around tasks users need.

Participation in AI Grant’s fourth batch helped connect that work to builders. The team credits Nat and Daniel with helping it work methodically through ambiguous startup decisions. Other companies in the program supplied demanding product feedback: they combined models, switched them between application stages and optimized accuracy, speed and cost. Many were power users of the public data without being paying customers.

An intelligence index has to outgrow its original tasks

At recording time, the Artificial Analysis Intelligence Index combined ten evaluation datasets, including question-and-answer tests, agentic tasks—work performed through multiple steps and tools—and an in-house long-context reasoning dataset. The team presents it as its preferred single measure of intelligence, while retaining individual measurements to expose trade-offs that an aggregate score hides.

Across V1, V2 and V3, the index became harder and covered more use cases. A team member argues that almost every newly released model would now saturate the original version, leaving it unable to distinguish their capabilities. HumanEval, which asks models to write Python functions, exemplifies work that had become relatively trivial. Priorities shifted toward coding agents, comparable workflows in other professions and reliable reasoning over long contexts.

An evaluator recalls running DeepSeek-V3 tests on Boxing Day in New Zealand in 2024, watching results arrive while visiting family. Artificial Analysis ranked it as the best open-weights model at that point. The evaluator regards its strong base model as the foundation for R1’s reinforcement-learning success a few weeks later.

The terminology evolved with the coverage. The former Quality Index became the Intelligence Index. When hardware benchmarks introduced system-level measurements, the earlier throughput metric became output speed, reserving throughput for the system level. This distinguishes the speed one user experiences from the total work serving hardware can handle.

Knowing facts and admitting ignorance are different capabilities

The Omniscience Index asks what factual knowledge a model contains and what it does when it cannot answer correctly. Its score ranges from −100 to +100, with a point deducted for an incorrect answer. The rationale is behavioral: percentage-correct scoring encourages an attempt at every question, whereas a penalty can make abstention—declining to answer—preferable to misinformation.

That is distinct from confidence calibration: whether a model’s stated confidence matches how often it is right. Participants disagree about eliciting confidence directly. One team member argues against asking models how confident they are; others leave open a JSON confidence field. No calibration result resolves the disagreement. Omniscience measures factual answering and abstention, and its hallucination rate applies specifically to that setting.

The reported results separate knowing more from handling ignorance better. Claude models had the lowest hallucination rates among those evaluated at the time. Gemini 3 Pro substantially improved factual accuracy over earlier Gemini models without a comparable change in hallucination rate. The evaluators report no strong relationship between general intelligence and willingness to admit ignorance. Differences in training after the base model is built are a proposed explanation for Claude’s behavior, not an established cause.

Only 10% of the Omniscience test set had been published. The team withheld most questions because factual recall tests are especially vulnerable to contamination and planned updates to preserve their usefulness. Some topic-level breakdowns were already available, with finer breakdowns planned. The team also acknowledged that its earlier intelligence scoring had not penalized hallucination this way.

The desired behavior changes with the task. For factual questions, admitting ignorance can be more useful than guessing; for generating ideas, an uncertain attempt can help. Academics involved in Critical Point, a benchmark of difficult physics research questions, reportedly used high temperature settings to encourage speculative outputs while exploring new ideas. Artificial Analysis was a launch partner for the benchmark. Its broader strategy combines internal development with academic and company partnerships, with disclosure where independence concerns arise.

Omniscience also prompted a discussion of model size. The team says factual accuracy tracks total parameter count more closely than its other metrics, without supplying a correlation statistic. Extrapolating that relationship to undisclosed frontier-model sizes remains conjecture. Its advice to developers is more actionable: compare measured capabilities, token consumption and cost at actual serving prices.

GDPval evaluates deliverables, not just answers

GDPval covers white-collar work beyond coding. Tasks provide detailed instructions and often files such as spreadsheets, presentation decks and PDFs. The team compares them to difficult take-home interview assignments: sufficiently specified to run as an evaluation, but narrower than workplace activity because they require bounded inputs and explicit instructions.

Artificial Analysis turned OpenAI’s dataset into a cross-model evaluation using an agent harness, the software that gives a model tools and supports its multistep execution. Creating and grading the work follow separate workflows: 1. The tested model uses code execution, web search and a filesystem over multiple turns to produce deliverables. 2. A grading pipeline extracts visual and textual representations of the files. 3. Gemini 3 Pro Preview compares two outputs against the task’s explicit criteria and selects the better match. The judge assesses completed work rather than repeating the agent’s production process.

The team reports good alignment with human preferences after extensive testing, attributing it to the judge’s capabilities, suitable file representations and explicit criteria. It gives no numerical agreement rate or validation protocol. Gemini’s relatively weak performance as a contestant is offered as evidence that obvious self-preference was not appearing, though that is not a direct measurement of judge bias.

One task asks models to assemble a marketing video from clips found online. The team says models struggle: the work is difficult through a code interpreter, and computer interaction was not yet reliable enough. Deliverables like this explain the choice of Elo, a relative rating based on comparisons between outputs. Documents, audio and video do not always have a single correct result or a useful binary pass/fail label.

A participant proposes adding contractor-produced work as a human reference. That remained a suggestion: the Elo scale was not calibrated to a human-performance threshold. The designers chose it to accommodate many models and remain useful as new ones arrived, while noting that people and models may approach the same assignment differently.

The harness changes what the model can accomplish

In the reported GDPval comparisons, every tested model performed better in Artificial Analysis’s harness than in its corresponding web chatbot; Claude Opus 4.5 was the concrete example. Participants point to differing tool freedom, constraints and cost goals, with the evaluation allowing extended work. The chatbot references came from manual copy-and-paste and were not expected to be refreshed at the same scale. These results compare execution environments as well as models.

The surrounding application matters in everyday work too. Useful email drafting may require old emails, meeting notes, workplace files and database analysis. One participant describes models using read-only Supabase access to run SQL queries and create charts, alongside Gmail and Notion access, with varying support in ChatGPT and Claude in late 2025. The same participant qualifies the enthusiasm: the integrations barely worked at the time. The example shows the promise of connected context, not dependable automated drafting.

Artificial Analysis released its generalist harness as Stirrup, reportedly the day before the conversation. Its default components were deliberately modest: - Context management. - Web search and browsing. - A code execution environment. GDPval also needed a custom image-viewing tool: terminal operations could bring text into context, but visual material needed a separate route. The suggested customization workflow was to clone the small codebase and have a coding agent adapt it.

The team had also examined Harbor from evaluation and coding-agent perspectives and hosted Terminal Bench results. Its architectural view is that improving intelligence and tool use increasingly favor giving models a minimal toolset and letting them control the workflow, rather than prescribing the flow through a more elaborate framework.

Openness measures what developers can reuse

Artificial Analysis mixes open and proprietary components; its long-context reasoning dataset, identified as AALCR, was open-sourced. Its Openness Index extends the question beyond whether model weights are downloadable. The index considers licenses, disclosure and permitted use of pre-training and post-training data, methodology transparency, and training code. At recording time, these categories contributed to a maximum score of 18.

A participant challenges the weighting: should a little additional training data count as a greater contribution than substantially more capable open weights? The team distinguishes what was released from its industry impact. It wants clearly defined categories and argues that releasing data and training code remains valuable even when the associated model is below the frontier.

Coverage also affects the comparison. A team member describes AI2’s OLMo 3 32B as an openness leader, prompting a challenge about a missing small Hugging Face model. The team says intelligence benchmarking was still needed before inclusion and promises to add it. The intention is to maintain openness alongside the Intelligence Index so developers can compare the two dimensions.

Licensing questions helped motivate the index. Familiar licenses such as MIT and Apache 2 occupy its top category; other categories distinguish additional conditions or commercial-use restrictions. The team argues that standard licenses simplify adoption decisions. Participants also credit NVIDIA’s Nemotron work with substantial investment and use in synthetic-data generation, while recognizing its alignment with NVIDIA’s commercial interest in expanding AI adoption.

Cheaper intelligence can still produce larger bills

Artificial Analysis tracks the cost of running its Intelligence Index within buckets of capability scores. The team reports rapid declines within each bucket, with newer capability levels experiencing faster reductions. Estimates in the conversation put GPT-4-level intelligence at over 100 times cheaper than GPT-4’s launch pricing, or roughly 1,000 times cheaper using Amazon Nova models. These are differing estimates without a specified common capability threshold or pricing basis; the cost material being discussed was also acknowledged to need updating.

The mechanism behind the team’s inference-cost “smile” is clearer than the exact multipliers: costs fall at a fixed capability level while the work developers ask models to do expands. Developers still choose frontier models, spend tokens on reasoning and run agents through long sequences with large input and output volumes. Lower unit costs and larger total bills can coexist.

Hardware comparisons need a concrete workload too. The proposed procedure is to select a relevant model, choose a realistic per-user serving speed, then measure throughput per GPU at that speed. Serving each user faster can reduce aggregate throughput and raise costs. One evaluator argues that moving from NVIDIA Hopper to Blackwell can yield substantially more than a two- or threefold gain for large sparse models under realistic serving conditions, but supplies no specific configuration or measured result in the exchange.

The remaining room for sparsity—using a smaller active share of a model’s total parameters—is contested. One participant proposes roughly 5% active parameters as a possible floor; another disputes it, recalling approximately 3% for Kimi K2 while expressing uncertainty. The team sees Omniscience accuracy’s reported relationship with total rather than active parameters as a reason further sparsity might be possible, not as an established lower bound or scaling law.

One evaluator forecasts that hardware gains will combine with software and model improvements to drive another order of magnitude of decline in fixed-capability costs—and another order of magnitude of growth in how much inference developers can use. The forecast extends both sides of the same argument: cheaper work enables more ambitious workloads.

Measure the cost of finishing the task

The reasoning/non-reasoning distinction had become less predictive of cost. Earlier in the year discussed, the average reasoning model used about ten times as many tokens per Intelligence Index query as the average non-reasoning model. By the conversation, selectable reasoning strengths and more than an order of magnitude of variation in token efficiency made that binary comparison less useful.

Artificial Analysis retained the classification: a reasoning model provides a separate chain of thought, or intermediate reasoning, normally through a distinct API parameter. But the label no longer necessarily implies more tokens or longer end-to-end latency on the same task. The discussion also distinguishes GPT-5’s routed ChatGPT consumer experience from API access, where developers can select versions and reasoning strengths.

Per-token price is only one part of task cost. The team recommends cost to run the Intelligence Index as a starting comparison because it includes differences in token consumption. It also tracks whether models spend more tokens on harder questions and fewer on easier ones. Those analyses remained internal, although the team reported improvement over the year.

For agents, the number of turns matters too. On Tau-2bench Telecom, a team member reports that GPT-5 was cheaper to run than some smaller open-source models despite its higher per-token price, because it resolved customer queries in fewer turns. The comparison does not identify the smaller models or quantify savings. Its practical lesson is to measure what the complete workflow consumes, including both tokens and turns.

Multi-turn evaluation is already part of the service. GDPval AA permits up to 100 turns through Stirrup. Running such evaluations quickly across hundreds of models creates substantial infrastructure and parallelization challenges; the team says it has built support for that workload.

New behaviors, new tasks and explicit version boundaries

Beyond language models, Artificial Analysis benchmarks speech, images, video and hardware. Its video arena uses pregenerated outputs so voters can compare preferences without waiting for generation. The team selects image and video categories around uses it believes matter to creative professionals. Users can submit prompts for neglected weaknesses, potentially turning those weaknesses into measured targets. One proposed category is infographics that faithfully follow supplied references, emphasizing routine professional work alongside artistic output.

The team forecasts continued progress along existing trends, with another year of gains potentially making AI feel substantially different. One participant argues that demand for smarter AI will remain insatiable: people would rarely turn down an opportunity to make themselves or a colleague better at their job. Artificial Analysis plans to keep measuring raw intelligence while expanding into model behavior and possibly personality. Hallucination is the starting point; a dedicated personality benchmark remains only a possibility.

At recording time, the Intelligence Index was at V3, with V4 planned to incorporate GDPval, Critical Point, Omniscience and hallucination rate. The exact combination was unspecified. Different numerical scales make weighting consequential, and the team says it wants to avoid misleading distortions. Its commitment is to keep scores stable within each version: the version number is part of the measurement. V4.1 scores were explicitly described as incompatible with V4 scores.

The last technical exchange returns to benchmark quality. A participant raises concerns that some Tau Bench tasks may be impossible. The response allows that earlier versions could have problems but expresses confidence in Tau-2 Bench Telecom, where increasingly strong model performance is the team’s more immediate concern. The exchange supplies no task audit to settle the broader dispute.

The closing thanks return to the community that helped the project find an audience. An unidentified Discord member first brought Artificial Analysis to the podcast’s attention. A developer’s need became a shared tool, and another community member recognized something worth passing along.