Artificial Analysis

Artificial Analysis benchmarks AI models and agents, the infrastructure serving them, and the chips they run on. Cofounded by Micah Hill-Smith and George Cameron, it combines a free public comparison website with paid enterprise insights and private benchmarking. Its central methodological decision is to run evaluations itself under consistent conditions, comparing capability alongside speed and cost.[2][1]

That combination grew out of the founders’ own model-selection problems in 2023, before Artificial Analysis became their main business.[1]

Hill-Smith recalled that building a legal AI research assistant in 2023 made each stage of the application a benchmarking problem: which model could deliver sufficient accuracy at an acceptable speed and cost? He and Cameron began a comparison site as a side project while continuing their other work. Cameron described the impetus simply: “We built it because we needed it as people building in the space and thought, Oh, other people might find it useful too.”[1]

Cameron recalled buying a domain, connecting it to their Vercel deployment, and tweeting the project. Latent Space dates the public launch to January 2024. The founders credited swyx’s early amplification with bringing attention to it; swyx said an unidentified Discord community member had first brought the site to him. Hill-Smith said the founders initially paid benchmarking costs personally, spending roughly hundreds of dollars.[1]

The founders subsequently joined AI Grant’s fourth batch. Cameron described Nat Friedman and Daniel Gross as mentors who helped them work through difficult startup decisions, while other AI Grant companies supplied feedback as power users. Latent Space’s January 8, 2026 introduction reported that Artificial Analysis had raised a full seed round from Friedman and Gross.[1]

In the interview published January 8, 2026, the founders described two paying customer groups: enterprises seeking benchmarking and insight subscriptions, and enterprises or AI companies commissioning custom private evaluations. Subscription reports addressed decisions such as choosing serverless inference, managed deployment, or leasing chips to run inference internally. Public website inclusion was separate from those services: Hill-Smith stated, “So no one pays to be on the website.”[1]

Hill-Smith explained that independently running evaluations avoids combining lab-reported scores produced with different prompting methods. Repeated runs also help control statistical variance; at interview time, he described targeting approximately ±1 point at 95% confidence for the Intelligence Index. Artificial Analysis’s “mystery shopper policy” uses accounts outside its own domain to test intelligence and performance without identifying the company, addressing the possibility that a private evaluation endpoint differs from the publicly served model.[1]

The Intelligence Index combines multiple evaluations into a synthesis score, with individual results available for examining trade-offs. Hill-Smith described versions V1 through V3 as progressively harder and broader in use-case coverage, moving beyond question-answering toward agentic tasks and long-context reasoning. Artificial Analysis launched V4 in January 2026.[1][6]

AA-Omniscience, submitted to arXiv on November 17, 2025 by Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith, evaluates factual recall and knowledge calibration across 6,000 questions covering 42 topics in six domains. Its index ranges from −100 to 100, penalizing incorrect answers and rewarding abstention when uncertain; zero means equal numbers of correct and incorrect answers. The paper reported Claude 4.1 Opus leading its evaluated model set with a score of 4.8, one of only three models above zero.[3]

The founders also described an Openness Index extending beyond access to weights and licensing to disclosure of pre-training and post-training data, methodology, and training code. Its interview-time maximum was 18 points; the score measured openness rather than intelligence or the impact of a contribution. Their research included academic partnerships: Hill-Smith identified Artificial Analysis as a launch partner for Critical Point, an evaluation of difficult physics research problems.[1]

On December 10, 2025, Artificial Analysis announced Stirrup, an open source framework used in its agentic benchmarks, including GDPval-AA. Influenced by Claude Code, Stirrup lets the model choose how to approach a task while providing tools and context management. Its launch features included web search and browsing, code execution through local, Docker, or E2B backends, an MCP client, and document input and output. Automatic summarization keeps conversation history within context limits.[4]

GDPval-AA adapts OpenAI’s GDPval dataset into an evaluation that can compare models through a reference agentic harness. The founders described allowing up to 100 turns to produce work artifacts, then extracting visual and text representations for Gemini 3 Pro Preview to compare against task criteria. Cameron said the grader had been tested for alignment with human preferences. Results used relative Elo comparisons because outputs such as documents lacked a simple correct-or-incorrect ground truth; Hill-Smith emphasized that the scale had not been anchored to a human-performance threshold.[1][4]

The September 4, 2026 V4.2 update made the trade-off between index stability and relevance explicit. Artificial Analysis said it had held back changes through major model launches, then accelerated an interim release as frontier capabilities advanced. It added its in-house AA-Briefcase and Surge AI’s GDP.pdf, and removed GPQA Diamond, which it described as saturated. The company reported that private held-out test sets accounted for 40% of index weighting, double the V4.1 figure.[6]

AA-Briefcase evaluates multi-week knowledge-work projects with linked tasks and thousands of input source files, combining rubric and pairwise grading of task success, analytical quality, and presentation quality. GDP.pdf, created by Surge AI, evaluates single-turn document reasoning across 100 PDFs, ten domains, and 4,592 pages. Its responses are graded against 1,275 expert-authored criteria; the headline All-pass Rate credits a task only when every criterion is satisfied.[6]

Hill-Smith reported just over 20 people at interview time in the episode published January 8, 2026. On September 18, 2026, the company’s About page identified him as CEO and Cameron as CPO, listed San Francisco as headquarters, and also listed Melbourne and remote work locations.[1][2]

On September 18, 2026, Artificial Analysis reported 500+ models benchmarked, 100+ inference providers, 1,000+ endpoints, and 1T+ evaluation tokens. These are benchmarking-volume counters, rather than customer or revenue figures. Its listed backers were Nat Friedman, Daniel Gross, Andrew Ng, Adam D’Angelo, Clem Delangue, Guillermo Rauch, Shawn Wang (swyx), and Charlie Songhurst.[2]