Micah-Hill Smith

Micah Hill-Smith is a cofounder of Artificial Analysis, alongside George Cameron. His work focuses on independently comparing AI models and providers across intelligence, speed, and cost. He and Cameron chose to run evaluations themselves under consistent procedures, rather than combine laboratory results obtained with different prompts—a decision Hill-Smith explains as essential to useful model comparisons.[2][1]

In an interview published January 8, 2026, Hill-Smith described the controls behind that approach. During benchmark development and Intelligence Index upgrades, the team repeated evaluations to choose repeat counts targeting approximately ±1 index point at 95% confidence. He also described a “mystery shopper policy”: registering accounts outside the company’s domain to check intelligence and performance on public endpoints without providers readily identifying Artificial Analysis. The precaution addresses the possibility that a private evaluation endpoint could differ from what ordinary users receive.[1]

Hill-Smith argues that benchmark relevance requires continual renewal. Once a test becomes a target for model developers, scores can improve without equivalent gains in broader capabilities. He distinguishes this from training on test sets: optimizing for a measured task can itself narrow what progress the score represents. His response is to build new evaluations around capabilities that matter to users.[1]

One example is the team’s factual-knowledge evaluation, which tests both what models know and whether they abstain when they do not know an answer. Hill-Smith favors penalizing incorrect answers because ordinary percentage-correct scoring encourages guessing. As he put it, “There’s no incentive to say, I don’t know.” He said only 10% of this test set had been published, with most questions held back to reduce contamination. He treats it as a specific measure of knowledge and hallucination, rather than a complete account of hallucination in every use case.[1]

For GDPVal AA, Cameron described adapting OpenAI’s GDPVal dataset into an evaluation that could run across models, using a reference agent harness and Gemini 3 Pro Preview as an evaluator. Hill-Smith explained the distinction between producing an artifact and grading it: an agent uses tools over many turns to create files; a separate pipeline extracts visual and text representations of those files for comparison against the task’s criteria. The founders reported testing the evaluator’s alignment with human preferences.[1]

The team released its compact agent harness as Stirrup. Cameron described a minimalist set of capabilities, including context management, web search and browsing, and code execution. Hill-Smith explained that GDPVal also required an image-viewing tool to bring visual information into the model’s context. He recommended adapting the small codebase with a coding agent. He said the GDPVal evaluation allowed models up to 100 turns, extending benchmarking beyond single-response questions.[1]

Hill-Smith’s explanation of the team’s Openness Index separates access to model weights from disclosure of how a model was made. Its categories cover pretraining and post-training data, methodology, training code, and licensing. He defended using defined disclosure categories rather than estimating a contribution’s industry impact: “It’s just how much of your data did you release?” In his view, releasing data and training code can be useful even when the resulting model is not the strongest in its class.[1]

He also distinguishes falling costs at a fixed intelligence level from the possibility of spending more on AI overall. Frontier models, reasoning tokens, and long agent workflows can multiply consumption even as older capability levels become cheaper. For application builders, he argues that both token efficiency and the number of turns needed to complete a task matter; their relative importance depends on the application.[1]

These evaluation choices have an earlier practical context: before Artificial Analysis became his main work, Hill-Smith was trying to choose models for a legal AI assistant.[1]

Hill-Smith tried GPT-2 during his computer science degree and later worked as a McKinsey business analyst in Sydney from 2020 to 2022. In a 2025 alumni interview, he recalled working on big-picture strategy projects for banks and said McKinsey’s approach to feedback and culture had informed his career.[2][3]

In 2023, he was building a multistage legal AI research assistant. He recalled that each stage became a benchmarking problem: finding the “minimum viable model” while balancing accuracy, performance, and cost. He and Cameron began Artificial Analysis as a side project to make those trade-offs visible across models and providers. Hill-Smith initially continued working on the legal assistant; Cameron likewise described building the comparison tool because they needed it themselves.[1]

His later public views emphasize augmentation. In the 2025 McKinsey interview, he argued that AI could free consultants from manual deliverables to spend more time on substantive strategy: “The value of BAs and consultants in general was never the manual work of putting things into a slide; it’s the big ideas.” He also singled out coding agents as the most magical tools in his daily work that year.[3]