Clémentine Fourrier

Clémentine Fourrier is an AI researcher at Hugging Face whose work includes the Open LLM Leaderboard, lighteval, the LLM Evaluation Guidebook, and the GAIA assistant benchmark. She led Hugging Face’s LLM and agent evaluation efforts and collaborations between 2023 and 2025. In March 2025, she announced the leaderboard’s retirement, explaining that the team wanted to stop it before obsolete benchmarks encouraged optimization in irrelevant directions.[2][6][5]

She uses the handle clefourrier. As of September 2026, her personal site identifies her as a Hugging Face researcher and says she is on sabbatical until December 2026.[2][3]

Fourrier helped develop the internal evaluation suite that became lighteval and worked with engineer Nathan Habib to turn an existing research leaderboard into a production engineering service. In her July 2024 Latent Space interview, she credited Edward Beeching and Lewis Tunstall with the initial benchmark choices, drawn from evaluations common in reinforcement-learning papers. Tunstall also helped select evaluations for version two.[1]

Her work involved checking whether implementations measured what they were supposed to measure. She recalled removing DROP from version one after discovering that its implementation treated a dot as an end-of-sentence token, truncating floating-point answers and marking them incorrect. For version two, the team spent about a month testing evaluations across models, checking formatting, fixed few-shot examples, and tokenizer behavior.[1]

Fourrier reported that the team’s MMLU prompt-format experiments produced a 30-point variation on a 100-point scale, despite asking about the same content. The team chose log-likelihood-based multiple-choice evaluation partly because it was cheaper and easier to batch than generating answers. It selected only level-five MATH questions to distinguish model performance while limiting compute costs.[1]

Those choices reflected practical constraints. In July 2024, Fourrier explained that leaderboard evaluations used spare capacity on Hugging Face’s shared research cluster. Their jobs had the lowest priority and could be interrupted when other research filled the cluster, making evaluation queues dependent on available capacity.[1]

On March 13, 2025, Fourrier announced the leaderboard’s retirement after reporting that the team had evaluated more than 13,000 models over two years. Reasoning models and language-model assistants were changing what evaluations needed to test. She wrote that the leaderboard “could encourage people to hill climb irrelevant directions in the field.” In a follow-up, she directed users toward capability-specific community leaderboards and explained that there was no single drop-in replacement.[5]

In her May 2024 essay, “Let’s talk about LLM evaluation”, Fourrier separated three purposes often conflated: checking whether training is sound, ranking models, and assessing capabilities. Training checks need score trajectories and expected performance ranges. A capability claim requires more caution: success on a task is only a hoped-for proxy for a broader ability. She argued that evaluation still provides useful signals about training methods and model performance within those limits.[4]

Fourrier has been skeptical of LLMs as judges, warning in 2024 about preferences for their own model families, first-position answers, and verbose responses, alongside inconsistent numerical scoring. If a model judge must be used, she recommended reproducible models such as Prometheus or JudgeLM and relative rankings rather than scores. Her essay qualified the suggestion that smaller specialist judges might reduce subtle bias: that possibility remained to be rigorously tested.[1][4]

She also distinguished human preference from factual correctness. In July 2024, she described arenas as useful sociological experiments about users and their interactions with models, while questioning their reproducibility as capability evaluations. Her practical advice to engineers was to use general leaderboards to identify promising candidates, then test those models on their own use cases.[1]

Fourrier made the LLM Evaluation Guidebook to share the practical insights and theoretical knowledge her team gathered while managing the Open LLM Leaderboard and designing lighteval. Her announcement emphasized that shared experience; her personal site describes the guidebook as one of the team’s principal projects.[8][2]

Fourrier coauthored GAIA with Grégoire Mialon, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Submitted in November 2023, the benchmark uses real-world questions requiring reasoning, multimodal information, web browsing, and tool use. Fourrier explained that the team began with tasks it expected assistants to perform, rather than confining agents to artificial environments with narrowly specified APIs.[6][1]

The GAIA paper introduced 466 questions and withheld answers to 300 for leaderboard evaluation. Its November 2023 abstract reported 92% performance for human respondents versus 15% for GPT-4 equipped with plugins. That historical comparison illustrated the benchmark’s focus on questions that were conceptually simple for humans but challenging for the tested assistant.[6]

Before her evaluation work, Fourrier trained in geology and pursued machine-learning research in other domains. In her 2024 interview, she recalled graduating in geology in 2015, moving into computer science, working on brain-disease prediction, and later becoming a research engineer before an INRIA-funded PhD. Her personal timeline lists an October 2022 PhD on neural approaches to historical word reconstruction. She described geology as an experimental science and said she viewed machine learning similarly.[1][2]

In May 2022, Fourrier and Benoît Sagot published “Probing Multilingual Cognate Prediction Models”. They investigated character-based neural translation models used for cognate prediction, a historical-linguistics task. Their probing found that the models stored linguistic and diachronic information, but did not do so in the ways earlier interpretations had assumed.[7]

Near the end of her PhD, Fourrier decided against staying in academia and applied for an internship at Hugging Face. She initially worked on pre-trained graph transformers. According to her 2024 account, the team stopped the project after her three-month internship and three further months as an employee because it was difficult to accomplish with the resources available. It left reusable graph-machine-learning artifacts. She then spent a month reading evaluation papers and began the internal evaluation work that became lighteval.[1]