Loubna Ben Allal

Loubna Ben Allal works on open foundation models, spanning data curation, pretraining recipes, and evaluation. On her personal website, she describes herself as a Research Engineer at Hugging Face and says she led SmolLM, SmolLM2, and SmolLM3 and helped build pretraining datasets such as FineWeb-Edu. Her work includes collective research on how carefully selected data and longer training can make small models more capable.[2][5][8]

Ben Allal coauthored the July 16, 2024 SmolLM announcement with Anton Lozhkov and Elie Bakouch. The family comprised 135-million-, 360-million-, and 1.7-billion-parameter models. Their account presented small models and local deployment as ways to reduce inference costs and improve privacy, while publishing the corpus and explaining its curation.[5]

A consequential decision concerned prompt design rather than a stronger generator. For Cosmopedia v2, the team tested textbooks generated by Llama3-70B-Instruct, Mixtral-8x22B-Instruct, and Qwen1.5-72B-Chat, but reported no significant improvement when training models on those alternatives. They instead improved the prompts: a book-classification-based list of 34,000 topics supplied subject coverage, and a search tool retrieved relevant web pages to serve as seeds. Each prompt combined a topic, a seed passage, and an intended audience or generation style.[5]

The team also tested whom the synthetic textbooks should address. Models trained on middle-school-oriented textbooks achieved the best reported scores on all tested benchmarks except MMLU, which includes more advanced questions. Cosmopedia v2 consequently used a mixture of 40% middle-school content, 30% college content, and 30% other audiences and styles, including stories and course-based textbooks. The resulting corpus contained 28 billion tokens across 39 million synthetic documents.[5]

Synthetic data also supplied labels for filtering real data. For FineWeb-Edu, the team used educational-quality annotations from Llama3-70B-Instruct to train a classifier that selected educational web pages. They applied a similar recipe to Python: Llama3 annotated 500,000 samples from The Stack, and a classifier retained samples scoring at least four. This reduced 40 billion Python tokens to a four-billion-token educational subset. SmolLM's corpus combined synthetic textbooks, educational Python, and 220 billion deduplicated educational web tokens.[5]

The authors reported continued gains from training beyond the Chinchilla compute-optimal point. They trained the 1.7-billion-parameter SmolLM on one trillion tokens and the two smaller models on 600 billion tokens each; gains after 400 billion tokens began slowing on some benchmarks for the smaller models. These training budgets count tokens processed during training, rather than the size of the unique corpus.[5]

Ben Allal is the first-listed author of the SmolLM2 paper, submitted February 4, 2025, with 21 other authors, including Lozhkov, Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, and Lewis Tunstall. It describes training a 1.7-billion-parameter model on approximately 11 trillion tokens in stages mixing web text, mathematics, code, and instruction-following data. Small-scale ablations and preceding-stage performance guided adjustments to the mixture. The team introduced FineMath, Stack-Edu, and SmolTalk where it judged existing datasets too small or low in quality.[8]

Before the paper appeared, Ben Allal announced the SmolLM2 training and evaluation toolkit on November 24, 2024, under the heading “Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit”. It included nanotron pretraining code, lighteval evaluation, distilabel synthetic-data generation, TRL and alignment-handbook post-training scripts, and llama.cpp tools for on-device use. She described the toolkit as Apache 2.0 licensed; the V2 pretraining data mixture was still forthcoming at that point.[6]

Ben Allal also coauthored the July 8, 2025 SmolLM3 announcement, alongside Bakouch, Carlos Miguel Patiño, Lozhkov, and other contributors. The three-billion-parameter model combined multilingual support, reasoning and non-reasoning modes, and long-context inference. The team reported a concrete tradeoff: reasoning mid-training degraded long-context performance, while APO preference-alignment training used sequences limited to 24,000 tokens. They addressed this by merging a mixture of Anchored Preference Optimization checkpoints with a mid-training checkpoint strong at long contexts, using weights of 0.9 and 0.1 respectively. They reported recovering the base model's RULER score up to 128,000 tokens. Long-context training reached 64,000 tokens; YaRN supplied the extension to 128,000 at inference.[9]

In public replies, Ben Allal explained on July 23, 2025, that reasoning mid-training used ChatML without a system prompt, whereas supervised fine-tuning used SmolLM3's final chat template. On July 31, she pointed readers to the training logs and confirmed that they had been released with intermediate checkpoints.[9]

Ben Allal presented “The State of Synthetic Data and Smol Models in 2024” at the Latent Space NeurIPS event in Vancouver in December 2024; Latent Space published the recording that month. Her personal website also identifies her as the author of The Smol Training Playbook and describes more recent work in AI for biology, including the team's release of Carbon, an open generative DNA foundation model.[2][1]

Before the small-language-model work, Ben Allal worked on code generation with BigCode. That earlier work provides context for her experience with training-data preprocessing and evaluating smaller models.[2][7]

In September 2023, GOSIM described Ben Allal as a Machine Learning Engineer in Hugging Face's Science team working on language models for code, a BigCode core-team member, and a coauthor of The Stack, SantaCoder, and StarCoder. Her own career account also includes StarCoder2 before her move to small language models.[4][2]

Ben Allal is the first-listed author of SantaCoder: don't reach for the stars!, with 40 other authors, including Raymond Li, Denis Kocetkov, Chenghao Mou, and Christopher Akiki. Submitted in January 2023, the report covered collaboration progress through December 2022. Its experiments trained 1.1-billion-parameter models on Java, JavaScript, and Python subsets of The Stack. The authors found that stronger near-duplicate filtering improved performance, while restricting files to repositories with at least five GitHub stars significantly worsened it. Their best model reportedly outperformed the larger InCoder-6.7B and CodeGen-Multi-2.7B on generation and infilling in those languages on MultiPL-E.[7]