Elie Bakouch

Elie Bakouch contributed to Hugging Face’s SmolLM series, coauthoring the original small-model release and the later SmolLM3 training recipe. His work spans training data, efficient architectures, and the tradeoffs between reasoning and long-context performance. On April 6, 2026, he announced joining Prime Intellect to work on pretraining and mid-training.[3][4][6]

Bakouch described Prime Intellect’s appeal as “pushing the frontier in the open.” In the Hugging Face Science AMA, he argued that training data is the most important part of model training, including beyond small models, while treating data and architecture as largely independent dimensions to optimize.[6][5]

Before that move, Bakouch was the first-listed author of the July 8, 2025 SmolLM3 announcement, alongside Carlos Miguel Patiño, Anton Lozhkov and a wider collaborative team. The release paired a 3-billion-parameter model with a public training recipe covering pretraining, reasoning adaptation and post-training. It supported six languages, selectable reasoning and non-reasoning modes, and context windows up to 128,000 tokens.[4]

The team tested architectural changes in 3B models trained on 100B FineWeb-Edu tokens. It reported that grouped-query attention matched multi-head attention performance while reducing the inference key-value cache. Bakouch separately explained the choice to split training across two GPUs with tensor parallelism: the model did not fit on each GPU for training without ZeRO or activation recomputation, and the split had little effect on throughput.[4]

Reasoning adaptation introduced a consequential tradeoff: the team traced declining long-context benchmark performance to reasoning mid-training. Its solution combined an Anchored Preference Optimization checkpoint “soup” with a mid-training checkpoint that retained strong long-context performance. The team reported that a 90%/10% linear merge recovered the base model’s RULER score on contexts up to 128,000 tokens.[4]

Bakouch emphasized that intermediate checkpoints gave other researchers control over adaptation: they could repeat the decay phase with a chosen language mixture, use continual learning, or apply supervised fine-tuning. For specialized tasks, he favored fine-tuning or reinforcement learning on existing small open models over training from scratch “in most cases.” The release also supplied practical reproduction material: Patiño confirmed the training datasets’ release in SmolTalk2 on July 15, 2025, and Loubna Ben Allal confirmed the release of training logs and intermediate checkpoints on July 31.[5][4]

The series began before SmolLM3. Bakouch recalled doing his end-of-studies internship with Loubna and Leandro and staying immediately afterward. On July 16, 2024, he coauthored the original SmolLM announcement with Loubna Ben Allal and Anton Lozhkov. The team released 135M, 360M and 1.7B parameter models, together with a corpus combining synthetic educational text, educational web pages and filtered Python code.[5][3]

The original team reported continued gains from longer training beyond the Chinchilla optimal point in its scaling experiments. It consequently trained the two smaller models on 600B tokens each and the 1.7B model on 1T tokens. Its experiments also established limits: the tested stronger synthetic-text generators produced no significant improvement, and neither adding instruction datasets nor upsampling curated Cosmopedia subsets during cooldown significantly improved results.[3]