Hugo Laurençon

Hugo Laurençon is an AI researcher whose contributions include the Idefics vision-language models, the WebSight screenshot-to-code dataset and data-filtering tools for BigScience’s ROOTS corpus. His work with collaborators combines released models and datasets with experiments that explain consequential training choices. He uses the Hugging Face handle HugoLaurencon.[2][3][5][7]

In May 2024, Laurençon, Léo Tronchon, Matthieu Cord and Victor Sanh published What matters when building vision-language models?. They argued that unsupported design decisions impede progress by obscuring which choices actually improve performance. Their experiments covered pretrained components, architecture, data and training methods, culminating in Idefics2, an eight-billion-parameter vision-language model. The team released base, instructed and chat versions alongside datasets created for training. The paper states that author order was chosen randomly.[3]

A central choice was how images enter the language model. In a cross-attention architecture, encoded image information is injected through additional blocks interleaved with the language model’s layers. Idefics2 instead uses a fully autoregressive architecture: visual tokens are concatenated with text embeddings and passed together into the language model. Pooling the visual tokens into a shorter sequence can reduce computation.[3]

Laurençon’s subsequent paper with Andrés Marafioti, Victor Sanh and Léo Tronchon explained why this comparison depends on training conditions. In experiments using Mistral-7B and the SigLIP-SO400M vision encoder, cross-attention performed better when both pretrained backbones remained frozen. Adapting parts of both backbones with LoRA reversed the comparison, despite the cross-attention model having more parameters overall. The study did not evaluate text-only benchmarks, leaving that aspect of the trade-off unmeasured.[4]

That August 2024 paper, Building and better understanding vision-language models: insights and future directions, also presented Idefics3-8B as a practical tutorial in building a vision-language model. The collaborators trained it exclusively on open datasets and introduced Docmatix to improve document understanding. They reported that Idefics3 significantly outperformed Idefics2 and released the model and datasets created for its training.[4]

Laurençon, Tronchon and Sanh also explored converting interface screenshots into HTML. Their March 2024 WebSight paper introduced an open synthetic dataset of two million HTML-and-screenshot pairs. They fine-tuned a vision-language model on those pairs to generate code from webpage images, addressing what they described as a shortage of suitable training data for the task.[5]

For the resulting Sightseer model, the team selected training checkpoints by manually inspecting generated samples. Validation loss kept falling over several epochs, but this did not translate into better generalization to websites unlike those in the training set. Their evaluation decision therefore depended on the quality of the generated pages as well as the training metric.[5]

The authors documented practical limits: Sightseer struggled with complex layouts, excessive text and designs far from its training data. Some generated images, text or buttons disappeared when the code was rendered, including cases caused by text matching its background color or incorrect syntax. These failures qualified the model’s demonstrated screenshot-to-HTML capability.[5]

These 2024 model-building projects followed Laurençon’s earlier work on training corpora. That history includes both explicitly credited data-engineering contributions to ROOTS and collaborative work on interleaved image-text documents for the original IDEFICS models.[7][6]

For BigScience’s ROOTS corpus, Laurençon developed the filtering library used to clean OSCAR and a visualization tool to help choose filtering parameters. He ran OSCAR filtering jobs and contributed to dataset cleaning, OSCAR deduplication and the paper’s writing. The larger collaborative corpus comprised 1.6TB across 59 languages and was used to train the 176-billion-parameter BLOOM language model.[7]

Laurençon was then a coauthor of OBELICS, first submitted in June 2023. Unlike collections of isolated image-caption pairs, OBELICS preserves documents in which images and text are interleaved. The paper’s revised version described 141 million Common Crawl web pages, 353 million associated images and 115 billion text tokens. The team released the dataset, code and nine- and eighty-billion-parameter IDEFICS models to demonstrate its use.[6]

Laurençon’s academic work also centered on vision-language training: Matthieu Cord’s research-team roster lists him among alumni with a 2023–2025 PhD entry on that topic. In January 2024, he appeared alongside Tronchon on Latent Space, introduced by the publisher as members of Hugging Face M4, for an episode on training large multimodal models.[8][1]