Training IDEFICS: the engineering behind an open multimodal model

First, make the data fit

Before training a model to answer questions about images, Hugging Face’s M4 team needed to process data larger than memory. Documents required cleaning, images needed downloading, and transformations had to run in parallel. Hugo Laurençon used Hugging Face’s Datasets library throughout the pipeline: write the transformation function, then let the library handle mapping, filtering and multiprocessing. The project also pushed the library’s maintainers into workloads they had not handled before.

The result was IDEFICS, an open-access vision-language model that accepts sequences of images and text, describes images and answers questions about them. It reproduced the approach of [Flamingo](https://arxiv.org/abs/2204.14198) at 9B and 80B parameters, with instruction-tuned versions of both, including the 80B instruction-tuned model. The companion article dates development from March to August 2023. The main conversation was recorded on Halloween 2023; an appended 2024 update revisits the unfinished successor.

Hugo came to multimodality after working on the dataset and model for BLOOM, the 176B-parameter multilingual language model. Leo Tronchon joined through an internship after a master’s project combining the training objectives of two computer-vision approaches, DINO and Barlow Twins. Leo estimates that only two of the original six or seven project members had multimodal experience. Building IDEFICS developed practical intuition that papers alone had not supplied; he credited that experience and lower communication overhead with helping a smaller team move faster on the successor.

Leo saw open research as serving both Hugging Face’s business and its tools: the company benefited from releasing strong models, while internal researchers gave library developers direct feedback. M4 used Transformers for modeling and Datasets for training data, making IDEFICS a test of infrastructure other builders would use.

Connect pretrained vision and language

IDEFICS starts with pretrained backbones: existing models that supply visual representations and language capabilities. A vision encoder turns an image into representations that can be connected to a language model. Hugo recommends an encoder pretrained with text, such as [CLIP](https://arxiv.org/abs/2103.00020), rather than one trained only for image classification. He presents this as a research-backed preference, without quantifying the advantage.

An adapter provides the connection. In a lightweight design, a trainable linear layer transforms the vision encoder’s output embeddings—the image representations—and passes them into the language model. The team describes useful multimodal behavior from training this adapter alone. Flamingo uses a heavier connection: new cross-attention components let the language model access visual information within its layers. Training updates those components while keeping both pretrained backbones frozen. Whether all those extra adapter parameters were necessary remained unresolved.

The companion article identifies the large IDEFICS model’s backbones as Llama v1, specifically huggyllama/llama-65b, and laion/CLIP-ViT-H-14-laion2B-s32B-b79K for vision. The multimodal training connected these existing components; it did not train every part from scratch.

Answer wording changes benchmark scores

Established evaluations include visual question answering (VQA)—answering a question about an image—and image captioning. VQA v2 tests the former; COCO and Flickr support the latter. Because a generative model produces an open-ended answer, its wording can materially affect its score.

The same benchmark does not guarantee comparable evaluation conditions. One model may be fine-tuned directly on benchmark examples to learn the expected answer format. Another may receive a few examples in its prompt, a setup called few-shot evaluation. Scores need to be read alongside those conditions.

Question quality creates another blind spot. One participant recalled examples believed to come from VQA v2 that ranged from trivial to implausibly difficult, with some apparently wrong answers. Newer benchmarks such as MMBench and SEED were attracting attention. Generating questions and answers by prompting GPT-4 with images was another emerging approach, but it was not yet widely adopted at the time.

Why interleaved web documents mattered

Flamingo’s interleaved image-text dataset was unavailable, so M4 built an open alternative: [OBELICS](https://huggingface.co/datasets/HuggingFaceM4/OBELICS). The companion article reports 115 billion text tokens, 141 million English documents and 353 million images, curated from Common Crawl dumps spanning February 2020 through February 2023.

An image-text pair is easy to collect: download an image found in HTML and pair it with its alt text. Such pairs are noisy, but a good caption directly describes the image. An interleaved document preserves a longer sequence of text and images. Its text may not describe each image precisely, yet the elements share context. Building that sequence requires deciding which page content to keep and cleaning it carefully.

The distinction mattered by task. In the team’s experiments, LAION-style pairs supplied strong image-text alignment and worked well for captioning; web documents were particularly important for VQA v2 and OK-VQA. The team interpreted longer context as useful for reasoning and the combination as valuable data diversity. These were qualitative comparisons in the conversation, without numerical gains.

There was no precise mixture recipe. An ablation, an experiment varying a component of the training setup, substantially changed the sampling probabilities of web documents and LAION pairs. It found a workable range but no clear optimum; beyond an unspecified threshold, the mixture seemed to matter little. Evaluation covered captioning, VQA and classification, including some text recognition through captioning and VQA. Detection and segmentation were outside this setup; Qwen-VL was cited as a model trained with bounding boxes to support detection.

Preserve pixels, then clean the documents

Optical character recognition (OCR)—reading text from images—exposed two bottlenecks. IDEFICS v1 used an image resolution described as 224, then a perceiver compressed the vision encoder’s output into fewer visual tokens. Small writing lost detail during resizing, and the team suspected token compression further reduced precision. Higher resolution was also expected to help with details such as a hand’s color or a finger’s action.

The dataset retained full-resolution downloads and resized images during training, preserving flexibility for later models at a substantial storage and processing cost. M4 also wanted more explicit reading data. Nougat, trained on PDFs paired with their text, offered an example of that approach; adding similar material to IDEFICS pretraining was still a proposal.

Text quality needed a separate assessment. Alongside manual inspection, the team used perplexity, a measure derived from how probable a small language model finds a document’s text, normalized for length. Its own text-only measurements gave better scores than C4 and OSCAR. A proposed extension would use a multimodal model so that image-text alignment also affected the measurement: a relevant image should make the surrounding text easier to predict.

The document pipeline required four steps: 1. Parse raw HTML and inspect which tags carry useful content. 2. Clean the document’s tree of HTML elements and select nodes to retain. 3. Extract text and images. 4. Deduplicate and apply further filtering. The team estimated roughly a month of exploration by one or two people, followed by up to two months downloading images and running processing scripts. Filtering continued to evolve during the project.

Deduplication must run against each project’s actual collection, but the checks themselves can be shared. M4 drew on prior literature and rules developed for BLOOM. A related discussion of Together’s dataset highlighted configurable filtering: compute filter values for each document, then let downstream users choose thresholds. One participant defended the initial cleaned collection as a starting point, expecting actual training subsets to be much smaller.

Synthetic captions offered another way to improve the signal. LAION-COCO paired LAION images with captions generated by BLIP. The team reported better performance with substantially fewer examples in recent experiments, suggesting that more direct image-text alignment explained the improvement. Although no task scores or experimental configuration accompanied the observation, recaptioning became an important direction for the successor.

Training: find what explodes first

IDEFICS trained on the new web documents alongside LAION and public multimodal datasets, including Conceptual Captions. M4 tested at small scale, but fully trained the large model before the small one. Runs overlapped; the larger run started earlier because it took longer, making scheduling partly a compute-resource problem.

Available backbones determined when the goal became feasible. The team judged OPT and GPT-Neo too far behind Flamingo’s Chinchilla backbone to reach its target; Llama’s release changed that assessment. Training remained unstable, however. Even the final, more stable run needed several restarts despite checkpointing every 250 steps.

The major breakthrough was query/key layer normalization: normalizing the queries and keys used by attention. Borrowing the technique from work on a 22B-parameter Vision Transformer allowed training to proceed further. One speaker tentatively recalled its role as preventing attention from becoming effectively hard attention; the remembered mechanism was less certain than the observed improvement in trainability.

Their debugging advice was to locate the first blow-up before choosing a remedy. Aggregate activations and, where useful, gradients to find where values become unstable, then investigate bugs, normalization, problematic data, learning rate and warmup settings. These quantities interact, so a suspicious layer is a lead rather than a complete diagnosis. Periodic logging plus a custom aggregation and display script made the training internals interpretable.

A separate image-attention masking bug let a few tokens access images they should not see. A mask intended to allow attention to no images instead produced uniform attention, described as 1/n. When unrelated image-text pairs were packed together, those tokens could access images from other documents. The team said the long-web-document-only case would not be affected in the same way. It fixed the bug later; better performance was expected, but no gain was quantified. A model could still train and appear capable with this incorrect information flow.

Improve the successor by shrinking it

EVA-CLIP initially gave a substantial training boost, but instability limited its use. Once that problem was resolved, longer experiments again showed a benefit over the earlier smaller encoder. A subsequent comparison slightly favored the roughly 400M-parameter SigLIP, which was also much faster in training and inference. Resolution differed too, so this was not an isolated test of parameter count; no task-level scores were given.

The planned successor, provisionally called V2 or V1.5, combined three changes: - Better data: synthetic captions and further perplexity-based filtering of web documents. - Stronger, smaller backbones: Mistral for language and SigLIP for vision, with higher-resolution inputs. - A simpler adapter: linear layers transforming visual embeddings for direct input to the language model, replacing Flamingo-style cross-attention inside its layers.

The team reported that improved data and backbones had already let an intermediate 9B model match a Flamingo comparison, before changing the architecture. Benchmark conditions were unspecified. Removing adapter parameters was expected to bring the successor to about 7.5B and improve inference; beating IDEFICS 80B remained a goal. Commercial use also motivated the switch: the original Llama-based model had noncommercial licensing constraints, while the anticipated Mistral–SigLIP combination offered a more promising licensing path.

Separate visual hallucinations into actionable failures

Visual hallucination—producing details that do not match the image—was too broad a label to guide improvements. M4 separated it into more useful categories: - Incorrect object attributes, such as color. - Objects invented by the model. - Faulty comparisons between image elements. - Counting errors. - Incorrect descriptions of the surrounding environment. Each category pointed toward different missing training data or fine-tuning work.

Counting made the data gap concrete. Ordinary captions might mention one apple or two computers without teaching a model to count 10, 15 or 20 objects. M4 proposed purpose-built datasets for counting, OCR and particular hallucination types. More general web data would not necessarily provide that supervision.

One participant reported GPT-4V inventing menu items and prices. The team judged it less prone to some visual hallucinations, but comparisons were mostly qualitative: reproduce an IDEFICS failure in another model and see what happens. Such errors damage trust immediately when users can see the contradiction—for example, a claimed teapot that is absent from the image.

IDEFICS’s conventional evaluations did not directly measure these failure categories. SugarCrêpe, described as choosing the better of two captions for an image, and Winoground, with deliberately tricky captions, were suggested additions. Reinforcement learning with human or AI feedback was another promising direction: the hope was to teach finer-grained handling of uncertainty, though the extent of its multimodal benefit remained unknown.

Why images came before video and audio

The closing case for multimodal foundation models was practical: vision broadens the tasks a model can address. Proposed applications included photographing table parts for assembly help, asking about handwriting and solving an exercise shown in a picture. Robotics offered another reason to build a visually capable backbone. These ambitions came with an acknowledgment that hallucinations still made the models immature.

Video adds more than a series of images. A system must choose frames, decide how many to retain and accommodate arbitrary durations. Storage and preprocessing grow, and each frame adds visual tokens. One speaker illustrated the pressure with tentative estimates of roughly 256 tokens at resolution 224 and more than 700 at resolution 384. These were encoder-dependent examples, not universal tokenization rules.

Compressing frames into fewer tokens can contain sequence length, but introduces another information bottleneck. M4 judged that tradeoff premature while image-text modeling still offered substantial opportunities at lower compute cost. The project originally contemplated audio, video, text and images; audio and video were dropped when it committed to reproducing Flamingo.

For speech output, generated text could already feed a separate audio model. Spoken language remained interesting training material because it differs from writing, but the team considered its compute cost unattractive then. Multilingual coverage had also ceased to be an explicit target, although other languages remained in the data.

Even the name was becoming historical. IDEFICS expands to “Image Aware Decoder Enhanced a la Flamingo with Interleaved CrossAttentions.” Moving away from Flamingo would make that expansion inaccurate for the next iteration. The name would survive the architecture it described.

The 2024 update: release the data along the way

In the appended 2024 update, Leo said the planned new model had not yet been trained. M4 was spending more time on data, OCR and image-to-code capabilities. It also wanted to address concerns about child sexual abuse material in LAION-derived training data before starting the run.

Meanwhile, M4 released [WebSight](https://huggingface.co/datasets/HuggingFaceM4/WebSight) to demonstrate useful synthetic image-to-HTML data at scale with open models. Hugo led the effort using Mistral and DeepSeek Coder. The companion article adds the construction details: 823,000 HTML/CSS examples for synthetic English websites, each paired with a screenshot rendered using Playwright. The pairs supplied visual inputs and code targets for screenshot-to-code training.

The team had fine-tuned an early IDEFICS 2.0 version on WebSight for a demo and made the model, demo and dataset available on the Hub. Following WebSight’s reception, it planned incremental releases: publish datasets, architecture details and training insights during development, then release the model. The work between major training runs would become useful to other builders as it happened.