Synthetic data and the case for smaller models

Specify the training data you want

Training a language model from web pages starts with a constraint: you can filter what exists, but you cannot specify what was written. Synthetic data—material generated by a model—adds that control. A prompt can request the subject, audience and presentation you want in the training corpus. In late 2023 and 2024, this approach expanded into pretraining after already finding uses in instruction training and evaluation, where language models judge other models’ answers.

An early example in the 2024 talk sketches how far this could go: train a one-billion-parameter model on 150 billion Cosmopedia tokens, teach it to answer instructions with generated examples, apply another synthetic-data training stage called DPO, then evaluate with model judges through MT-Bench or AlpacaEval. This is a proposed workflow, not a measured end-to-end result; the talk does not explain the DPO procedure.

Stronger open and closed models made such generation useful. Lower cost and faster turnaround than human annotation made it attractive, while inference frameworks such as vLLM, TGI and TensorRT supported generation at scale—given enough GPUs.

Model collapse depends on the training setup

Once generated text enters the web, even a nominally web-trained model may learn from other models’ outputs. The FineWeb team investigated this by tracking expressions such as “as a large language model” and the frequency of “delve” across Common Crawl snapshots. Their ratios rose after ChatGPT’s release, suggesting more synthetic content. But word-frequency proxies neither measure its total share nor reliably identify individual synthetic pages.

Models trained on newer crawl snapshots also performed better in a reported aggregate comparison on language benchmarks. Newer web data remained useful in that experiment, but the comparison did not isolate synthetic content’s contribution. No numerical scores or benchmark list accompanied the account.

The argument against inevitable model collapse, or deterioration through repeated training on generated outputs, rests on a distinction between two setups: - Repeated self-training: a small model generates continuations, trains on them and repeats. The speaker argues that this process gives little reason to expect improving quality. - Large-to-small distillation: a larger model supplies training material that transfers some of its knowledge to a smaller model. Here, improvement is the intended outcome. The talk argues that careful curation can avoid collapse. That is a position about how to design the pipeline, not a guarantee for all synthetic data.

Cosmopedia: diversity starts with the seed

Microsoft’s Textbooks Are All You Need work made synthetic pretraining compelling: small models trained on textbooks generated by a larger language model reportedly outperformed much larger models. Unpublished training data, however, left questions about possible contamination that model weights alone could not settle. Hugging Face’s [Cosmopedia](https://huggingface.co/blog/cosmopedia) attempted an open reproduction of the documented approach, producing almost 30 billion tokens of textbooks, blog posts and stories with inspectable training material.

The practical obstacle was repetition. Repeated requests for a linear-algebra textbook produced similar outputs even when sampling temperature changed. That would not scale to millions of distinct samples. Cosmopedia instead varied the information in each prompt: a web-page extract supplied a seed, and the generated textbook had to relate both to that extract and to a chosen subject.

For example, an extract about cardiovascular bioimaging seeded a medicine textbook. Selecting those extracts became a retrieval problem: the team needed pages relevant to its subjects, not arbitrary web text. A search tool over FineWeb organized topics, subtopics and matching pages to supply varied, relevant seeds.

The intended audience also changed downstream performance. College-level textbooks helped on the MMLU benchmark, while middle-school textbooks helped on OpenBookQA. No effect sizes were given, but the observations suggest testing presentation styles against the intended evaluation rather than choosing one universal textbook voice. Cosmopedia V1 and V2 also reportedly outperformed FineWeb throughout a training comparison; the metric, model sizes and numerical advantage were not specified.

Rewrite existing information to reduce the generator’s burden

Generating a new textbook asks a model to supply substantial knowledge. [Rephrasing the Web](https://arxiv.org/abs/2401.16380) instead takes samples from the C4 dataset and rewrites them as Wikipedia-style passages or question-and-answer pages. Because the source supplies the information, the speaker argues that a smaller generator can do this work. Training on the rewritten material reportedly improved on C4 alone, although no quantitative comparison was given.

Nemotron-CC scaled rewriting to a reported 1.9 trillion synthetic tokens, with two purposes: - Improve low-quality pages: rewrite them into a more useful presentation, such as a Wikipedia-style article. - Reuse high-quality information: turn already-filtered pages into question-and-answer pairs, open-ended questions or multiple-choice questions. The information recurs, but the presentation varies without verbatim duplication.

ProX generates cleaning programs instead of directly rewriting each page. Those programs normalize text and remove leftover metadata or unwanted lines. The speaker considers this promising but potentially less scalable than the preceding rewriting approaches.

Synthetic labels can select better web data

Synthetic data can also improve a corpus without changing its text. FineWeb-Edu transfers a large model’s judgments into a classifier, then uses that classifier to select pages across a much larger collection: 1. Ask Llama 3 to score educational value from zero to five. Textbook content useful in school receives higher scores; advertisements and promotional material receive lower scores. 2. Train a smaller, BERT-like classifier on those synthetic annotations. 3. Apply it to 15 trillion FineWeb tokens and retain pages scoring above three, yielding 1.5 trillion tokens. FineWeb-Edu reportedly beat the other public web datasets in the aggregate benchmark comparison presented, though the account did not specify benchmark names, scores or training conditions.

Different classifiers encode different selection criteria: - DCLM targets a different signal from educational value. Its classifier uses OpenHermes instruction-training data and Explain Like I Am Five subreddit content. The resulting corpus is described as information-dense. - Nemotron combines several quality signals. An ensemble includes DCLM and classifiers similar to FineWeb-Edu’s, combining their scores to retain pages. It reportedly produced better results than the speaker’s team’s datasets, without a numerical comparison.

Post-training: vary the task, persona and teacher

Instruction training faces the same diversity problem as synthetic textbooks. Microsoft’s [AgentInstruct](https://www.microsoft.com/en-us/research/blog/orca-agentinstruct-agentic-flows-can-be-effective-synthetic-data-generators/) targets skills including coding, brain teasers and open-domain question answering. It starts with source code or raw text, rewrites those seeds into material suitable for instruction generation, then generates the instruction examples. Mistral 7B fine-tuned on this data reportedly outperformed the original Mistral Instruct model; the exact baseline version and scores were not specified.

Tülu3 varies the imagined user. Allen AI’s supervised fine-tuning mixture uses PersonaHub personas to diversify tasks. A prompt might request a coding problem for a machine-learning researcher interested in neural networks. Changing the persona changes the task context; reward models, which score generated examples, can then filter the results.

A broad-task dataset rendered as “SmallTalk” in the transcript also reportedly improved Mistral 7B over the original Mistral Instruct model, particularly in mathematics and instruction following measured with IFEval. The companion supplies a dataset link labeled [Smol Talk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk); the spoken name’s spelling is uncertain.

Multilingual Data Arbitrage pools complementary teachers. Cohere’s problem was that no single model excelled at every target language. A router sends prompts to multiple teacher models, and a reward model evaluates their completions and retains the best response. This makes teacher selection part of data generation: a synthetic corpus need not inherit one model’s strengths and gaps.

Small models become a practical deployment choice

The second half of the talk turns to small-model progress in 2024. Llama 3.2 1B approached Llama 2 13B in human evaluation on LMSYS Arena—explicitly one data point, not universal equivalence. A separate comparison from the Qwen 2.5 blog placed models with three to four billion parameters above 65 on MMLU and described a 3B model outperforming an earlier 33B model. The older model’s identity is unclear in the transcript.

Pocket Pal makes the deployment consequence tangible. In an iPhone demonstration, the 3.8-billion-parameter Phi 3.5 runs locally after being selected from Hugging Face. Asked for a NeurIPS joke, it produces one that underwhelms the speaker—but it runs on the phone. Latency is judged acceptable, without a device specification or measured response time.

Longer training trades an upfront cost for cheaper recurring inference. Larger models raise serving cost, latency and infrastructure demands; efficient smaller models open up consumer-hardware deployment. The talk illustrates longer training with an increase from one trillion tokens for an unspecified earlier Llama model to 15 trillion for Llama 3 8B. The economic argument is to spend more during training while keeping the model affordable to run repeatedly.

Architecture and data mixtures both matter

Several architecture studies explore how to use a small parameter budget: - MobileLLM favors depth over width. Meta’s study of models below one billion parameters found benefits from more layers, especially at a few hundred million parameters. It also reported benefits from tied embeddings and GQA, whose mechanics the talk does not explain. - Apple Intelligence uses pruning followed by distillation. Its setup includes a server model and a three-billion-parameter on-device model. The reported comparison favors that training approach over training from scratch, alongside specialization for summarization and rewriting. - NVIDIA explores a hybrid architecture. An unnamed paper combines state-space models and transformers in a 1B model described as performant with comparatively few training tokens. No token count or benchmark results accompany the claim.

The small-model family introduced as “SmolM2” in the transcript emphasizes data-mixture experiments and sustained training. The team tested which pretraining datasets helped and how to combine them. It also claimed an advantage for its 1.7B model over Llama 1B and Qwen 2.5, but without the Qwen model size, benchmark names or scores needed to interpret that comparison. The family’s spoken spelling is uncertain.

Training increased from one trillion tokens for the predecessor to 11 trillion, with performance reportedly continuing to improve rather than plateauing. The particular model size and evaluation metrics behind that observation were not specified. The release also included pretraining code, fine-tuning code and datasets, and evaluation resources, making the training work available for inspection and reuse.

Small-model development also extends to vision-language models. SmolVLM, at 2B parameters, is highlighted for low RAM requirements and good performance. Moondream 0.5B is described as sacrificing relatively little performance against Moondream 2B. These are qualitative comparisons, without numerical memory measurements or evaluation conditions.

Local applications and the return to specialization

Local inference keeps queries and data on the device, avoiding the need to send them to a third party. MLX, MLC, llama.cpp and Transformers.js are named as options for running models locally. A browser-chat example demonstrates the boundary: load the model, disconnect from the internet and continue chatting.

For text extraction, two approaches serve different needs: - Fine-tune for the task. NuMind fine-tuned SmolLM on extraction datasets and reportedly approached much larger models, although no benchmark, scores or comparison-model names were given. - Constrain the output format. Structured generation makes a model follow a defined JSON schema without fine-tuning. A browser example converts a free-text GitHub complaint into an issue containing priority, type, title and an estimated time to fix. It assigns high priority and the type bug; the demonstration establishes the output format, not the accuracy of the repair-time estimate.

Looking ahead from 2024, the speaker expects domain-specific synthetic data to grow in importance, particularly mathematics data intended to improve reasoning, with Qwen 2.5 Math as an example. The corresponding deployment argument is conditional: when a fine-tuned small model already meets a task’s needs, a modest quality gain may not justify a much larger model’s cost. The same qualitative tradeoff is proposed for vision and audio.

The closing prediction is a return to specialization. Task-specific BERT fine-tuning gave way to prompting larger, more general models; recurring inference costs could push applications back toward fine-tuning smaller ones. Alongside that shift, the speaker anticipates more on-device frameworks and applications in 2025, pointing to Pocket Pal and Ollama. The predicted cycle puts model adaptation back at the center of the cost-performance decision.