Better training data starts with what the model still needs to learn

Between storage and the data loader

A corpus sitting in storage is not yet a training plan. Someone must decide which examples to retain, how often to sample them, what synthetic data to generate, and how to sequence and batch the result. In the August 2025 Latent Space conversation, Ari Morcos describes Datology as an effort to automate that interval between stored data and a model’s data loader. At trillions of tokens or billions of images, inspecting every example by hand cannot scale.

The intended payoff is to reach a target performance with less training, improve performance within the same compute budget, or train a smaller model that preserves the capabilities that matter. The hard question is what makes an example useful given the other data and what the model has already learned.

When architectural insight stopped paying off

Morcos entered machine learning through neuroscience: teaching mice to count, recording thousands of neurons, and analyzing the resulting datasets. That background shaped his view of deep learning as an empirical science. Researchers specify a system’s design, but experiments reveal the unexpected properties that emerge during training.

His ideal paper explained why a representation worked, then used that explanation to improve it. The second step proved much harder: a property correlated with good models might not improve them when directly optimized. He estimates that roughly 30 of his papers delivered the explanatory half, while only three or four also produced an effective intervention.

Around 2020, several projects challenged his emphasis on inductive biases—assumptions built into an architecture or training objective. ConvIT initialized a vision transformer to behave like a convolutional network, giving it a starting preference that training could undo. Morcos describes assigning attention heads to positions in a convolutional kernel, then allowing learning to relax the imposed weight tying. This made convolution a soft bias rather than a fixed architectural constraint.

In the image self-supervised experiments he describes, the initialization helped especially below roughly 500,000 examples. Its advantage diminished with more data and became mildly harmful around or above a million examples—approximate thresholds for those experiments. The work still found uses in limited-data science, including volcano prediction with roughly 1,500 data points. Morcos drew a broader conclusion: enough data could overwhelm the architectural assumptions he had spent years designing.

Abundant data changes the bottleneck

Fixed-dataset benchmarks helped direct research attention toward models. ImageNet evaluation and Kaggle competitions commonly asked researchers to improve performance given an existing dataset. In the supervised-learning regime described here, labels required human labor, data was scarce, and human inspection supplied a partial quality floor. Data cleaning remained lower-prestige work despite its influence on model quality.

Self-supervised learning derives prediction targets from the data itself: recover missing content or predict the next token, rather than obtain a separate human label. This opened access to vastly more training material. The comparison between roughly a million ImageNet examples and trillions of text tokens illustrates that expansion, though the units are not equivalent. One strong position in the discussion is that learning from unlabeled data was more indispensable to recent capability gains than the transformer itself.

The bottleneck becomes how to spend finite compute on abundant, uneven material. Models can underfit that data: training ends while useful patterns remain unlearned. Redundant examples and low-information content consume opportunities to learn something more valuable. The proposed objective is to choose the next example that teaches the model the most, conditioned on everything it has already seen. Solving that selection problem remains a research frontier.

Ten thousand good Hamlet summaries

DataComp LM (DCLM), an open effort to curate Common Crawl-style datasets, illustrates why expert intuition may be insufficient. In the study recounted during the conversation, researchers who had spent roughly two years developing filtering criteria could not predict above chance whether the resulting classifiers would keep or reject individual examples. The account gives no sample size or evaluation protocol, but challenges the assumption that human ratings must remain in the curation loop.

Consider the discussion’s 10,000 slightly different Hamlet summaries. Each could be accurate, clear and well written. Independent quality ratings could approve every one while missing their collective redundancy. Training value depends on relationships across the dataset, beyond what a reviewer can judge from a single example.

Eliminating repetition is not the answer. One view of a golden retriever cannot teach recognition across situations. The dogs-versus-elephants analogy makes the proposed distinction: the greater visual variation among dogs calls for more examples. A useful filter would discover concepts without labels, estimate their complexity, and preserve more redundancy for concepts that need it.

Where should the boundaries fall—dogs, elephants, or mammals? Concept granularity is a hyperparameter, a setting tuned through experiments. Datology’s challenge is both to find effective curation methods and to make them adapt automatically to unfamiliar datasets. That broader adaptation remains a requirement to pursue, rather than an established universal capability.

Source reputation does not establish training value

Better crawling is described as a major investment at large labs, but a source’s reputation offers only a weak shortcut to training value. For code, a participant cites StarCoder and other research for the finding that GitHub repository stars poorly predict improvements in coding capability, explicitly distinguishing those experiments from their own work. Simple heuristics such as length sometimes did better, but the account identifies no strongly predictive signal or precise effect size.

Books introduce content and acquisition questions too. Books3 is described as containing substantial material that is not safe for work; the category “books” does not settle what belongs in a training corpus. A participant interprets an unnamed court ruling as distinguishing purchased books from pirated copies, without establishing a general legal rule. A separate recollection describes increasingly difficult dataset approvals and publishing constraints at Meta.

An interviewer speculates that legal restrictions hurt Llama 4. The respondent allows the possibility but puts more weight on the difficulty of continuing to scale naively. Neither explanation is established as the cause; the exchange leads into the economics of scaling itself.

Bending scaling means sustaining information gain

A predictable scaling curve can still be expensive to follow: successive tenfold increases in data deliver diminishing marginal performance gains. Beyond Neural Scaling Laws, described as foundational to Datology, explores whether better data selection can change that relationship.

The mechanism links power-law scaling to a power-law decline in marginal information gain, the new information learned from each additional example. Keeping that gain approximately constant would accelerate learning instead of yielding progressively less per example. The proof discussed applies to small-scale perceptrons; realizing its potential in practical training remains Datology’s goal.

Datology reports progress along three axes: - Tokens to target: matching a DCLM baseline trained to convergence with fewer than 10% of the tokens, described as about 12× faster. This measures token efficiency, not wall-clock time. - Accuracy at fixed compute: improving average accuracy by four to five absolute percentage points over leading open datasets, depending on the dataset and evaluation. - Parameters at target performance: training models with fewer than half the parameters that outperform larger models trained on uncurated or alternatively curated data.

These company-reported comparisons use average accuracy across 15 benchmark tasks appropriate to model size. The full suite and exact model sizes are unspecified. To limit benchmark overfitting—particularly when developing synthetic data—Datology says it reserves test sets for occasional inspection and additional evaluations for later assessment. Strong visible scores alone can still conceal brittle behavior.

Why make data the business?

The founding account describes choosing a separate company over an opportunity to build a centralized data team at Meta. Its rationale is organizational: data teams compete for attention while their improvements accrue to the wider model organization. A dedicated company can make dataset improvement its primary objective.

That choice creates a tension with open science. Customers ask Datology to beat the best open dataset, while publishing a fully reproducible recipe could give them an alternative to buying. Datology describes scientific know-how and engineering infrastructure as its current competitive advantages, with brand recognition a longer-term aspiration. Its publication policy is to explain mechanisms and intuition without disclosing everything needed for reproduction.

The product also serves combinations of open data, purchased data and proprietary business records. A single public corpus would not cover those customer-specific mixtures. The commercial outcome remains a better, more cost-effective model, even when the work happens entirely on its data.

Synthetic data: where does the information come from?

Curation extends beyond filtering. Changing sampling weights makes some examples appear more often without necessarily deleting others. Source mixing, batching and curricula change how examples reach the model; synthetic data changes the examples themselves.

The discussion distinguishes synthetic-data approaches by their information source: - Generation from model knowledge: the generator supplies much of the knowledge in new examples, transferring it to another model through a form of distillation. - Rephrasing existing material: the source supplies the intended information; a model rewrites or reformats it to make it easier to learn.

Rephrasing can clean material, make information more accessible, or express it in formats resembling downstream tasks. Datology describes this as moving post-training-like material into pretraining. The argument for moving work earlier depends partly on making pretraining affordable enough to revisit, rather than treating it as an exceptionally expensive, rare event.

The proposed model-collapse mechanism is recursive loss of rare cases. A generator overrepresents common patterns and underrepresents the distribution’s tails. Successive models trained on those outputs can narrow the distribution further. The respondent associates this risk especially with generation from a model’s own knowledge, while arguing that filtering between generations can introduce information through selection. Rephrasing is not presented as proof that all such risks disappear.

Rephrasing also offers a route for a student to exceed its generator: the source can provide information beyond the rewriter’s own knowledge. Datology calls its approach Beyond Web and previews experiments in which rewritten data reportedly beats training on the raw material and produces students stronger than the generating models. No model identities, effect sizes or compute comparisons accompany the preview. The respondent is more skeptical of surpassing a teacher through newly generated content alone, except where rejection sampling—selecting acceptable outputs—adds supervision.

Diversity remains essential. Textbooks can be valuable while representing too narrow a distribution for a complete training corpus. Rewriting everything into one polished style risks narrowing coverage further, so Datology reports using many styles and formats. The respondent favors repeating high-quality examples over spending the same token budget on new material of average or unknown quality. Whether paraphrasing beats another pass over those originals remains unresolved here: a Kimi experiment is mentioned, but its results are not given.

Curricula matter when training time is scarce

Curriculum learning deliberately orders training material. The founding recollection includes strong skepticism about it around mid-2023, but a dependency-graph argument explains its appeal. Let concepts be nodes and an edge from A to B mean learning A helps with B. Ordering offers no advantage if there are no dependencies, or if every concept helps every other equally. Unequal dependencies make ordering worth investigating: addition and subtraction, for example, help with multiplication and division.

The proposed benefit is primarily efficiency. When repeated supervised training already extracts nearly everything useful from a dataset, learning faster may leave the final score unchanged. When training ends with abundant data still underfit, learning useful prerequisites sooner can improve what fits within the compute budget.

This treats pretraining, mid-training and post-training as one learning process: mid-training is a later curriculum phase, with post-training continuing it. Datology says it was expanding into post-training curation in response to customer requests. Optimizing these stages together could let downstream results guide earlier data choices, although separate teams at large labs make that feedback difficult.

The proposed targets include stronger improvement from reinforcement learning and from additional computation while answering, alongside greater resistance to jailbreaking. The respondent hypothesizes that alignment learned during pretraining would be harder to undo than behavior added later. A reported contrast between Qwen and Llama motivates this direction: synthetic reasoning traces are suggested as a reason Qwen improves more readily with reinforcement learning. Without specified versions or controlled comparisons, that remains a hypothesis.

The economic target is often a smaller model

Datology’s typical customers train from scratch or continue pretraining open models on proprietary domain data, usually with at least a few tens of billions of tokens. Demand includes large enterprises and sovereign-AI efforts seeking capabilities suited to particular languages and cultures. The latter are described more precisely as government grants to private companies or public-private partnerships, rather than necessarily government-owned models.

Customers usually prioritize more capability within an existing budget. Faster training also shortens the experimental loop: an overnight run instead of a ten-day run would let the same team test more ideas. For mature deployments, however, smaller models may matter most because inference dominates ownership cost.

A hypothetical business case assumes $50 million in annual inference spending on a model twice as large as necessary. Under that simplified accounting, $25 million could be avoided, making a $2–3 million training investment attractive. The crucial conditions are that the smaller model matches the workload’s performance and can be built reliably without great difficulty; these figures are not demonstrated savings.

The target is a model that handles a few tasks exceptionally well at low serving cost. The respondent credits Mosaic, SageMaker and Together with reducing the training-infrastructure barrier, while arguing that data preparation remains difficult. Datology aims to make suitable training data accessible without requiring customers to learn through repeated failed runs.

Fewer weights, cheaper reasoning, less memorization

Pruning an existing model offers another route to smaller systems. The respondent recalls work with Jonathan Frankle on a reusable lottery-ticket initialization: starting weights intended to reliably produce successful subnetworks. Data dependence frustrated the idea. Even small changes in the training distribution substantially changed which initializations succeeded.

Parameter sparsity does not guarantee faster inference. Unstructured pruning removes individual weights, but sparse matrix operations introduce overhead and can use GPUs inefficiently. Structured pruning removes whole units, making speedups easier to realize, though the respondent says it preserves performance less effectively. Training smaller models with better data remains complementary to these inference optimizations.

Asked for the minimum size needed for a capability, the respondent gives no numerical lower bound, but predicts that most commonly used models could have single-digit billions of parameters or fewer within roughly three years of the conversation. Test-time compute—spending additional computation on answering a problem—strengthens the incentive. In the simplified accounting, problem cost ≈ cost per thinking step × number of thinking steps; cheaper steps make extended reasoning more economical.

An interviewer introduces Andrej Karpathy’s cognitive core concept: a model that retrieves knowledge through tools instead of storing as much internally. The respondent endorses its feasibility and connects it to randomized-label experiments. Random labels remove genuine input–label associations, forcing memorization. In the research recounted, deleting units damaged memorizing models much more than models that learned generalizing solutions. The interpretation is that memorization consumes substantial capacity, motivating investigation into how much knowledge a useful model needs to store.

Curation can build on curation

Returning to a customer project, Datology describes combining DCLM, Nemotron and FineWeb into roughly 25 trillion source tokens and producing a seven-trillion-token dataset. The resulting model reportedly surpassed Gemma before consuming one trillion training tokens. Model versions, sizes, tasks and scores are not specified, limiting the quantitative comparison.

The implementation lesson is that filtering and synthetic data required substantial integration work: their gains did not automatically combine. Already curated open datasets also benefited from further curation without requiring a fresh crawl. Applying Datology’s process separately to DCLM and FineWeb reportedly improved both while preserving their original performance gap. Upstream data quality still mattered after further optimization.

The practical advice is to make dataset decisions early. Datology describes customers planning multimillion-dollar runs and selecting architectures, then seeking a good dataset only two weeks before training. Domain-expert material is difficult to obtain, but customers may also be unsure which data they need. The recruiting anecdote captures a useful habit: inspecting examples closely enough to have favorite and least-favorite entries in the C4 dataset.

The larger research goal is valuing data relative to a downstream task. Enterprise customers may require Datology to bring its system to their proprietary data, with limited access for inspection. The system must adapt to unfamiliar distributions and adjust filtering, synthetic generation and other stages to the requested capabilities. Some techniques reportedly support this already; extending it across the pipeline remains a goal. No curation is universally optimal independent of what the model must do.

Investing in the people and data behind compute

The closing exchange turns to Meta’s FAIR research organization. The respondent recalls Yann LeCun preferring hands-on science, with Joelle Pineau and Antoine Bordes handling organizational leadership and Pineau leading for most of that period. Praise for Pineau’s advocacy for science accompanies concern about FAIR after her departure, tempered by hope that fears for its future prove wrong.

The respondent also recalls FAIR spending roughly one and a half to two years within Reality Labs, arguing that some foundational AI investment therefore came through the metaverse effort. The timing is explicitly a recollection. Optimism that this investment will eventually pay off rests partly on Reality Labs foundations contributing to Ray-Ban glasses; it remains a forecast.

The final argument treats both data and talent as compute multipliers: large compute budgets can justify substantial investments in using that compute well. Datology distinguishes optimizing already collected data from annotation providers’ emphasis on collection. The closing hope is that major AI investments give this additional layer of curation more attention.