Datasets 101: What a Model Actually Learns From

An online presence does not guarantee model knowledge

Ask a language model about someone with an online presence, and it may know nothing useful about them. Alessio uses this gap to challenge the idea that GPT has read the entire internet. For a specialized application, understanding the model’s training material helps identify what additional data could make fine-tuning useful.

The Datasets 101 conversation, published in July 2023, separates the amount of information that exists from the amount of usable training data available. The token crisis concerns whether that usable supply is running out. Research on data constraints and a reported view from Ilya Sutskever—that OpenAI was nowhere near exhausting the data it wanted—frame an unresolved question: what gets collected, retained, and counted?

The companion article describes GPT-3’s dataset as a little over 600 GB, drawn primarily from Wikipedia, books, WebText, and 2016–2019 Common Crawl. Its central point is that useful training data requires substantial filtering. More available bytes do not eliminate the need to decide which ones deserve expensive training time.

Text supplies its own prediction targets

A conventional machine-learning tutorial might split one dataset, such as Iris, into training, validation, and test sets. Language-model development often separates these roles: a broad corpus—the collected training material—supports pretraining, while separate benchmarks test capabilities. Prediction loss on the training-data distribution remains useful, but answers a different question from a task evaluation.

Self-supervision creates training targets from the text itself. In masked prediction, a word is hidden, the model predicts it, and the prediction is checked against the original word. This makes large corpora usable without manually labeling every example. Wikipedia can supply training material while separate evaluations, such as HellaSwag and BIG-bench, test what the model can do.

Tokens are representations, not word counts

Training budgets count tokens: units of text represented by integer identifiers. The approximation “one token is three-quarters of a word” is not a splitting rule. A tokenizer converts between readable text and these IDs; capitalization and whitespace can change the representation. An autoregressive language model predicts the next token ID from the preceding sequence.

Alessio’s example assigns different IDs to three closely related strings: - red, with a leading space: 2266. - Red, with a leading space and capital letter: 2297. - Red, without the leading space: 7738. The encoding is unspecified, so these IDs belong to the reported example. The distinction matters for logit bias, a control that changes the likelihood of particular token IDs. Suppressing one representation of egg in an egg-free cake request can leave other representations available.

Word boundaries are not guaranteed either: latent space reportedly becomes lat, ent, and space. In the GPT-3 tokenizer demonstration, 1, 12, and 123 each occupy one token, while 1234 becomes 12 and 34. That representation does not explicitly express decimal place value. A speaker proposes it as a contributor to arithmetic difficulties, without testing that explanation comparatively.

Equivalent content can also require different token counts across languages. The reported pizza example uses four tokens in English, seven in French, and 15 in Chinese, though the exact encoding and Chinese text are not established. These illustrate a disparity, not a reusable conversion rate. And a trillion-token corpus does not contain a trillion distinct words: the count includes repeated material.

Optimal for training—or for serving?

Model size and dataset size are coupled decisions. For a proposed 100-billion-parameter model, Alessio suggests counting the tokens available from existing sources before determining what additional data is needed. Compute-optimal training asks how to allocate model size and training data within a fixed budget of computation, constrained by GPU count, runtime, and cost.

The starting comparison is GPT-3’s reported 300 billion training tokens for 175 billion parameters, or about 1.7 tokens per parameter. The conversation presents Chinchilla’s prescription as approximately 20 tokens per parameter. Applied to the same model size, that gives 3.5 trillion tokens, motivating the interpretation that GPT-3 had too many parameters for its training data. This is a training-compute allocation argument, not an application-independent requirement.

The speakers then consider roughly 200 tokens per parameter, informally calling it “Llama optimal” while questioning whether it represents a new optimum. The consequential distinction is training efficiency versus inference efficiency. Inference means running the trained model; a smaller Llama model can receive more training than a training-compute optimum prescribes to improve latency, memory requirements, and serving cost.

Additional training expenditure can therefore make sense when it reduces the cost of repeatedly serving a production model. Benchmark capability still matters, but response time and resource use become part of “performance.” The appropriate optimization target depends on the application’s lifetime costs.

A model file is smaller than its training corpus

Parameter storage provides some exact arithmetic. At eight bits per parameter, one billion parameters occupy one decimal gigabyte. A 175-billion-parameter model therefore requires 175 GB at eight bits or 350 GB at 16 bits—for the weights alone. Compared with an 80 GB A100 GPU, the latter explains the need to distribute a large model across multiple devices with high-bandwidth connections.

This motivates a lossy compression analogy: thousands of gigabytes of training material become a smaller set of parameters, from which generation produces approximate reconstructions. It is an analogy, not a promise of database-like retrieval. Stable Diffusion supplies a visual counterpart: the speaker estimates its downloadable file at roughly 2–4 GB, despite its ability to generate subjects ranging from horses and beaches to computers and flying saucers.

Quantization reduces the precision used to represent weights, moving from 16 bits toward eight or four. Within the analogy, it compresses an already compressed representation. One speaker reports running a three-billion-parameter RedPajama model, occupying a few gigabytes, on a phone with 64 GB of storage. Its training corpus would not fit there; the example establishes a storage contrast, without speed or quality measurements.

The analogy briefly stretches into speculation: the conversation recalls George Hotz suggesting that a person’s knowledge or life experiences might fit into two gigabytes. The speaker doubts whether that much compression would preserve a useful copy, leaving digital twins and mind uploading as speculative possibilities.

For practical scale estimates, the episode offers rough landmarks: a sentence at around 10 tokens, the Fermi estimation Wikipedia article at around 1,000, the Taylor Swift article at around 10,000, and English Wikipedia at roughly a billion. MPT-30B, cited as a recent release at recording time, reportedly trained on approximately a trillion tokens.

An assistant needs several kinds of data

A foundation model changes the enterprise starting point: it arrives pretrained with capabilities that can be adapted through prompting and fine-tuning. Companies need not first collect a vast corpus and train from scratch. The suggestion that dozens or perhaps 100 code examples might suffice is an illustration of this reduced burden, not a measured data requirement for a specified task.

Drawing on Karpathy’s State of GPT presentation, the conversation describes four stages with different data requirements: 1. Pretraining: raw text supplies broad training material, at a representative scale of trillions of tokens. 2. Demonstrations: tens of thousands of prompt-response pairs show ideal assistant behavior. 3. Comparisons: hundreds of thousands of judgments between candidate outputs support reward modeling—learning which outputs are preferred. 4. Reinforcement learning: tens of thousands of additional prompts support the final training stage. These are representative scales for a ChatGPT-like system, not a disclosed inventory of ChatGPT’s data. Comparisons can be more numerous because choosing between answers is easier than writing an ideal answer.

Instruction tuning, which uses demonstrations to teach response behavior, belongs in the second stage. The episode cites roughly 15,000 examples collected by Databricks and tens of thousands of responses from Open Assistant. The LIMA paper enters as research questioning how much instruction data is necessary, without a discussion of its experiment sizes or results.

A web corpus is a sequence of selection decisions

Common Crawl is public infrastructure for collecting web data useful beyond search indexing, a motivation the conversation connects to founder Gil Elbaz. The nonprofit’s limited resources matter: it covers only a fraction of the web and samples disproportionately from the United States.

Coverage gaps arise before training begins. Crawler exclusions, oversized pages, duplicated spam, and limited JavaScript handling constrain collection. Information inside Facebook, Discord, and Slack can remain inaccessible. Models trained downstream inherit this selected view of the web and its omissions.

C4 is a subset of Common Crawl cleaned with heuristic rules during Google’s T5 work. One ingredient is the List of Dirty, Naughty, Obscene, and Otherwise Bad Words: described as 402 English terms and one emoji, originally created by Shutterstock to suppress unwanted search autocomplete suggestions. A rule built for one product became part of language-model data preparation. Such filtering is incomplete: the speakers report that some material from 4chan and Kiwi Farms remained.

Wikipedia contributes extensive human editing, but concentrated participation also shapes the corpus. Steven Pruitt illustrates that influence: the episode reports more than five million edits, touching a third of English Wikipedia articles. Contributing to a widely used source can consequently influence future training material.

WebText, used for GPT-2, applies a different quality signal: Reddit upvotes. Its reported construction selects approximately 45 million web links from submissions through December 2017 with at least three upvotes, excludes Wikipedia, and applies further cleaning. Knowing the selected URLs is insufficient to reproduce the result when those cleaning rules are unavailable. OpenWebText and OpenWebText 2 enter as open reproduction efforts.

Books, code, mixtures, and images

Books bring long-form material and rights questions together. The episode describes a corpus of 196,000 plain-text books included in The Pile, while questioning whether every work’s copyright status is clear. A model’s commercial availability does not by itself resolve questions about its training inputs.

The Stack makes filtering tangible. Its reported construction proceeds from a GitHub archive to permissively licensed code: 1. Start with 102 TB from 153 million repositories. 2. Remove unwanted file types, leaving approximately 69 TB. 3. Filter for permissive licenses, retaining approximately 6 TB. The license step discards roughly 90% of the remaining material because it has no license or falls outside the selected licenses. The practical appeal to developers follows: include a license file so downstream users can determine the terms.

The Pile combines 22 smaller datasets into an approximately 825 GB mixture, including PubMed, GitHub, the Free Law Project, Ubuntu IRC, Hacker News, and YouTube material. It also illustrates deliberate upsampling: repeating selected data to increase its exposure during training. Some repetition therefore reflects intentional weighting rather than accidental duplication.

For images, LAION grew from efforts to reproduce DALL-E openly: a 400-million-image collection drawn from Common Crawl in 2021, followed by LAION-5B and an aesthetic subset in 2022. Image curation adds decisions about sexual content, unsafe material, and copyrighted characters. In a recalled Adobe Firefly demonstration, difficulty generating Spider-Man was presented as a feature for commercial customers concerned about rights disputes.

The audio examples are LibriSpeech, Common Voice, VoxForge, Switchboard, and Fisher Corpus, named in connection with Whisper without specifying their training or evaluation roles. For broader dataset discovery, the episode recommends the Hugging Face Hub, with Kaggle as another starting point.

Useful data takes work—and permission

Model training attracts attention that data preparation often does not. Enterprises face the same imbalance when pressure to adopt AI outpaces investment in preparing company information. BloombergGPT illustrates the potential value of that work: Bloomberg’s licenses to decades of financial reporting support domain-specific applications.

Holding data does not automatically confer training rights. Notion supplies the example: the proposed opportunity is to enable customers to adapt models to their own company knowledge, rather than assume the platform can train on everything it stores. In the episode’s 2023 context, Reddit and Stack Overflow API restrictions signal interest in charging for accumulated content. Open instruction-dataset replication offers a counterweight, enabling others to build ChatGPT-like systems.

Information density matters alongside volume. A thousand pages of parking fines may add little new information when they repeat the same format. Microsoft’s Phi-1, described as a 1.3-billion-parameter code model trained on seven billion tokens, illustrates the quality argument. Its reported HumanEval and MBPP coding-benchmark performance was comparable to models roughly ten times larger; no scores or evaluation settings accompany that task-specific comparison.

Training rights and user protections are separate questions. The Stable Diffusion and GitHub Copilot lawsuits challenged whether creators’ work could be used under fair use; these were unresolved disputes in the conversation. A reported GitHub offer to defend Copilot users shows another response: a provider can absorb some legal risk without guaranteeing that copyrighted material never appears in output. The offer’s eligibility and coverage are not specified.

Service terms add another layer. The speakers describe OpenAI’s then-current terms as restricting training on its outputs, citing Alpaca and Vicuna as models built using those outputs. The original LLaMA’s noncommercial restriction likewise motivated interest in RedPajama as an open replication. These examples concern the terms discussed in 2023.

Privacy also requires data preparation. Hugging Face is described as providing a pipeline to remove personally identifiable information, along with a way to check whether one’s information appears in The Stack.

Duplicates change learning; contamination changes the score

Deduplication removes repeated material that consumes compute and encourages memorization without adding information. The episode cites a striking reported result: a sequence appearing ten times in training was generated, on average, 1,000 times more often than one appearing once. The experimental setup is unspecified, so the magnitude is not a universal rule; the underlying concern is that repeated exposure can disproportionately affect output.

Benchmark contamination occurs when test material has already appeared in training, making scores less informative about performance on new problems. A CodeForces example in the GPT-4 discussion reports ten successes out of ten pre-2021 problems and zero out of ten later problems. Without the problem-selection procedure and evaluation protocol, that contrast raises a question about the training cutoff but does not establish contamination as its cause.

Falcon 40B’s reported leaderboard position receives the same scrutiny. The speakers leave contamination unresolved: its dataset was unavailable to them, and lower scores reported by others had not been reproduced by the speaker. Publishing training data and evaluation methods helps readers assess both what the model encountered and how the test was run.

Whose language and communication?

Tokenization has a direct product consequence: when an API charges per generated token, languages requiring more tokens for equivalent content cost more to generate. The closing discussion returns to this English-centric imbalance and names CMRC, DuReader, and ChID as Chinese-language datasets for further exploration.

Coverage must also extend beyond writing. Where oral communication plays a larger role, voice datasets are crucial; a text-only model addresses less of how people actually communicate.

The closing appeal is to recognize dataset creators. Downstream models depend on the people who collect and clean their training material. The speakers make their own contribution explicit by welcoming use of the Latent Space Podcast as training data.