TinyStories: Making Language-Model Reasoning Small Enough to Study

A small model that remembers why lunch went wrong

Jane tastes Tom’s soup and makes a face. A useful continuation must explain her reaction and carry the problem toward a resolution. In an example Nathan Labenz reads from the TinyStories paper, a 28-million-parameter model makes the soup bitter, has Tom apologize and offer bread and cheese, and ends with Jane thanking him. This single completion demonstrates something modest but concrete: a problem and its resolution remain connected across sentences.

TinyStories is a synthetic story dataset for inexpensive language-model research. Created by Ronen Eldan and Yuanzhi Li of Microsoft Research, it addresses a practical bottleneck: small models were cheap to experiment with but often failed to produce coherent text, while experiments resembling large-model behavior demanded substantial compute. In the Cognitive Revolution conversation, the researchers describe natural language simple enough for small models, yet rich enough to combine grammar, facts and reasoning.

The July 2023 introduction connects this work to phi-1, a 1.3-billion-parameter model reported to score above 50% on HumanEval, a code benchmark, after about 800 A100 GPU-hours of training. The companion article extends the theme to data cleaning through RefinedWeb, user feedback and synthetic generation. Together they frame data quality as another optimization dimension alongside model size, token count and compute. The benchmark figures are reported without evaluation settings.

Restrict complexity, then engineer diversity

Small vision datasets such as CIFAR-10 and MNIST offered a model for fast experimentation. The researchers wanted that accessibility while retaining natural language’s intertwined demands. Children’s stories supplied the constraint: each example should be understandable to a small child, while the collection should cover a broad range of that child’s knowledge.

Generating enough stories was easier than generating enough different stories. In one reported experiment, asking GPT-4 for a short story 1,000 times at temperature 1 produced stories about children afraid of playground slides roughly one-fifth of the time. Li attributes the repetition to a tendency toward high-probability continuations when little content is specified. Content constraints could push generation beyond those familiar plots.

Their generation procedure supplied variation explicitly: 1. Assemble roughly 2,000 simple words thought to be understood by three-year-olds. 2. Select a random noun, verb and adjective for each prompt, and ask the generator to combine them creatively. 3. Vary additional requirements, such as dialogue, a plot twist or a bad ending. The researchers estimate roughly a million generation calls and about 1.5 million stories, mostly written by GPT-4, with some by GPT-3.5. The design aims for vocabulary coverage without exhausting possible combinations, leaving room to test whether models can combine familiar words in unfamiliar ways.

Eldan distinguishes constraint satisfaction from coherent integration. GPT-3.5 sometimes included every required word but changed topics awkwardly to do so; GPT-4 more often combined the words into a fluent story. Supplying example stories—a few-shot prompt—could clarify the task, but could also bias generation toward those examples’ plots.

Instruction following made this generation process practical. A model trained to continue text can plausibly answer a story request with another question instead of obeying it. Eldan credits reinforcement learning from human feedback, or RLHF, with helping GPT-4 follow the word and vocabulary constraints. Li likewise distinguishes completing a supplied story from following instructions to create one. The earlier GPT-4 version available to the researchers already had some RLHF; it was not a clean test of an entirely unaligned base model.

At then-current retail pricing, generating one million stories entirely with GPT-4 was estimated to cost about $10,000, assuming roughly 300 tokens and one cent per story. That was a hypothetical generation cost, not the project’s reported bill.

Code may prepare reusable attention operations

The conversation next considers curriculum learning: arranging training data so that earlier practice prepares a model for later skills. Asked whether logic or code should precede ordinary language, Li proposes a concrete mechanism. A variable reference requires finding an earlier definition; a function call requires retrieving relevant context. Code could exercise attention operations that match repeated tokens and transfer information from an earlier occurrence to a later one.

Eldan treats reasoning improvements from code training as suggestive, with an uncertain cause. The benefit might come from calibrating precise token comparisons and relative-position lookups. Useful preparation need not mean that pretraining already taught a sophisticated reasoning procedure.

Lego, an earlier synthetic reasoning task, provides an analogy. It asks a model to resolve variables through chains of assignments. A pretrained BERT transformer reportedly learned the task faster; further investigation attributed the advantage to simple attention heads acquired during pretraining. Initializing a model with those heads also accelerated learning. Reusable components offered an explanation for transfer without requiring a general reasoning faculty.

Reasoning begins with respecting an exclusion

Here, reasoning means applying elementary logic to stay consistent with earlier context. The scope is narrower than planning a mathematical proof, which also requires choosing a method and deciding the next step. A pet request makes the distinction tangible: a child likes cats and dogs, asks for a dog, and is refused. What should she request instead?

The completion separates three abilities: - Grammar constrains the answer’s form: the request needs a noun. - Context supplies the candidates: cats and dogs are the relevant pets. - Elimination selects the remaining option: once a dog is ruled out, a cat remains. This is a loose hierarchy of capabilities, not a strict partition of language skills.

On the discussed prompt, the researchers report that their smaller models and GPT-2 XL, with 1.5 billion parameters, favor dog. They interpret this as following the recent, repeated noun rather than applying exclusion. Later, the interviewer contrasts that failure with TinyStories models that can choose cat. These examples concern a restricted story task, not general superiority over GPT-2 XL.

The researchers report different scales for two capabilities: - Story coherence: a roughly one-million-parameter model struggled, while almost all stories from a ten-million-parameter model were coherent. - Reasoning prompts: models with one to five million parameters failed all their prompts, while a model around thirty million succeeded on almost all. The discussion gives neither the reasoning-prompt count nor the scoring procedure. These are reported observations within TinyStories’ restricted setting.

The interviewer argues for treating competence as a collection of smaller skills: an adversarial failure does not erase repeated successes. A respondent also resists defining emergence as a sudden jump after a tiny increase in size. Here, the term describes a qualitative capability becoming apparent at one scale when little trace was visible at smaller scales. No particular circuit transition is established as its cause.

Rare consistency decisions can change the whole story

Why might a tiny model maintain consistency better than a larger model trained on broad web text? The proposed explanation is competition for limited capacity. Broad corpora reward learning many facts, associations and stylistic details. TinyStories reduces those demands, making consistency a more prominent remaining route to lower training loss—the error the model is trained to reduce. The claim that broad-corpus models systematically learn frequent facts before reasoning remains an untested conjecture.

Consider the conversation’s candy-or-pizza example. Bob dislikes sweet food, so pizza fits his preference. But the association between sweet food and candy pulls toward the wrong completion unless the model resolves the negation. One inconsistent word can make an otherwise fluent passage seem uncomprehending. Under this account, a small improvement in aggregate prediction loss can produce a large change in perceived competence because it fixes a few consequential decisions.

The discussion extends that intuition to mathematics: the logical connections between statements can determine a proof’s coherence even when much of its text consists of formulas and intermediate results. These difficult connections may be learned late, making greater model size or more training data appear to unlock a new ability.

A suggested exercise makes the hypothesis inspectable: read an ordinary article word by word and ask what each prediction requires. Would grammar suffice? Would nearby nouns and verbs be enough? Is a fact needed, or must the prediction respect a distant constraint? This exposes different learning opportunities without assigning measured proportions of language to each skill.

A curriculum must teach skills to work together

The interviewer proposes rebalancing synthetic data toward progressively harder skills. The researchers suggest varying mathematical exercises, comparisons of familiar historical events and explanations of simple physical rules without continually adding facts. They distinguish breadth—the facts and vocabulary to capture—from depth—the layers of logic to learn. A tradeoff between them seems plausible, but is not established.

Two possible arrangements follow: - Teach symbolic reasoning first, then add knowledge: begin with abstract alternatives and exclusions before introducing factual associations. - Mix reasoning and knowledge examples: adjust their ratio to balance skill practice with necessary facts. Both are proposals. Some knowledge remains essential: a model must know that candy is sweet to reject it on that basis.

The central objection is that separately learned skills may not compose unless training rewards their combination. If every sample requires either factual recall or symbolic reasoning, successful autocomplete gives the model no reason to integrate them. A few combined examples do not guarantee that it will discover the intended connection. The speakers consider curriculum learning promising but offer no demonstrated recipe for this integration.

A related concern is catastrophic forgetting: losing earlier learning during subsequent training. The interviewer recounts MosaicML staff describing how they mixed general pretraining data into client-data training to avoid it. That secondhand account favors maintaining a mixture over strictly separate phases, but preserving skills and teaching them to work together remain distinct problems.

The child-development analogy also has limits. A child repeating a request for ice cream seeks an outcome; grammatical polish and originality may contribute little to getting it. Next-word training rewards different behavior. The researchers suggest that grammar develops earlier because local regularities are easier to learn and affect more predictions than maintaining an entity across several sentences.

A GPT-2 XL completion illustrates the gap. Sarah and Ben build a snowman, and a dog destroys it. The continuation wanders through a red button, fur and saving the world. Individual sentences remain locally plausible while the original event loses control of the narrative. The proposed explanation is that nearby context and a few salient entities can predict many words without tracking the whole story.

Depth buys rounds of information transfer

Model depth offers a possible way to connect distant constraints. Each attention layer gives tokens another round of information transfer. Following a request for a bad ending, for example, may require combining the instruction with the wider narrative across several rounds. The researchers present this as an intuition requiring further study.

The pet example illustrates two dependent transfers: first combine negation with dog to represent that a dog is unavailable, then combine that exclusion with the original alternatives to select cat. The interviewer calls these two attention hops, and the respondent agrees. Factual recall suggests a different computation: combine a country with a relation such as capital, then retrieve the associated fact. Greater hidden dimension, or width, might help represent more entities and facts.

Layer count is not a strict limit on reasoning steps. A wider layer could enumerate combinations and collapse multiple steps, with growth from N to N² offered as an illustration of the cost. The researchers also report follow-up work on Lego, outside that paper, in which models perform multiple reasoning steps within one layer. Depth may make such computations more efficient without acting as a literal instruction counter.

Readable attention patterns are not yet verified circuits

The researchers observe a striking division of labor in a model trained with one attention block: - Distance heads attend by token separation: different heads operate over different ranges. - Semantic heads attend to relevant entities: one focuses on the main characters, Tom and Lucy; others focus on objects or places such as banana and park. They had not checked how this separation behaves across multiple interacting attention layers.

The distance patterns resemble ALiBi, a scheme that supplies distance-based attention at multiple scales. Their model used absolute positional encoding and had to learn these patterns. A respondent sees this as a reason to test building in distance-based attention, though no controlled comparison with ALiBi is reported.

The neuron inspection examines individual hidden coordinates in the model’s feed-forward layers: 1. Run stories through the model. 2. Collect the tokens and surrounding snippets that most strongly activate each coordinate. 3. Look for a shared role among those examples. The discussion credits this approach to the 2015 paper “Visualizing and Understanding Neural Models in NLP.” In one small model, a coordinate appears to activate when the main character is introduced.

That observation suggests a copying mechanism: detect the protagonist’s introduction, then use an attention head to retrieve the name when another mention is needed. But an interpretable activation does not establish the causal connection. The researchers had not checked the proposed connection sufficiently. Their inability to find clear meanings in inspected GPT-2 XL neurons likewise does not establish that the larger model lacks interpretable components.

Two hypotheses could explain the small models’ relative clarity: - Simpler tasks make functions easier to recognize. Maintaining character identity is more tractable to inspect than complex code generation. - Limited capacity may favor compact, separate functions. A larger model can spread a computation across several heads or layers; a small model may need a more economical organization. Under the second hypothesis, interpretability emerges as a side effect of efficiency. The training loss itself does not reward human-readable organization.

Transfer remains open; usefulness may outpace understanding

The closing discussion returns to the project’s largest unresolved condition: an architecture that wins on TinyStories need not win on broad language corpora. Do depth–width relationships remain stable across dataset mixtures, architectures and model sizes? The researchers claim only a few indications of universality. Establishing transfer would make inexpensive experiments more valuable, reduce research energy costs and widen access to language-model research.

Their proposed next step was to expand the stories toward elementary-school knowledge, potentially supporting a model of a few hundred million parameters. Training it on a laptop GPU in a day or two was an aspiration, not an achieved benchmark. Even without transfer to larger models, TinyStories could support mathematical work on how gradient-based training produces synthesis and generalization beyond memorized text.

The final recommendations return to Lego’s attention mechanisms and “Transformer Feed-Forward Layers Are Key-Value Memories,” described as finding meaningful roles for some neurons. Yet one speaker doubts that partial interpretability will lead to comprehensive understanding or sufficient mechanistic control to ensure alignment. Gradient descent has no obligation to find a solution humans can explain.

An alternative is behavioral investigation through repeated interaction: probe capabilities, persistent tendencies and typical failures across conversations. A participant connects this approach to the Sparks of AGI paper and compares it to psychological study. Such investigation seeks useful regularities in behavior even when internal computations remain difficult to explain.

The closing analogy is horseback riding for the brain: a system could extend human cognitive reach without its users understanding each internal mechanism. Horse training illustrates the possibility of learning to guide behavior without explaining individual neurons; it does not establish that the same approach suffices for AI alignment. The speaker ends pessimistic about fully understanding neural networks, but optimistic about their usefulness and the prospect of aligning them efficiently.