What Can a Language Model’s Numbers Tell Us?

When scale changes the research question

Ask a small language model who was US president in a particular year, and it may answer George Washington regardless of the date. Jack Morris uses this qualitative example to explain a practical problem for academic research: moving from roughly 100 million parameters to eight billion changes the questions researchers can investigate. Broad factual recall makes a model a different experimental subject from one that mainly associates familiar words.

In this Latent Space conversation, published in July 2025, Morris identifies his affiliation as Cornell Tech. His interest in machine learning for language began in 2017–2018, followed by research exploration around 2019, a Google AI residency in 2020–2021, and graduate school in 2021. Experiments with BERT helped shape his lasting interest in models with roughly 100 million to one billion parameters. He chose graduate school because professors and PhD students seemed to be producing much of the work he most wanted to pursue.

ChatGPT’s arrival in late 2022 changed that environment. Relatives suddenly wanted to understand his field, while research questions increasingly exceeded academic resources. In Morris’s account, companies’ investment in training infrastructure left universities catching up: buying GPUs, obtaining useful seven- or eight-billion-parameter models, and learning to run them. By the interview, he felt academic access to inference and fine-tuning at that scale had largely stabilized.

Morris distinguishes publication strategy from scientific judgment. Entering a field just after a paradigm shift can reduce incumbents’ advantage: o1’s September 2024 release prompted reproduction efforts, then questions about data, evaluations, and training schemes. That can be an effective academic strategy without necessarily being the best way to choose scientific problems.

Systems skills offer another route to independence: more experiments on a limited budget. Morris says students generally learn distributed training from peers and internet resources. The host recommends GPU Mode, fast.ai, and Microsoft’s DeepSpeed team; the discussion also highlights vLLM and SGLang for students entering inference research. Learning CUDA can be valuable, but GPU programming is only one part of understanding models, data, and efficient computation—not every newcomer’s best starting point.

Information you can actually use

The interviewer introduces Contextual Document Embeddings, praising its two-stage approach to efficiency, before turning to a broader question: what makes information accessible? Consider a readable paragraph about New York City and a reversibly encrypted version. In the example’s framing, they carry the same Shannon information, but an observer unable to decrypt the second file can do much more with the first.

V-information measures how much information an observer can extract under computational constraints. Introduced through the 2020 paper A Theory of Usable Information Under Computational Constraints, it makes the observer’s computational abilities part of the measurement. A representation may retain information while making it difficult to use.

This offers a possible explanation for pretraining’s value. Compared with randomly initialized weights, pretrained weights may make useful information easier to extract on a downstream task, allowing earlier computation to be reused. It is an interpretive lens, not an established explanation. A printed tensor may look unintelligible, yet its numbers reflect training data, the training algorithm, and model computation.

The ambition is an abstraction with the explanatory power that the bit supplied for communications: something precise enough to measure information and guide system design. The researcher argues that deep learning still lacks an equivalent foundation for explaining where information resides and how computation makes it accessible. V-information is a promising beginning.

Counting bytes does not measure knowledge

A model with 175 billion parameters stored at 16 bits each occupies 350 GB. Does that mean GPT-3 can store a couple of Wikipedias? The participants reject the inference. Wikipedia’s repeated patterns make it compressible, and its file size depends on the languages, histories, and representation included. Their Wikipedia estimates remain unresolved; dividing file sizes cannot establish model capacity.

Compression remains a useful perspective: a model can be smaller than its training corpus while generalizing with some acceptable loss. But storage size, recoverability, and usefulness ask different questions. A proposed experiment would make one of them concrete: train on Wikipedia until the whole corpus can be extracted perfectly, then use exact recovery to investigate storage capacity. No result from that experiment is reported.

Recovering text from an embedding

The same capacity intuition motivated embedding inversion: recovering source text from a numerical vector used for tasks such as search. Vectors representing short documents contain many bits, suggesting substantial recovery might be possible. Vector databases made the question practical: when an application sends a vector instead of customer text, what has it disclosed to the recipient—or to someone who breaches the database?

The project followed roughly a year and a half of exploration and rejected proposals. Once the advisor approved trying to recover as much text as possible, the researchers tested decoding algorithms, sampler adaptation, architecture changes, and training tweaks. Reinforcement learning was a retrospective suggestion; they did not try it.

One diagnostic sustained the effort: re-embed the reconstructed text and compare it with the target vector. Their best guesses still produced embeddings far from the originals. They interpreted that gap as evidence that the methods were leaving information unused. The eventual breakthrough became a defining graduate-school experience: substantial progress after prolonged frustration, rather than a complete solution.

The work became Text Embeddings Reveal Almost as Much as Text, published in 2023. The researcher reports roughly 90% exact text recovery for inputs around long-sentence length. The dataset, encoder, precise length, and percentage denominator are unspecified in the conversation, so the figure describes that reported setting rather than an expected rate for arbitrary embeddings.

Inversion also made transformations of embeddings easier to inspect. Researchers applying a gender-debiasing procedure reportedly reconstructed a sentence about a woman doctor working at Weill Cornell using person and they instead. That supplied text-based evidence of the transformation’s effect. Inversion could serve as both a privacy probe and a tool for understanding representations.

Unknown encoders and limits on recovery

The original system required knowing the encoder—the model that produces the embeddings—and making many queries to it. Unknown or independently fine-tuned encoders fell outside its assumptions. The follow-up, Harnessing the Universal Geometry of Embeddings, was a preprint awaiting publication at the time of the conversation. Its aim was to map another model’s representations into a space where an existing text-reconstruction system could operate.

Early clues included interchangeable representations among some models fine-tuned from the same base, and shared nearest neighbors across models with different training backgrounds. The Platonic Representation Hypothesis offers a broader interpretation: models trained on the same world may converge toward similar representations as data and model size increase. The cited work’s correlation-based evidence across vision and language models motivates alignment; it does not establish universal interchangeability.

Recovery also has a more fundamental limit. The inversion experiments use short sequences and relatively large embeddings, a favorable setting. If two inputs produce exactly the same vector, the vector alone cannot identify which input produced it. Two books differing by one word are suggested as a possible collision example, not a tested result. Because actual vectors use finite floating-point values, their capacity also depends on numerical precision; they are not arrays of mathematical real numbers with unlimited precision.

The discussion briefly shifts from vectors to weights. In a capacity experiment, a language model trains on independent random strings for many steps. As the training set grows, total memorization reaches an approximately stable plateau, termed model capacity. The mechanism remains unknown: separate computational paths might serve individual examples, or shared operations resembling loads and stores might reconstruct many independent strings.

Can a small model reason without knowing much?

A participant questions the practical value of studying memorization when practitioners tend to choose the largest available dataset. The Cognitive Core proposal gives the question a different purpose: build a small model with little stored world knowledge but enough reasoning and tool-use ability to obtain information externally. The attraction is fast inference on a device. The objection is that memorizing facts and learning useful ways to generalize are difficult to disentangle; strong reasoning with very little world knowledge remains an aspiration.

Search makes the dependency concrete: a model needs enough knowledge to formulate useful search terms. A participant recounts a conversation with Noam Brown as supporting the view that reasoning needs a sufficiently knowledgeable base model. One proposed alternative is to introduce search during pretraining, letting a model obtain missing information while learning to predict words correctly. Memory, online learning, or both might help reduce what must be stored in weights, but this remains a proposed design.

Aligning models without paired examples

The alignment method draws inspiration from CycleGAN, a 2017 computer-vision model that learns mappings between distributions without paired examples. Separate collections of horses and zebras can support a transformation without pictures pairing each horse with that same animal as a zebra. The researcher interprets this as learning semantic structure, including the animal, its surroundings, and camera position.

The embedding project applies that idea to representations from different models. The researcher tentatively identifies the example as GTR, a T5-based retrieval model, and GTE, described as BERT-based. Despite different architectures, data, and objectives, their embedding spaces reportedly align strikingly well under the custom CycleGAN-inspired method. This is a qualitative account, with no numerical alignment metric or evaluation distribution specified.

One participant finds shared representations profound; another finds them intuitive, while acknowledging the gap between intuition and evidence. The consequential details are which models align, how closely, and on which distributions. Establishing those conditions turns an attractive hypothesis into something researchers can use.

A proposed application goes beyond inversion: retain a roughly two-billion-parameter language backbone and load separately trained vision, audio, or text-to-speech components when needed. Alignment could make them swappable and stackable, with hoped-for benefits in efficiency and adding capabilities without losing existing ones. Those benefits are not demonstrated here. The related discussion of Gemma also distinguishes released weights from a release that includes training data and code.

How far small models can improve remains open. Could 100 million parameters eventually support the capabilities of the then-current o3-pro, through better data, training, or architecture? Or are those models near a plateau? A more practical definition of small is a model a graduate student can run for inference in reasonable time on one GPU—perhaps around seven billion parameters.

Measuring information retained in weights

The agenda separates two forms of dense information storage. Activations and embeddings raise questions about what a vector represents or reveals. Weights raise questions about what survives training in the parameters used to perform computation. The memorization-capacity study treats the latter as a basic measurement problem, even while its immediate practical use remains uncertain.

For transformers trained in 32-bit precision, the study’s reported estimate is approximately 3.6–3.9 bits of stored information per parameter. The comparison is between the numerical storage allocated to a weight and the information the experiments could get the model to retain through memorization. It measures capacity reached through the study’s optimization process, not a hard architectural ceiling.

Two kinds of improvement could raise that estimate: - Better storage: an architecture that retains information more effectively. - Better optimization or measurement: procedures that reveal capacity the current experiments miss. A Llama-style architecture is suggested as a possible improvement, without a reported test. The longer-term goal is to predict the model size and architectural properties needed for a given dataset and performance target. Both that predictive tool and the underlying storage mechanism remain unresolved.

A proposed capacity leaderboard exposes an objective mismatch: the best memorizer may not be the best generalizer. Maximizing stored information could reward compression without improving intelligence. The researcher agrees that repeatedly optimizing an established measurement can become less valuable than asking a new question.

Recovering a dataset’s effect from weight changes

Careful measurement also calls for careful presentation. A brief exchange emphasizes accurate references, professional figures, and correct captions as part of taking responsibility for published research.

Approximating Language Model Training Data from Weights asks a more applied question: what can model parameters reveal about the data that produced them? Directly reconstructing text from weights remains difficult. This proof of concept instead selects examples from an existing corpus to approximate the effect of the original fine-tuning dataset.

The method requires both a base checkpoint and its fine-tuned checkpoint. Its basic sequence is: 1. Compute the parameter difference between the checkpoints. 2. Score candidate examples from a web corpus by how their gradients align with that fine-tuning change. 3. Select high-scoring examples as a substitute training dataset. The gradient supplies a per-example signal in parameter space; alignment connects that signal to the observed weight change. The method selects data intended to reproduce a training effect rather than directly decoding the original corpus.

Training on the selected data reportedly produces a reasonably capable model, but falls short of the original model’s performance; no numerical comparison is given. A candidate pool of trillions of tokens is suggested for a hypothetical company deployment, not a completed experiment. The discussion also identifies synthetic checkpoints as linear interpolations in weight space, without explaining their precise role in selection.

New datasets and the next source of progress

The essay There Are No New Ideas in AI, Only New Datasets supplies the closing technical argument. Drawing on Thomas Kuhn’s account of paradigm shifts followed by smaller innovations, it challenges the prestige attached to new methods that improve performance on existing data. Its thesis is that major AI advances combine techniques with new data sources—and that the data receive too little credit.

The argument traces four shifts: - Deep neural networks: AlexNet trained on ImageNet’s millions of images. - Web-scale language pretraining: transformers, BERT, and early GPT models learning from vast text collections. - Instruction tuning: human preference pairs supplying examples of desirable and undesirable chatbot responses. - Reasoning models: calculators, symbolic verification, coding environments, and unit tests supplying numerical feedback on outputs.

The provocative counterfactual is that sufficiently good datasets and scaled-up older techniques might have produced something resembling InstructGPT. Preference pairs, for example, can support supervised learning without the original InstructGPT techniques. The researcher also speculates that a sophisticated recurrent neural network could have supported something like ChatGPT, while acknowledging worse scaling curves and declining to extend that confidence to reasoning models.

The counterargument gives efficiency its own causal weight. Better compute or data efficiency can function like access to additional data that would otherwise be expensive to collect. Better optimizers can reduce training time and compute requirements. Discovering a new learning signal and improving how effectively a system learns from it both matter.

The dataset thesis suggests that the next major advance may depend on a learning signal not yet being used, perhaps in video, embodied systems, or reasoning. Asked to name the next paradigm, the researcher declines: the next wave cannot be derived from first principles with confidence.

The closing discussion returns to the value of a narrow research territory. A graduate student can become the authority on a small set of questions, eventually earning an advisor’s deference within that niche. Here, that territory is information in model weights and activations, including models beyond language. The invitation is to engage with the work and discuss related problems by email; Substack and Twitter are also mentioned as places to follow it.