Moving the AI Frontier: Distillation, Data Movement, and Better Specifications

An accurate model that is too expensive to serve

An image classifier can become more accurate and less practical at the same time. Early in the Latent Space conversation, Jeff Dean recalls training on 300 million images across perhaps 20,000 categories—the category count is uncertain. After broad pretraining, the team trained specialists on enriched streams of images, such as mammals or indoor scenes. Combining perhaps 50 specialists improved performance, but serving the entire ensemble was unwieldy.

Distillation transfers capability into a deployable model. The teacher in that early example was an ensemble; today, Dean explains, it is often one much larger model. Its outputs provide richer training guidance than hard labels alone, helping a smaller model learn through repeated passes over a large dataset. When Alessio credits Dean and Hinton with distillation, Dean adds Oriol Vinyals.

This connects Gemini Flash’s serving economics to Gemini Pro’s frontier capabilities. Dean argues that both matter: - Frontier models reveal new capabilities and handle difficult reasoning and mathematics. - Smaller models make applications such as coding agents more affordable and responsive, drawing capabilities from larger teachers through distillation. Dean reports that across multiple generations, the next Flash has matched or substantially exceeded the previous Pro, without naming benchmarks or quantifying the gap. Google also has internal models that need never be served, though he does not confirm the host’s suggestion that an unreleased Ultra supplies the teacher.

Swyx asks whether distillation might also combine gains from reinforcement learning without losing capabilities elsewhere. That remains an aspiration. Dean emphasizes a concrete serving problem: moving from a request for a for loop to an entire software package requires many more tokens before completion. Lower latency makes larger assignments practical, alongside the savings from a smaller model.

Harder requests need harder evaluations

If inexpensive models eventually satisfy most existing tasks, why keep funding the frontier? The answer depends on whether user requests stay the same. As models improve, people delegate harder work. A worldwide report on solar-panel deployment illustrates a demanding prospective assignment. Such requests also expose failures that guide the next generation.

Benchmarks have a useful lifespan. The preference expressed here is for initial scores around 10–30%, leaving room for meaningful progress; scores near 95% suggest diminishing returns from continued focus. These are rules of thumb. High public scores can reflect either acquired capability or exposure to benchmark material during training. Held-out internal evaluations therefore target capabilities the model still lacks, helping distinguish a need for specialized data from architectural or other model improvements.

Long context—the material available to a model while answering—makes this distinction tangible. Finding one isolated fact in a large input, the single-needle-in-a-haystack task, is described as largely saturated around 128K tokens. More useful tests require combining several pieces of evidence or producing an answer from a thousand pages or multiple hour-long videos. Locating a product number tests less than synthesizing the surrounding material.

The longer-term goal is effective access to internet-scale information. Existing quadratic approaches cannot simply be extended from million-token contexts to billions or trillions: their cost grows too quickly. Specify the desired capability before choosing the architecture. Reaching that scale requires algorithmic and system improvements, potentially supporting information across many videos and, with permission, a person’s emails, photos, documents, and travel details.

Video becomes records; retrieval makes context selective

Multimodal training balances breadth and volume. Gemini was intended to work across text, images, video, and audio from the start. The argument extends to specialized data such as vehicle sensor readings, medical imaging, and genomics. Including every available example may be a poor pretraining tradeoff, while some exposure can familiarize the model with a data type and its meaning.

A recalled Gemini demonstration illustrates video-to-structured-data extraction. Given a compilation of roughly 18 sports moments, the model produced an 18-row table identifying events, dates, and short descriptions. No accuracy measurement accompanies the example. Video matters here because motion helps a model interpret events over time; that does not establish the interviewer’s suggestion that vision supersedes every other modality.

Access to much larger collections brings the problem back to search. A retrieval cascade spends increasingly sophisticated computation on progressively fewer candidates. The proposed language-model version follows three stages: 1. Find candidates cheaply and in parallel. Lightweight models identify perhaps 30,000 relevant documents. 2. Narrow the evidence with stronger models. A more capable model or ensemble reduces that collection to a smaller set. 3. Use the most capable model on the final evidence. It examines the selected documents to carry out the task. The example’s 30 million candidate tokens and 117 final documents are hypothetical sizes, not production settings.

Effective access to trillions of tokens can come from selective retrieval. The final model examines a relevant subset, much as search exposes the internet through a small collection of results. Language-model representations also allow relevance to depend on a page’s topic rather than the presence of exact query words—a goal that predates extensive language-model use in search.

Growth can make a different architecture viable

Around 2001, Google search faced two distinct scaling pressures: - Index coverage: adding pages improved the chance of finding a relevant document. More shards—partitions of the index—bounded per-query latency as the index grew. - Serving capacity: additional replicas of each shard handled increasing traffic. In the recalled configuration, 60 shards with 20 copies each meant 1,200 machines. Their aggregate memory could hold one complete index, enabling the reported move of Google’s search index into memory in 2001.

Cheaper access enabled better retrieval. With the disk-based index, each additional query term required a seek on every shard. In memory, expanding a three- or four-word query to about 50 terms became practical. Variants and related words—restaurant, restaurants, cafe, bistro—helped recover the user’s intended meaning. A storage change opened a new way to improve search quality.

The design rule is to identify the parameters that matter—throughput, index size, bytes per document, retrieval behavior—and accommodate roughly 5–10× growth in those parameters. At 100×, a different architecture may become viable. Traffic-driven replication supplied the memory for the new index design; extending the original architecture indefinitely could have missed that opportunity. Explore the design space before investing heavily in code.

Freshness changed even more dramatically, from monthly index updates to updating an individual page in under a minute. Google News could not satisfy people entering news queries into ordinary search. Recrawl scheduling therefore considered both the likelihood of change and the value of detecting it: an important, rarely changing page can still deserve frequent checks.

Count data movement before multiplications

Latency Numbers Every Programmer Should Know supplies ingredients for quick design estimates. For image-search thumbnails, compare precomputing small images with generating them from larger originals on demand: how much disk bandwidth and how many seeks would each require? Develop the same intuition for the libraries actually in use, including hash-table lookups and sorting a million numbers.

For model training and inference, start with where state must move: memory on the chip, memory attached to the accelerator, main memory, or the network. Then compare movement costs with the arithmetic performed after the data arrives. An illustrative contrast is roughly 1,000 picojoules to move a parameter across a chip versus roughly one picojoule for a multiplication. Those figures lack a specified chip, precision, and transfer width; they illustrate how movement can dominate the cost of using a weight.

Batching amortizes weight movement. Once a parameter reaches an arithmetic unit, reusing it across a batch of 256 spreads the transfer cost over many operations. Batch size one offers attractive latency but little reuse, creating an energy and cost penalty. The serving tradeoff has a physical basis.

Memory placement offers another route. Google’s TPU accelerator chips form regular two- or three-dimensional meshes, each with attached memory. Model parallelism spreads a model across chips; a sufficiently small model might fit entirely in their faster on-chip memory, called SRAM. The example uses 16 or 64 chips and describes potential improvements in both throughput and latency. The benefit depends on the distributed model fitting in SRAM.

Designing chips for future models

Co-designing TPUs with frontier machine-learning research requires forecasting workloads roughly two to six years ahead. A chip may take about two years to reach a data center and must remain useful for several more. Substantial research-driven features often target generation N+2; smaller changes may fit into N+1.

The commitment determines how much evidence a feature needs: - Small speculative features can be worthwhile when they occupy little chip area and offer a large possible payoff. - Major architectural changes warrant extensive model experiments before committing. Co-design also runs in the other direction. Model architectures must suit the chips that will train and serve them, sometimes anticipating lower-precision support in future hardware.

Fewer bits reduce transfer energy. One approach combines low-bit weights with scaling vectors shared across groups of weights, without committing here to a particular format. Diffusion models are raised as another interesting direction because they avoid sequential token decoding. Energy-based models are also mentioned, without a measured comparison.

Speculative decoding reuses weights across proposed tokens. Processing eight proposed tokens together increases the effective batch dimension by eight. If five or six are accepted, the transfer cost is spread over roughly five useful tokens. That illustrates better movement efficiency, not a fivefold end-to-end speedup. Evaluate such techniques jointly through physical energy consumption, latency, and throughput.

Analog computing could offer low-power execution, but conversions between analog and digital representations can consume much of that advantage at system boundaries. Specialized hardware remains a promising route to greater efficiency; the whole system’s costs determine the benefit.

Reliable reasoning beyond easily checked answers

Efficient execution leaves several capability problems unresolved: - Long-task reliability: models must complete complex assignments containing many subtasks. - Model orchestration: one model could use others as tools to accomplish larger pieces of work. - Reinforcement learning without readily verifiable answers: extending gains from math and coding to less easily checked domains could broaden capabilities, but remains an open problem.

A model acting as a critic offers one possible judgment mechanism. Given 2,000 retrieved items, it might identify the 50 most relevant. The evaluator could even be the same model prompted for criticism instead of retrieval. Such techniques are described as effective, without quantitative results or a general solution to verification.

Mathematics supplies a retrospective example of rapid progress: from struggles with elementary word problems to solving difficult competition and Erdős problems in natural language. The specific system comparison is a move from translating problems into Lean and using a specialized geometry model to a unified model described as close to production, with a larger inference budget. General models are absorbing some specialized tasks; generalization to unfamiliar tasks remains the test. The comparison supplies no scores or detailed evaluation conditions.

One participant argues that separate symbolic machinery need not be essential, drawing an analogy to humans manipulating symbols through distributed neural representations. This is an intuition about cognition. The broader prediction—that general models will outperform specialized models in most cases—raises a practical question about where knowledge should live.

What belongs in weights, retrieval, and specialist training?

Retrieval saves capacity, but reasoning still needs world knowledge. Knowing the Golden Gate Bridge’s length can ground a sense of scale; memorizing every obscure bridge’s length has less value when those facts can be retrieved. Alternating retrieval with reasoning over intermediate results could make a model more effective. A proposed personal Gemini would use a shared model to retrieve from emails and photos, rather than require a separately trained model for each person.

Vertical models specialize a capable base through domain training. That leaves data-mixture tradeoffs: more robotics training might improve robotics while hurting translation; broader language coverage could displace training for less common programming languages or multimodal reasoning. These are illustrative tradeoffs, not reported regressions.

A proposed modular system would combine a general base with language, robotics, or healthcare modules invoked when relevant. Knowledge could come partly from retrieval and partly from training. An interviewer asks how such specialists can stay ahead of improving frontier models. Private healthcare data offers a possible source of differentiation for bespoke models, but the exchange establishes neither a necessary token count nor a measured advantage.

Language specialization illustrates two routes: - Learning through context: Kalamang is recalled as an example, without details of the contextual materials or an evaluation of performance. - Additional training: Somali and Amharic illustrate languages where more available text could improve capabilities beyond the base training mixture. A participant then proposes using models to investigate shared concepts across languages—a speculative extension of the idea that different representations can converge on related concepts.

An early language–vision model, whose name is transcribed as “Devise,” provides a concrete example. It fused word representations with an image classifier’s upper layers and trained the combination further. In the recollection, telescope and binoculars were image-training categories, while microscope was absent. The combined model could nevertheless label a microscope image: word representations helped extend recognition beyond the image model’s categories.

Scaling computation—and concentrating the effort

A later recollection traces parallel neural-network training to a 1990 undergraduate thesis. A departmental computer with 32 processors enabled more interesting models, but still lacked enough computation for practical problems in the speaker’s assessment. Around 2008–2009, larger datasets and more compute changed what was feasible. Neural-network work at Google in late 2011 revived earlier ideas about parallel training.

Sparse activation separates total capacity from work per execution. A model could retain trillions of parameters while activating only 1%, 5%, or 10%. The retrospective account credits early sparse models with roughly a tenfold improvement in time or compute needed to reach a given quality, and Transformers with roughly 10–100× lower compute cost than the earlier models compared against them. These are historical estimates without benchmark conditions. The central argument is that hardware, software, systems, architecture, data, and training improvements can multiply together.

The organizational counterpart is making specialized contributions reach the same model generation. Pretraining, infrastructure, post-training, data preparation, evaluations, multimodal capabilities, and coding all need focused attention and coordination. The account of Google Brain’s beginnings is more informal: a micro-kitchen encounter with Andrew Ng, whose students were seeing promising speech-recognition results, prompted a proposal to train much larger networks.

Google’s available data-center resources then were CPUs, without accelerators. Distributed training produced a two-billion-parameter vision model, believed by the team to be about 50 times larger than previous networks. A subsequent ImageNet 22K recollection reports a 70% relative error improvement after multiple weeks, without identifying the comparison baseline. For six or seven years, the working principle remained empirical: larger models and more data repeatedly improved speech, language, and vision results.

An interviewer relays David Luan’s criticism that individual compute quotas fragmented Google’s efforts. The response partly agrees, recalling a one-page memo arguing that separate language and multimodal projects—including DeepMind’s Chinchilla and Flamingo—divided compute, people, and ideas. Combining them into one broadly capable model, multimodal from the start, is presented as the origin of Gemini. The name evoked two organizations coming together as twins, with a second reference to NASA’s Gemini project on the path to Apollo.

Coding agents make specifications consequential

As coding tools improve, engineers can delegate more complex work. Asking for tests gives a model a different role from asking for performance ideas. The interaction style should follow the task: - Delegate independently when the result is clear enough to specify. Give the model a larger assignment and let it return with the result. - Interact frequently when the work still needs shaping. Use successive exchanges to guide the direction. No single style suits every problem.

Many agents introduce unresolved interface questions: when to interrupt, request guidance, and hand back work. One proposed structure resembles 50 interns organized into five subteams. An interviewer worries that accumulated agent context could isolate engineers from human collaborators. The counterhypothesis is that five people managing virtual teams might communicate more easily than five people each coordinating a large human organization. Both remain possible outcomes.

Omitted requirements become implementation choices. Leave out a critical corner case, required behavior, or performance priority, and generated software may miss the intended goal. Clear specifications become more consequential when they drive implementation. A participant compares prompting to careful executive communication and highlights Antigravity’s video input as a high-bandwidth way to convey intent.

Reusable engineering guides could help both agents and the people directing them. A proposed distributed-systems guide would explain failure modes and techniques such as replication or sending a request to two destinations when one successful response suffices. A compact repertoire could inform many tasks without a custom guide for each. A separate concern remains unresolved: models may accept supplied premises too readily and fail to question an assumption.

An interviewer proposes comparing two approaches: - Three fast calls with feedback: a person reviews each output and revises the next prompt. - One stronger call with a developed specification: the person spends more effort expressing the task before execution. There are no comparative results. The hypothesis is that some apparent capability failures arise from underspecified intent: several outputs satisfy the prompt, but only one matches what the user wanted. Fast iteration could resolve that ambiguity.

Faster inference can buy better work

Low latency improves interaction and lets a system do more between user exchanges. Deep Think illustrates the continuing capability–latency tradeoff: its capability would be attractive more often if cost and latency were lower. Even if improved hardware made today’s capability cheap and fast, the expectation is that a stronger model would emerge that benefits from substantially more computation.

Two closing predictions connect the threads: - Personal context could make models much more useful. A shared model could retrieve information users opt to make available, including email, photos, and watched videos. - Specialized hardware could make capable models faster and more affordable. The prediction has no specified timeline or quantitative target.

Would 10,000 tokens per second be useful? The proposed benefits are longer reasoning, more parallel attempts, and more code generation and checking—not a reported performance result. The final example makes the purpose concrete: 10,000 generated tokens could support just 1,000 tokens of code, with 9,000 spent reasoning about it. Faster generation could leave the engineer with a smaller, better-considered artifact to read.