From On-Policy Learning to IMO Gold: How Much Can One Model Learn?

The usefulness threshold starts with a spreadsheet

A large spreadsheet of experimental results needs to become a plot. The researcher has the numbers but dislikes the visualization work; giving a model a screenshot removes a recurring chore. That example opens the Latent Space conversation, alongside a personal assessment that AI coding and Nano Banana image generation have crossed a practical usefulness threshold.

The research story begins with a return to Google after roughly eighteen months away. The guest compares it to resuming a saved game: familiar infrastructure and the same username, although Brain had become part of Google DeepMind. The motivation was frontier model research. Deep Think, the researcher says, did not yet exist when they returned.

The technical transition was larger. After working on architectures and pretraining, the researcher moved into reinforcement learning (RL) with almost no prior background in it. Their explanation is that research skills transfer across toolsets: unfamiliar methods need not prevent useful experimentation. RL became their principal modeling tool.

On-policy learning: generate, reward, update

The key distinction is where training trajectories—the outputs produced while attempting a task—come from: - Supervised fine-tuning (SFT) on another model's outputs: imitate externally generated examples. The guest uses this as an off-policy example. - On-policy reinforcement learning: generate outputs from the model being trained, evaluate them with a reward, and update that model using its own attempts.

The on-policy loop resembles self-distillation through rewarded generations: the model learns from its own outputs. The researcher believes this generalizes better than imitation, while acknowledging that the gap between SFT and RL still needs scientific investigation.

Sports provides a compact analogy. A beginner copies movements or watches tutorials, then learns through games and feedback from actual performance. Imitation still has a role. The guest cautions against taking human–model analogies literally; the interviewer is more willing to use machine learning as a guide to human learning.

That disagreement leads to belief revision. The interviewer recalls Stable Diffusion running on a laptop overturning their expectations for generative images and prompting the pivot that produced Latent Space. A human “learning rate,” in this metaphor, is the willingness to reconsider related assumptions when a major expectation fails. The researcher agrees with the broader scientific habit: new paradigms can invalidate methods that once seemed definitive.

Multiple attempts connect training to parallel reasoning. The guest says their RL training samples several outputs and relates this to self-consistency, while stressing that self-consistency is more nuanced than majority voting. The interviewer describes another way to select candidates: a language-model judge can inspect the reasoning, assessing a trajectory's internal consistency rather than only counting matching answers. This does not specify Deep Think's selection algorithm.

IMO gold with a text-in, text-out model

The IMO mathematics competition story begins with a timing distinction. Roughly a week applied to training the particular competition model, in the contributor's estimate, not the full training process or research program. The broader effort was longstanding, with AlphaGeometry and a separate proof system already involved the previous year.

The newer goal was Gemini taking text in and producing text out, without a second solving system. The contributor viewed success by the model itself as a meaningful test of progress toward artificial general intelligence, or AGI. Their responsibility was to train the model with colleagues and deliver the checkpoint; others led the broader competition effort.

The interviewer had expected a language model to operate a computer, write proofs in Lean, and run a verifier. The response reframes the question: how much tool-like capability can be encoded in model parameters? Calculator-like behavior offers a modest analogy; proof systems and physics engines suggest more ambitious possibilities. The contributor favors a general model that absorbs many capabilities, retaining tools for what it cannot absorb. Those limits remain open, and the contributor does not claim to know the internal debate behind the architectural decision.

According to the contributor, the IMO model also launched as general-purpose Gemini Deep Think, but the public and competition inference configurations differed. Some mathematicians received the full IMO configuration; inference cost limited its distribution. Sharing a model did not mean every user received the same computation.

The contributor could not parse the mathematical output themselves—a striking example of researchers building a system that succeeds beyond their own domain expertise. That makes verification consequential. Here, “non-verifiable” means non-trivial to verify: checking a long proof requires more than checking a final answer. The detailed verification machinery and the amount of domain-specific post-training separating mathematics from programming competitions remain undisclosed.

A live competition changes the engineering

The IMO was an unfolding event, not a benchmark to rerun until a target was reached. Team members were in Australia, and designated people ran inference as the competition unfolded. The contributor recalls gathering with colleagues in London and seeing the pieces come together with the intensity of a hackathon.

Four training captains worked across London, Mountain View, and Singapore. They passed ownership around flights, meetings, and availability: someone about to leave for twelve hours handed the run to another captain. Bugs and failed jobs still needed attention. The memorable IMO Cat label was simply a job configuration name chosen by someone who likes cats.

During training, nobody knew whether the checkpoint would reach gold. The contributor describes waiting for committee verification and watching contestants' scores because the gold threshold depended on human performance. That suspense made the effort a personal highlight. They credit collective progress across Gemini, while declining to assess whether inference optimization was the hardest part: they did not work on inference-time scaling.

Pokémon tests whether knowledge becomes action

Pokémon shifts the focus to agents that must act over long sequences. The guest says the Gemini work under discussion included no Pokémon-specific work. They value the game as an occasional capability check that they believe receives relatively little targeted optimization.

The interviewer's concrete failure example is an agent repeatedly running into a wall, which they interpret as weak screen understanding and spatial reasoning. The guest points to the harness—the surrounding software that supplies observations to the model. Emulator memory and explicit game-state information give an agent different inputs from images alone. That distinction matters when interpreting its behavior.

The guest proposes two challenges beyond simply reaching the end: - Completion speed: measure how efficiently the agent finishes. - Completing the Pokédex: test broader planning and research than the relatively linear main game. A hypothetical agent might discover that a Pokémon evolves through trading, find a partner online, and coordinate the exchange.

The guest has not seen a model complete the Pokédex. The difficulty they identify is applying web knowledge within a changing visual state. Looking up the right instruction does not complete the task; the agent must connect it to what is happening in the game.

The interviewer pushes further: following a guide and inventing it test different capabilities. Scientific invention requires searching novel ideas efficiently. The guest initially frames guide creation as trial and error, but agrees that generating new knowledge matters. Their thought experiment is to restrict a model to knowledge available around 2012, assume no information leakage, and ask whether it could invent the Transformer. Success remains an open question.

Reasoning need not be readable text

The interviewer interprets reasoning as post-training eliciting capabilities already latent in pretraining. The guest uses a broader working definition: post-training and RL that improve capabilities through thinking. They emphasize that reasoning has no single agreed definition.

Two forms of intermediate computation make the distinction concrete: - Discrete chain of thought: decode additional intermediate tokens before producing the answer. - Latent thinking: perform intermediate computation in internal representations without decoding those tokens. Whether these approaches become equivalent remains open. The guest sees no requirement for model thought to resemble human thought.

The interviewer raises a related representation hypothesis: image and text models might develop similar internal concepts, such as numerical representations of a laptop, despite differences in their actual vectors. This is a proposal about convergence, not a demonstrated equivalence between the models discussed.

Published reasoning traces suggest another possible feedback loop: later models could learn from them during pretraining. But publication does not establish inclusion in training. The guest questions their prevalence and notes that identifiable sources can be filtered. Similarly, when the interviewer argues that pretraining on code improves reasoning beyond coding, the guest questions whether that finding still holds for contemporary models.

Measure coding assistance by time recovered

The opening spreadsheet chore returns as a more consequential workflow. A training job fails; the researcher pastes the error into Antigravity, requests a fix, and relaunches. They estimate that some issues would otherwise take roughly twenty minutes to diagnose, with occasional bugs that could consume a day. These are personal estimates, not controlled productivity measurements.

Initially, the researcher inspected every fix. As trust grew, they relaunched with less checking. Yet a model can claim a bug is fixed while leaving it unresolved. Its usefulness remains uneven: some problems are easy for models and hard for humans, while others reverse that relationship.

The guest proposes net time saved across a team as the evaluation target. Assistance can recover pieces of many people's work without replacing a whole junior engineer, like a passive game ability that improves everyone nearby. The interviewer separately raises the career question: if junior work is automated, how will newcomers gain experience?

The discussion distinguishes two routes to improvement: - General capability gains: the guest initially expects broader model improvements to remove coding weaknesses. - Targeted training data: the interviewer argues that datasets aimed at specific failures can accumulate into broader progress. The guest accepts the second route with a prerequisite: define the failure and its evaluation before targeting it with data. Coding specialists may already have those categories; the guest does not work on coding.

Architecture, learning algorithms, and why ideas still matter

Attention alone does not explain progress in the researcher's account. Large-scale pretraining, training tokens, and RL also matter; GPT and BERT helped reveal the Transformer's potential beyond its original translation setting. The researcher predicts that an eventual AGI system will still be a Transformer, unless the broader computational or learning paradigm changes substantially.

That forecast draws on efficient-attention research. Attempts to remove or simplify attention repeatedly ended up retaining at least one self-attention layer, the researcher recalls. Mixtures of local and global attention still count as attention in this view. The interviewer challenges the forecast with hypothetical contexts of hundreds of millions of tokens and continual learning.

The response separates architecture from the learning paradigm. Architecture is the interface between a learning algorithm and tokens; continual learning also raises questions about gradient descent and backpropagation. For extremely large contexts, the researcher proposes several avenues: - Learn during inference: accumulate memory through updates as information arrives. - Make context processing cheaper: develop less expensive architectures. - Improve accelerators: change the hardware cost of computation. These remain possible directions, not demonstrated systems.

New ideas must work with accumulated optimization. The researcher argues that compatibility with earlier research and engineering helps determine which ideas succeed, potentially keeping the field near a Transformer-centered local optimum. The interviewer remains unconvinced that this investment implies permanence: the field is young enough for a different approach to emerge.

The researcher also rejects the idea that recent progress has merely scaled existing systems. Their “sweet lesson”: ideas matter. Choosing what to scale remains consequential, and they believe useful ideas have not entered diminishing returns.

When data is scarce, learn more from each example

A brief hardware discussion leaves the serving bottleneck unresolved. The interviewer argues that networking and memory improve more slowly than compute and therefore constrain scaling. The researcher disagrees with low confidence, emphasizing that inference serving is outside their main focus.

Data efficiency yields a clearer question: given a fixed dataset, how much can a model learn? Repeated passes through the data—epochs—matter only when they produce meaningful improvements. The interviewer calls maximizing learning from limited data “data-optimal training.” The guest agrees with the goal, without claiming detailed knowledge of pretraining-data practices across labs.

Human learning motivates the question but does not explain the gap. Recognizing a dog after a few examples is the interviewer's intuitive comparison. The researcher considers computation per input, architecture, backpropagation, and learning from experience as possible explanations. Yet human exposure includes continuous, varied sensory inputs, making the denominator in a human–machine comparison difficult to establish.

The interviewer retracts an earlier belief that pretraining was effectively dead, citing conversations and talks that persuaded them investment continued. They then speculate about a different route to efficient learning: maintain internally consistent candidate worlds and discard those contradicted by new evidence.

The term world model covers several different proposals in the exchange: - Video or spatial models: represent a visual world. - Execution-state models: predict an execution environment's internal state as code runs line by line. - A broader latent account of the world: refine internal representations against observations—the interviewer's most speculative category. The researcher finds the umbrella term insufficiently defined and returns to the concrete problem of learning efficiency.

Their preferred direction is to spend more computation per token when data, rather than compute, is the constraint. More floating-point operations, or FLOPs, might extract more learning from each example. Architecture and algorithm improvements can overlap: a better architecture may effectively devote more computation to the same input. Which method best achieves this remains open.

RL environments raise a related question about where expertise enters training. The interviewer imagines shopping agents practicing across replicas of e-commerce sites and asks why labs would buy such environments instead of building them. The guest considers domain expertise encoded in an environment potentially valuable, but does not know the purchases or their rationale and does not endorse the shopping proposal.

Generative retrieval predicts document identifiers

Recommendation brings the discussion back to a concrete application. The interviewer groups feeds, search, ranking, filtering, and personalization as a particularly valuable consumer AI problem. The guest connects this to DSI, a retrieval project that aimed to store retrieval information in model parameters, using T5 models and document ranking on Natural Questions.

The initial mechanism was direct: assign each document an identifier and train the model to predict it. The contributor says the model could memorize these associations. Semantic IDs added semantic associations and a hierarchy, giving structure to the search space beyond arbitrary document identifiers.

Generative recommendation transferred that approach to recommending items, adapting the construction of semantic IDs to the domain. The contributor views retrieval and recommendation as closely related ranking problems. DSI remains one of their more creative projects, although they no longer closely follow the area.

The interviewer describes two reported deployments with different levels of architectural detail: - YouTube: public presentations described semantic IDs for recommendation, Gemini involvement, and uploaded videos encoded into a codebook. The account does not establish model size or retraining cadence. - Twitter: a reported switch to an LLM-based recommendation system leaves open whether it uses generative retrieval. Using a language model does not by itself establish that the system generates item identifiers.

The interviewer expects stronger language models to handle unusual query intent better, depending on the query set, but acknowledges not knowing the comparative results. The researcher emphasizes that BM25 remains a strong retrieval baseline. Their academic experiments often responded unpredictably to architectural changes that felt sensible in language modeling.

The researcher's analogy is badminton: ordinary modeling feels like hitting the racket's sweet spot; recommendation experiments can feel like hitting the shuttlecock and unexpectedly hearing glass shatter. They limit this frustration to academic experience and benchmarks they consider detached from industry needs. Online evaluation would offer a different experience, and the interviewer maintains retrieval's business importance.

Singapore: talent density and research judgment

The closing discussion returns to the Singapore Gemini team. Its formation was a gradual process of hiring, with a symposium intended to inspire the local community and demonstrate that Singapore could be close to frontier research. “Reasoning and AGI” expressed the ambition in a job posting; it was not a formal team name.

The researcher recalls visiting the Istana to discuss Deep Think and the IMO effort, before the conversation with Lee Hsien Loong turned toward macroeconomics and politics. On research geography, the guest emphasizes talent attracting talent: strong leaders and core model work can draw regional candidates who might otherwise join elsewhere. Time zones have advantages and disadvantages; round-the-clock coverage is not presented as the team's purpose.

Singapore also offers the researcher some distance from the Bay Area's pervasive AI culture while preserving professional connections. At the time of the conversation, the team was hiring for RL and reasoning on Gemini and Gemini Deep Think, starting small because talent density and compute per researcher matter.

Hiring signals included an RL research record, exceptional achievements such as coding competition results, and strong engineering ability. The recruiter considers machine-learning knowledge learnable. The interviewer offers a student-budget example: an unnamed Best Paper winner's research on thousand-layer networks for RL, followed by recruitment to OpenAI. They explicitly leave open whether production models will ever use that many layers.

The recruiter describes discovering published work and contacting its authors directly. The interviewer pushes back on the advice to simply publish good work: graduate students may need help choosing worthwhile problems. The recruiter distinguishes two signals: - Execution: solve a problem someone else selected. - Research taste: independently choose a useful problem and produce work that resonates with researchers in the field. Choosing the problem reveals judgment beyond the ability to complete an assignment. Both speakers recognize how difficult it is for students to demonstrate that judgment amid growing competition.

The final topic is sustaining the person doing the research. The guest describes tracking health data, losing substantial weight, and feeling near their personal peak in physical health. They believe this has improved their work and treat health activities as part of the job because they support performance. When the interviewer speculates that physical hunger might encourage intellectual hunger, the guest rejects the connection in their own experience: hunger makes them think about food and makes work harder.