Thinking Longer Is Only Part of the Problem

A strange move: bug or better strategy?

A Diplomacy bot makes a move that experienced humans would rarely consider. Is the system broken, or has it found something they missed? Noam Brown describes this as a debugging problem: recognizing an unconventional success requires domain knowledge. While developing Cicero, an AI for the seven-player negotiation game, he played tournaments and studied tutorials and game commentary. The bot’s unfamiliar strategies also taught him how to play.

Brown kept playing after Cicero’s late-2022 announcement and says he won the world championship in 2025. He did not use Cicero during the tournament, but believes inspiration from its play helped. That experience opens this Latent Space conversation: an AI’s strategy can improve a human’s understanding even when the human ultimately acts alone.

Strategic ability coexisted with conversational failures. Brown recalls Cicero denying its own earlier messages and accusing a player of lying despite the chat history supporting the player. He believes people overlooked these anomalies because they were not expecting a bot, interpreting them as fatigue, intoxication or trolling. The small language model was a bottleneck; the project reinforced the value of larger models.

Cicero conditioned language generation on concrete actions. A reasoning system steered the conversation toward behavior researchers could define and interpret. Brown reports that safety researchers welcomed this controllability. He had not tested O-series models on Diplomacy himself; having leading bots negotiate and compete remained a benchmark he wanted to see.

Extra reasoning needs capability and a measure of success

Test-time compute is computation spent working on a problem at inference, including extra thinking before answering or acting. Brown expected the progression from o1-preview through o1 to o3 to continue toward broader capabilities. His everyday example was web research: he used o3 as a smaller Deep Research workflow, describing responses in roughly three minutes—a personal usage example rather than a latency guarantee.

Deep Research grounds Brown’s objection to restricting reasoning models to math and coding. A research report has no simple answer key, yet he considers the product evidence that reasoning helps with less clearly defined work. His impression is that users can distinguish good reports from mediocre ones, despite the product’s imperfections. Improvement needs a measure of success; that measure need not be easy or entirely objective.

The System 1/System 2 analogy distinguishes fast learned capability from slower deliberation, but the two depend on each other. Extra thinking needs a sufficiently capable pretrained model; the respondent argues that applying the reasoning paradigm to GPT-2 would have accomplished almost nothing. Prompting models to reason through intermediate steps appeared to help larger models more than very small ones, though the respondent had not made the measurements needed to classify this as emergence.

The examples distinguish where deliberation might help: - Working through available information: GeoGuessr is offered as a visual task that benefits; ordinary image recognition is tentatively expected to benefit less. - Recovering missing information: Without additional information, thinking longer cannot reliably produce an unfamiliar person’s exact birth date. Historical context may support only an estimate. - Choosing a strategy: GPT-4.5 is described as drawing a tic-tac-toe board and making legal moves while still making strategic mistakes. The respondent considers deliberation necessary for perfect play at that point, while allowing that a stronger future model might play perfectly without it.

Learned intuition makes deliberation faster. A capable newcomer might reason through an unfamiliar game given enormous time, but experience reduces that burden. Stronger learned capability and extra thinking can reinforce each other.

Check before acting; expect scaffolding to change

Pokémon introduces an engineering disagreement. The respondent says o3 makes little progress without a specialized harness—the surrounding machinery that helps a model operate in the game. Their preferred research response is to accept the poor score and improve general capability. The ambition is for stronger models to outgrow much of this support.

Tools still have a place. The important distinction is when an action takes effect: - Check before committing: An optional tool can tell the model whether a proposed move is legal. - Act, then retract: The model makes an illegal move, receives feedback and substitutes another. The respondent objects to relying on this pattern and considers losing after an illegal move a reasonable evaluation rule.

A robot makes the distinction concrete. Simulation might reveal that a motion would break an object, allowing the robot to reject it. Once the object is physically broken, feedback cannot undo the damage. Testing a possible action and reversing a real action are different capabilities.

A router that selects among models faces another tradeoff: a less capable model may recognize a hard request and escalate it, but may also be fooled or overconfident. One participant predicts that unified models will absorb much of this machinery, just as reasoning models displaced some elaborate systems built from repeated GPT-4o calls. That remains a fallible prediction, with routing across providers acknowledged as a continuing possibility.

Build a router when it improves the product today, but budget for replacement. The advice is to avoid spending six months on scaffolding that model improvements might soon displace. Another participant challenges its practicality: developers repeatedly hear that they should build for future capabilities without being told how.

Data for reinforcement fine-tuning can remain useful across model generations. The discussion recommends this approach for specializing models with developer-held data often absent from the base model. A future, stronger model can be fine-tuned on that data too, making collection an investment that complements model scaling.

Promising experiments became a contested scaling bet

The historical account returns to poker, Hanabi and Diplomacy. A participant describes the benefit of thinking before acting as comparable to making a model 1,000–100,000 times larger. This is an asserted model-size equivalence, not an accuracy multiplier or speedup; no metric or experimental setup accompanies the range.

Recounting a late-2021 meal with Ilya, the speaker describes believing that pretraining would hit economic limits before reaching superintelligence, making a general reasoning paradigm necessary. Ilya reportedly agreed about the need but thought discovering it might be easier. The speaker now considers their expectation of a long wait mistaken.

Earlier reinforcement-learning work is described as supplying building blocks, while faster and more capable models made experimentation easier. The breakthrough emerged through increasingly persuasive experiments, with the speaker becoming convinced around October or November 2023. OpenAI’s subsequent investment in scaling those results is credited with affecting when reasoning models arrived.

Promising results did not settle resource allocation. Researchers debated whether reasoning and reinforcement learning primarily improved data efficiency or opened another way to use compute, and whether the gains would generalize. OpenAI was smaller and more compute-constrained in 2023; funding reasoning meant drawing resources from other work. A participant credits its startup-style organization and shared mission with helping teams pool resources and make those choices.

Data efficiency—how much experience learning requires—remains an open problem. The respondent considers models less efficient learners than humans, while noting that a child’s developmental experience has no straightforward token equivalent. Better algorithms could improve efficiency, and internet data does not exhaust the experience a system could learn from.

Coding agents move the bottleneck to review and experience

Asked about his coding stack, Brown says he had recently been using Windsurf and especially Codex. He tries Codex first for essentially anything he would otherwise code himself, describing tasks returning as pull requests roughly five minutes later. He values both the productivity and the opportunity to discover shortcomings that research should address.

Expectations adjust quickly. Brown recalls revisiting the original Sora announcement videos and noticing unnatural movement and inconsistencies that his initial excitement had obscured. Similarly, impressive coding output soon gives way to demands for a better working experience.

Participants identify three distinct constraints: - Interactive latency: A Windsurf user says o3’s reasoning time breaks coding flow. The response suggests GPT-4.1 for interactive pair programming and self-contained Codex tasks for asynchronous delegation. - Review capacity: Generating many pull requests can exceed the developer’s ability to review them. - Accumulated experience: A model may spend ten minutes on a task, then another ten minutes on a similar task without benefiting enough from the first.

The desired agent resembles an employee six months into a role: it has experience specific to the job. Asked whether manual Codex environment setup reflects compute constraints or capability limits, the respondent emphasizes limited development time without resolving that diagnosis. The broader expectation is that delegation will extend beyond software into other remote work.

Virtual assistants introduce the principal-agent problem: will someone carrying out a delegated task follow the user’s preferences? A participant suggests well-aligned AI might do better than a human would, carefully revising the comparison with what a human could do. But alignment must specify whose interests count. Obeying an individual with a catastrophically harmful goal can conflict with broader human interests.

Longer reasoning and an AI civilization

A participant describing their team’s research identifies two directions: extend useful thinking from minutes toward hours or days, and study agents that collaborate and compete. The longer durations are research ambitions, starting from the roughly fifteen-minute thinking periods described in the discussion.

The multi-agent motivation is a civilizational analogy. Semiconductors, nuclear reactors and lunar travel reflect accumulated cooperation and competition across generations, the participant argues. An AI civilization could similarly let billions of agents build on one another’s work over extended periods, producing capabilities beyond those of present systems. This is a long-term vision.

How new knowledge would be stored or incorporated remains undisclosed pending an announcement. The speaker’s stated preference is for approaches that scale, criticizing much existing multi-agent work as overly dependent on heuristics.

Robust play and adaptive play serve different objectives

Game-theory-optimal play aims to be unexploitable in expectation. Uniformly random rock-paper-scissors illustrates the principle: an opponent cannot gain an expected advantage from your choice pattern, although you can still lose individual rounds.

Poker exposes the choice between robustness and adaptation: - Protect against exploitation: An equilibrium strategy can earn positive expected returns when opponents make mistakes, though it may leave some potential profit uncollected. - Exploit a particular opponent: Bluffing more against someone who folds too often may earn more. Departing from balance also creates weaknesses the opponent can exploit.

The trap is easy to picture: someone plays paper five times, you switch to scissors, and their next move is rock. An observed pattern offers evidence, not certainty. The poker researcher reports using an equilibrium-oriented approach to defeat top human players, while acknowledging that an adaptive human expert might win more against weak opponents.

Sample efficiency limits opponent adaptation. The speaker estimates that humans can form useful profiles within about a dozen hands, whereas the poker systems they worked on in the mid-2010s needed around 10,000. They believe newer technology has reduced the requirement but give no updated measurement.

Diplomacy changes the requirement because players must cooperate as well as compete. The speaker says importing the equilibrium-oriented poker approach worked poorly. Modeling other players and adapting to their styles became necessary for cooperation—like switching to French when everyone else at the table speaks French. Adaptation can enable cooperation without exploiting weakness. Transferring these techniques back to exploitative poker remains a proposed research direction.

Self-play still needs the right objective

Does understanding other players require a separate component? One participant expects larger models to develop better implicit world models and theory of mind—an understanding of other agents’ motives and behavior. They describe abandoning their earlier conviction that explicit agent models were necessary. Another participant argues that interaction requires distinguishing among hypotheses about the entities involved. Whether scale can absorb that work remains an open expectation.

AlphaGo offers a tempting analogy: learning from human games resembles internet pretraining, and its Monte Carlo tree search represents extra computation used to choose an action, analogous to inference computation in reasoning models. The apparent next step is self-play, in which a model plays against itself and learns from the resulting games.

The analogy’s boundary is the objective. Two-player zero-sum games offer a clear target in minimax play: a strategy that protects against an opponent’s attempts to exploit it. The speaker describes self-play as guaranteed to converge to this target, without specifying the algorithm or assumptions required. Broader multi-agent settings do not automatically inherit a similarly useful target.

Increasing difficulty does not necessarily improve useful capability. A mathematical question-setting model could challenge a solver with thirty-digit multiplication: difficult for the models, but not progress in the intended direction. Choosing what self-play should optimize beyond these games remains unresolved; the respondent offers no replacement objective.

Iteration speed and evaluation shape research

Asked to choose between autoregressive generation and diffusion, the respondent favors exploring multiple research directions and declines to discuss diffusion-based reasoning. The robotics detour makes the value of experimentation concrete: a participant recalls moving from a robotics master’s program toward poker AI after seeing how slowly physical-hardware research progressed. They attribute some of software AI’s faster progress to easier iteration.

Robot embodiment presents competing considerations: - Fit human environments: Spaces made for people may favor humanoids, and familiar forms may appeal to users. - Avoid human physical limits: Other body plans can suit particular tasks better; drones are a concrete example. - Account for discomfort: Near-human resemblance might itself feel unsettling. These remain tentative arguments, with no strong conclusion about the best general body plan.

Reproduce promising papers, then test whether they scale. A participant describes academic research as an important source of ideas, with colleagues’ recommendations helping prioritize reading. Interesting results undergo internal reproduction because published promise sometimes fails to replicate or survive scaling. The same speaker regrets research visibility depending on social media, while recalling helping graduate students present their papers in Twitter threads.

Easy grading can narrow what gets evaluated. Humanity’s Last Exam illustrates the pattern of extremely difficult questions with readily scored answers. Valuable tasks often have less crisp outcomes, making benchmarks harder to construct and probably more expensive to run. The respondent insists that those qualities are measurable, even when measuring them is difficult.

The two ceilings: money and serial time

Test-time scaling faces two distinct constraints: - Economic cost: Extending reasoning from minutes toward hours, days or weeks consumes more resources. Spending remains finite even when further computation is useful. - Wall-clock time: A three-week run takes at least three weeks before its evaluation can inform another experiment. Some research decisions depend on completed earlier results, limiting iteration speed. These durations are hypothetical scaling examples, not demonstrated capabilities or a prediction that progress stops in a particular year.

Models can also do more with the same compute. The respondent describes o3 as substantially better than o1-preview despite only a modest increase in thinking time on some questions. The comparison is qualitative, but distinguishes two sources of progress: thinking longer and thinking more effectively.

Drug discovery adds an external clock: establishing whether a treatment extends human life without serious side effects can require a long observation period. A participant hopes reasoning models could help develop better biological simulations, while disclaiming expertise in biology and chemistry. Simulation is raised as a possible aid, not an established replacement for that observation.

Narrowing model behavior, widening the planning horizon

The late discussion of training terminology leaves midtraining loosely defined: it occurs between pretraining and posttraining, without a specified operational boundary or objective. OpenAI models are described as passing through both stages before release, becoming substantially easier to use than raw pretrained checkpoints.

A participant argues that chat posttraining narrows the range of outputs, losing unusual uses of raw models. Another accepts that narrowing and defends its purpose: reduced behavioral variety can make a model more useful.

For a forthcoming interview with Greg Brockman, the suggested focus is the range of outcomes five or ten years ahead and what individuals or society could do to steer toward better ones. The recommendation is to spend more attention on that horizon alongside near-term developments.

A lighter turn recommends Blood on the Clocktower, a social game compared with Mafia or Werewolf. A secondhand story about a startup using it for recruiting prompts skepticism: being good at deception and detecting deception does not obviously predict being a good employee.

Too many hidden possibilities—and the case for general reasoning

Magic: The Gathering closes the conversation with a concrete limit of specialized game AI. Hidden-state enumeration explicitly represents possible private information and assigns probabilities to it. The poker researcher describes enumerating possible Texas Hold’em hands, feeding their probabilities into a neural network and obtaining actions for the possibilities represented.

The participant gives 1,326 possible two-card hands and describes a six-player representation using five times that many entries for the other players. This describes per-player hand entries, not a fully specified joint-state count; the explanation does not detail how visible cards or incompatible hands affect it.

More private information makes enumeration unwieldy: - Omaha: Four private cards make the problem harder, even with possible heuristic reductions. - Stratego: Forty pieces provide a much larger example, described with a rough estimate near forty factorial possibilities. - Magic: The Gathering: The poker search techniques would not work out of the box; handling such large hidden-state spaces remains an active research problem.

This bottleneck belongs to the search techniques under discussion. The speaker distinguishes them from model-free reinforcement learning, which does not require this particular enumeration. They speculate that enough effort could already produce a superhuman Magic bot, without reporting such a result.

The closing preference is general reasoning over extending a specialized search method. Better poker search might transfer to Stratego or Magic, but would not thereby make a language model superhuman at Codeforces. The more compelling ambition is a general model that eventually plays Magic at a superhuman level out of the box. That prediction returns to the Pokémon argument: improve the underlying capability until the difficult game becomes one more thing the model can do.