Post-Training’s Moving Bottlenecks: Rewards, Tokens, and Systems

A failing run at half past midnight

At 12:30 in the morning, a training run looks wrong. The problem might lie in a colleague’s code that the researcher watching the run has never worked on. For Josh McGrath, the urgent question is whether he can understand the system quickly enough to diagnose it. Whether an internal or external team owns the code matters less.

McGrath introduces himself in this conversation on post-training as an OpenAI researcher working on thinking models, recently focused on search. He moved from pre-training data curation into post-training for the opportunity to change model behavior. That choice was a statement about what excited him, not a declaration that pre-training was exhausted.

In reinforcement learning (RL), each task can bring its own grading infrastructure: a different setup for deciding whether the model succeeded. Those graders add moving parts and failure points to a training run. McGrath’s comparison with pre-training concerns this operational complexity; he does not quantify the difference.

Codex changes the rhythm of engineering work. McGrath describes spending 30–40 minutes preparing something like a design document, then having the agent complete more work in roughly 15 minutes than he could do in a few hours. In his personal experience, the awkward part is the wait: specification and execution now alternate in a way he is still learning to organize his day around.

Steering an agent during execution

The shopping model makes that execution period interactive. Users can see which products it is considering and add a requirement—such as needing USB-C—while it searches. McGrath compares this with interrupting Codex to correct a prompt. Requirements can change while the agent works, relieving users of having to specify every constraint at the outset.

The separate model lets the team experiment with a Deep Research-style shopping experience that searches extensively across the internet. McGrath sees eventual integration into a general model as possible, without committing to a schedule. He points to GPT-5 Thinking with high reasoning as an example of capabilities converging: he considers its published evaluations roughly on par with or better than the original Deep Research model and personally uses Thinking on High. That broad assessment comes without named evaluations or task coverage; it is not a claim that every part of the product experience is interchangeable.

Comparable capabilities still leave room for different interaction preferences. McGrath says users care about personality and describes work on personality choices and toggles. His own custom instructions ask for direct, work-focused answers. The host frames the distinction as Anton versus Clippy: - Anton: quiet execution. Concentrate on doing the work without adding cheerfulness. - Clippy: friendly assistance. Make warmth and an overtly helpful manner part of the experience. McGrath prefers the tool-like style, while supporting users’ ability to choose.

The reward matters as much as the optimizer

The training discussion turns on what the model is rewarded for. Reinforcement learning from human feedback (RLHF) uses a signal about human preferences; reinforcement learning with verifiable rewards (RLVR) uses checkable success criteria. The exchange treats both as policy-gradient approaches and places the consequential distinction in their training data and reward signals.

The comparison is between two kinds of assurance: - Human preference: a learned predictor estimates which response users would prefer. Preference can itself be the desired target, but the estimate still deserves scrutiny. - Verifiable correctness: checking whether an answer solves a polynomial offers a less debatable success criterion for that task. One participant distinguishes preference from truth; the response asks how cleanly and reliably each signal represents its intended target. The argument is about a spectrum of signal quality, with different reasons to trust different rewards.

Trust also depends on what happens under extensive optimization. A signal must remain useful as training pushes hard against it. The critique of contemporary papers is that gradient variance and other optimization details can crowd out questions about where data comes from, how well researchers understand the application, and how rewards behave under sustained training. An algorithmic improvement may make a cleaner academic story than that domain-specific work.

GRPO, the optimization method introduced in DeepSeek Math, illustrates the competing emphases. One participant recalls first seeing a somewhat cheaper method, then recognizing its broader industry impact. The other emphasizes the trustworthy math reward accompanying it: a correct answer gives training a clearer target. The important advance, in this view, lies partly in the task and signal that make optimization productive.

Measure capability alongside token use

For agents working over long sequences of actions, elapsed time is an incomplete measure: it can include substantial human delays. Token efficiency—how many tokens a model needs to achieve a level of performance—offers an actionable optimization target. It connects capability to the resources required to deliver it.

The reported transition from GPT-5 to GPT-5.1 illustrates the desired direction: some evaluation scores improved while the tokens needed to achieve the measured performance fell substantially. No benchmark names, scores, or numerical reductions accompany the claim. The evaluation principle is nevertheless clear: inspect performance and token consumption together.

Efficiency matters in two connected ways: - Shorter waits: token consumption affects how long a task takes to complete. - More activity within a serving budget: greater efficiency can make room for additional tool calls and other agent actions. Reducing token use can therefore expand how much useful work an agent can perform within practical resource limits.

Allocating those resources introduces an interface problem. GPT-5’s explicit router selects a model, while thinking-effort controls govern how much reasoning it does. One participant worries that a poor top-level choice could keep a task from reaching a model capable of solving it—a possible failure mode, rather than a demonstrated result. The response anticipates one tool that decides how long to think, without offering a merger plan or timeline.

Long context must support work across the window

Context compaction raises a related question about who controls the information kept available to a model. The questioner describes that responsibility moving from the engineer’s surrounding software into model-managed behavior. The response offers no compaction details or conclusion about whether the shift is permanent. Its broader position is that interfaces must remain open to change, so newly discovered model improvements can become usable capabilities.

Graph Walks, an evaluation used for GPT-4.1, sharpens the distinction between holding information and using it. Retrieving a fact from one location is easier than performing multiple transformations across information distributed throughout the context window. Graph Walks tests the latter. The speaker reports improving performance and expects further gains, without giving scores or a date for reliable use of the entire window.

A company-support example exposes the scale problem: roughly 100,000 documents totaling eight billion tokens, too much to fit in a context window at the time of the discussion. Search remains useful. One participant finds agents using grep surprisingly effective, much as simple BM25 and n-gram search indexes had proved difficult to outperform in earlier information-retrieval work.

Better search and larger context can complement each other: a larger task might use its window to support many subsidiary searches. The participants nevertheless differ on where to invest: - Systems and context utilization: one argues that video and scientific information could require vastly greater capacity, questioning whether full attention can scale far enough. Growth of 10–100× might be insufficient if a task needs 1,000–1,000,000× more capacity. - Larger windows: another wants to keep expanding context to discover what becomes possible, even when current windows already support useful systems. Those multipliers express the scale of the concern, not measured requirements or a forecast of achievable capacity.

The bottleneck moves between ML and systems

The suggestion that engineers favor systems while researchers favor models receives pushback. A participant describes OpenAI post-training as co-design: the same person may build systems, create Graph Walks tasks, and work on learning. That range matters because the limiting factor can change several times within one project.

The staffing argument is a personal assessment of the field: too few people want to do both systems engineering and machine learning (ML) research. The proposed education combines distributed systems and core engineering with statistics and ML research skills. The responding speaker studied mathematics, then learned software engineering through mentors—a combination they would like education to provide more routinely.

Asked which side agents might automate sooner, the response identifies different obstacles: - Conventional ML research: some tasks can be treated more like black boxes. - Systems work: constructing an environment in which an agent can train is itself a complicated engineering problem. Building the training environment is part of the automation problem. The speaker tentatively judges the overall difficulty roughly equal, with explicitly low confidence.

Progress can wait for a new way of working

The closing discussion credits the shopping team and describes post-training as an exciting, collaborative workplace. It then returns to investment priorities. One participant reports hearing researchers argue that too much money is going to post-training, and recalls a Grok 4 chart as showing comparable compute devoted to pre-training and post-training. That approximate recollection prompts a question about spending; it supplies no figures for a general comparison.

The response cautions against declaring either approach exhausted. Its analogy is factory electrification: factories initially retained layouts organized around a single motor, then eventually rearranged workstations to exploit electricity’s flexibility. The point is reorganizing work to realize a technology’s benefits. Apparently limited returns may precede a more consequential change in how the technology is used.

Experimenting with ideas, funding them, and spreading them still takes human time. The speakers expect uneven progress and repeated swings between pessimism and renewed optimism. Their closing advice is emotional steadiness through those cycles, followed by an invitation to keep providing feedback.