From reasoning benchmarks to the whole software engineering job

A robot folds the laundry. What has it learned?

Ashvin recalls watching Physical Intelligence robots fold laundry in someone’s living room, including clothes tossed into the setup. The demonstration impressed him. His criterion for a robotics inflection, though, is generalization to a previously unseen situation. Seeing a task performed is encouraging; the details determine whether it demonstrates that broader ability.

Ashvin interned on robotics at OpenAI in 2017, before his PhD, then spent his doctoral work taking ideas from simulation to real robots. He sees a transferable discipline between robotics and large language models (LLMs): inspect the data, understand failures, and persist through the difficulty of making systems work.

He left robotics in 2022 expecting to return. By this conversation, he predicted that LLM agents could become a trillion-dollar market before AI robotics reached even $10 billion. His reasoning rests on two successive hurdles: establish a useful application, then make its economics work. Reliability, repairs, and maintenance remain difficult even after a capability is demonstrated. He qualifies this forecast as the perspective of someone no longer working closely in robotics.

Competition gold covers only part of the job

The guest recalls joining OpenAI’s Code Gen team around September 2022. The team had shipped Codex and was working on tool use and supervised fine-tuning for programming competitions. ChatGPT’s launch prompted substantial refocusing. At the time, the guest imagined that an AI earning gold in the IOI programming competition would mean AI was effectively solved. The later gap between competition performance and everyday working life became a persistent puzzle.

The host questions the original criterion: a competition medal was an inadequate test of artificial general intelligence (AGI). The guest emphasizes community-wide selection around benchmarks. This need not involve cheating. Researchers make many individually reasonable decisions about what to optimize, and those choices collectively steer progress toward prominent evaluations. Most programmers cannot perform well at IOI, yet automating their jobs remains difficult: competition skill captures only part of software work.

The guest connects this to reinforcement learning (RL) research during a Berkeley PhD with Sergey Levine, roughly 2017–2022. In their retrospective diagnosis, elaborate methods introduced more tunable choices, which researchers implicitly adjusted against familiar benchmarks. Selection across the literature compounded the effect. Academic incentives favored novel mathematical ideas, while simpler methods with fewer choices could transfer more reliably. The guest remains uncertain about why the period’s favored approaches failed to deliver the expected practical progress.

The diagnosis is uneven generalization, not an end to scaling. In the guest’s account, current RL can make language models exceptionally capable on their training distribution—the kinds of tasks they train on—while transferring only partially elsewhere. If RL remains the main improvement tool, economically useful work needs to become part of that distribution.

The missing context may be a year of experiments

Product and model must be designed together: the product exposes the context required for a job, and training teaches the model to use it. Coding offers a relatively convenient starting point because much of its context sits in a repository and terminal. Other jobs depend on dispersed documents, coworker conversations, and Slack messages. Even coding, the guest cautions, remains far from fully automated.

The host introduces GDP eval, an evaluation that assembles context for economically significant white-collar tasks. The guest provisionally agrees with its direction, having not examined its traces closely. Both favor original working materials, including PDFs, without artificial cleanup that removes difficulties a deployed system would face.

The guest’s own research makes the missing context concrete. At OpenAI, they studied how RL hyperparameters—adjustable training choices—affect scaling behavior. The work was analogous to studying how layer initialization affects deep-learning scaling, rather than the grid or architecture search suggested by the interviewer. Writing code was not the bottleneck. Over roughly a year, the researcher ran sweeps, inspected graphs, and accumulated an understanding of interactions between those choices.

Much of that understanding lived in the researcher’s head. Automating the work would require access to the experimental history and the knowledge built from it. The proposed sequence is to capture that context in a product, then train the model to use it. Better code generation alone would leave the central difficulty untouched.

This raises two competing explanations for specialized models: - Limits of general models: the host interprets separate GPT-5 and GPT-5 Codex offerings as evidence against a one-model-fits-all strategy. - Organizational priorities and data: the guest argues that product splits can reflect reorganizations and uneven access to domain data. A coding-focused organization may lack the material needed to train equally well on other work. The guest considers cross-domain benefits from broader training plausible. Product splits alone do not settle the scientific question.

Governance remained unresolved through the leadership crisis

The guest recalls learning of Sam Altman’s firing while coworking with two OpenAI friends during a Thanksgiving gathering. Initial disbelief gave way to a weekend of changing expectations; they also signed the letter discussed by the interviewer. Their enduring concern is how advanced AI should be governed, whether its arrival is two years away or ten. They recall being willing to set aside equity considerations to discuss that question.

The guest tentatively wonders whether a Microsoft-style corporate board, representing investors including pension funds, could offer broader stakeholder representation than a small nonprofit board. They do not endorse it as a solution: market incentives also fail to guarantee good societal outcomes, with unhealthy food and social media offered as examples.

Small reasoning models made the case for scaling

The account of o1’s origins begins with a longstanding conviction that RL could develop intelligence beyond copying internet data. The respondent credits Ilya Sutskever and Jakob Pachocki with helping establish that commitment, drawing a connection to OpenAI’s earlier Dota work. In this account, RL began paying off once pretraining became good enough. The respondent distinguishes that ambition from human-feedback training, which they characterize primarily as eliciting better behavior with less scope for continued gains from additional compute.

Around 2023, small reasoning models began producing unusually accurate reasoning traces for their scale. They did not dramatically outperform existing models overall, but an internal precursor to o1 combined those traces with surprisingly strong math performance. The respondent recalls that comparable gains would otherwise have required substantially more pretraining. That qualitative signal attracted more resources for scaling and additions such as tool use; no numerical compute comparison accompanies the recollection.

A visible research core understates the work required to ship. Asked about a dozen people on the original project, the respondent estimates roughly 50–100 contributors when counting everyone involved. They do not confirm the suggested 300-person figure for later reasoning work: safety, evaluation, and other contributions expanded as o3 became a product, and they had lost track of the total.

Smooth progress, volatile expectations

Inside the lab, progress felt comparatively smooth: experimental improvements accumulated, successive runs improved, and scaling continued. Public releases made that process look like a sequence of leaps. Competitive pressure also encouraged faster releases; the host estimates that the gap between internal availability and public release had narrowed to roughly one or two months.

The respondent recalls that by perhaps early 2024, strong math and programming competition performance seemed like a predictable continuation of the reasoning recipe. At The Curve, before o1’s release, they remember hearing forecasts for difficult math and exam tasks that internal models had already exceeded. The recollection lacks the evaluation details needed for a precise benchmark comparison. Its lesson is about expectations: even ambitious forecasters could underestimate near-term capability.

Both participants value registering predictions before outcomes are known, rather than claiming afterward that progress was obvious. The guest expects human-level intelligence around 2030. The host questions the tendency to place transformative developments within one’s own lifetime, while allowing that this forecast could prove right.

DeepSeek prompts competing interpretations of NVIDIA’s stock decline: - Chips became more useful: the guest sees DeepSeek as demonstrating greater value from existing hardware, making the decline seem counterintuitive. - The newest chips might be less necessary: the host suggests that competitive results on older or export-constrained hardware could weaken demand for the highest-end products. These are explanations of the market reaction, not measured hardware comparisons. The guest recalls little internal disruption at OpenAI and confidence that pursuing smarter frontier models remained valuable. Their broader impression is that labs were converging on similar forms of RL and roughly comparable outcomes.

Cursor brings deployment tasks into training

At Cursor, building models internally offers both less dependence on external labs and the ability to change product features and model behavior together. The guest sees close collaboration between product and ML teams as a way to bring the tasks encountered in deployment into the training distribution.

Tab’s reported two-hour policy updates make that opportunity concrete: Cursor frequently updates the model behavior behind its autocomplete product. The guest attributes the cadence largely to a small, focused organization with close product–ML collaboration. The host emphasizes the smaller, less complex autocomplete model. The conversation leaves their relative contributions unresolved; the reported cadence applies to Tab, not Composer.

Learning from user activity raises a separate question: how should training handle bad examples? - Imitation can dilute quality. The host worries that uncurated activity could pull performance toward the middle of the distribution. - Filtering discards the example. It removes undesirable behavior from what the model learns to copy. - Negative evidence teaches avoidance. The guest uses watching someone touch a hot stove as an example: a value function, which assesses how good or bad an outcome is, could make the mistake useful without encouraging imitation. The desired advance is durable learning from experience. The guest points to coding models repeating bugs—even within the same context—as a sign of the remaining gap.

Composer’s speed keeps engineers in the loop

The respondent clarifies that their own Cursor work has been on Composer, and that improving it is the ML group’s main focus. They describe a group of roughly 20–25 people and attribute Composer’s appeal to being capable enough to use, fast enough to stay engaged. Long waits encourage programmers to switch tasks; fast responses can preserve continuity of thought. This is a product judgment rather than a measured latency comparison.

Cursor’s co-founders are described as active, discerning testers who help judge whether the coding experience is ready to ship. The ambition beyond Composer 1’s prompt answering is a longer engineering loop: 1. Write code. 2. Inspect its behavior, including what appears in Datadog. 3. Form a hypothesis about an improvement. 4. Rerun the work to test it. This remains a target capability. Bringing more of that process into Cursor would give training access to more of the engineer’s actual job.

Understanding the data includes reproducing failures. Internal tooling lets staff connect to their own user environments and inspect whether code runs as experienced. A coding product also gives its developers an evaluative advantage: they can assess many failures directly, whereas a biology question might fall outside their expertise.

Continual learning should make mistakes useful

The closing technical discussion returns to continual learning: retaining lessons from deployment beyond the current context. The proposed behavior resembles learning from the current conversation or task, but with lasting memory: experience would enter the model’s weights so it need not repeat the same mistake. The guest sees this as a promising research direction, potentially accessible to academia.

The host raises finite capacity and forgetting. The response compares thousands or millions of deployment tokens with trillions of training tokens and argues that the additional experience should not necessarily overload the model. This is a capacity intuition, not a demonstrated retention mechanism. Years of accumulated experience could complicate the argument.

Two analogies expose different parts of the question: - Weights as a hard drive: how much information can the parameters store, and how much capacity is already occupied? - Weights as a processor: could a relatively small set of learned circuits perform substantial computation, rather than the network chiefly memorizing content? The host proposes tracking storage capacity; the guest recalls the contrasting framing from a conversation at Cursor. These are ways to investigate what weights do, rather than a settled explanation of how to retain deployment experience.

The guest would like to pursue these fundamental questions but sees a tension with empirical product improvement: deeper understanding may offer less immediate payoff, even over the next couple of years. Academia could investigate them, though access to sufficient compute remains a constraint.

The interview closes with two-day work trials as a more representative hiring test than conventional interview questions. Asked for an RL question anyway, the respondent suggests explaining why off-policy RL is unstable, while cautioning that it may test familiarity with the 2017–2022 research era. The mechanism is left for further study. Cursor’s recruiting appeal at the time of the conversation emphasizes a concrete need: people working on data and rewards for code.