Alex Zhang on What Agent Harnesses Can Teach a Language Model

A fast kernel is not a fast model

A GPU kernel can win a benchmark and still disappoint in the system that uses it. Suppose one operation runs fastest when it writes its result to high-bandwidth memory. A slightly slower first operation might leave that result in cache, letting the next operation start sooner. The relevant speed is the whole computation, not just the isolated kernel.

In the Latent Space interview, MIT researcher Alex Zhang uses that distinction to explain why GPU optimization needs both a performance target and an evaluation setting. A “speed-of-light” estimate depends on where the data starts and assumptions about overlapping computation and transfers. He cautions that theoretical limits may not be attainable and that many actual kernels remain far from them.

Zhang says many recent GPU Mode leaderboard solutions are AI-generated. In his example, a skilled community member also used AI, but steered the search; Zhang reports that this person’s solution was unusually stable in an end-to-end system among the top entries. He connects this to a verification gap: passing a narrow benchmark does not by itself establish robust performance in the larger application.

The same expert can narrow a costly search. Zhang suggests that knowledge about a problem can reveal a direction that otherwise consumes vast model compute; he presents the comparison as an intuition, not a measured token-saving result. His route into the area included work with the SWE-bench team at Princeton, GPU Mode, and KernelBench’s question of whether language models can automate GPU kernel code.

The harness is part of the computation

Zhang asks a similar systems question of language models. A model asked to fix a repository cannot usually inspect the whole codebase and solve the task in one call. A harness is the surrounding program that gives it tools, manages state and sequences calls. Its design shapes which pieces of a difficult task the model actually sees.

His Recursive Language Model, or RLM, is a harness design rather than a new transformer architecture. In his description, the model writes code in an execution environment; tools, including calls to subagents, are exposed as code functions. The larger context is stored in memory accessible to that environment, such as files, so the model can retrieve or process portions instead of placing everything into one prompt.

Consider a long collection of documents and a question requiring information from many of them. An RLM-style program can keep the collection outside the immediate prompt, select pieces, send smaller questions to subagents, and combine their returns in code. The selection, subanswers and aggregation each have to work.

The original motivation was long context, Zhang says, but his broader question is compositional generalization. His account of a training experiment is that a high-level program learned on a shorter task can transfer to a longer one when the strategy is effectively the same and only a length variable changes. The subagents still solve local problems; an appropriate decomposition is what makes the larger problem resemble the one seen in training.

He also discusses retrieval and aggregation tasks whose domains differ but whose higher-level procedures may have the same form. A program might generate candidates, delegate checks, and iterate against a verifier, while the details inside each subtask differ. Zhang treats broader transfer across domains as a research opportunity; the interview does not establish that every unfamiliar problem can be decomposed into reliable in-distribution calls.

An implementation and its tradeoffs

Prime Agent applies some of these ideas on top of the Pi harness. Zhang says it exposes IPython as its single direct tool and loads other capabilities as Python modules or scripts. It also retains context on disk, although it still uses a more conventional trajectory and compaction loop rather than offloading everything.

The design includes agent-to-agent communication and persistent subagents that can outlast the original agent’s runtime and receive further prompts. Zhang credits a separate continual-harness idea to Seth, another PhD student: it lets parts of the harness, including skills and available subagents, be modified. Zhang says he collaborated with Prime on the harness but is not involved in its model-training work.

Offloading and delegation are not free. Zhang says compaction can be faster and cheaper even where an RLM might solve a harder task. He also observes that current models are not necessarily optimized for RLM workflows, and describes some model comparisons as internal results rather than public evidence. The practical choice depends on the task, latency, cost and whether the added coordination improves the answer.

Search at swarm scale

A swarm extends the same question: how do many agents share discoveries without spending more than the problem warrants? Zhang’s account of an OpenAI effort is that agents used shared context to search a difficult problem. He is explicit that he does not know all the details, including the role of researchers’ intuition, and does not claim the system was necessarily an RLM.

He guesses that much swarm exploration may be unproductive and asks where to draw the line between a coordinated workflow and a large search. A small amount of wasted search may be the price of finding an unexpected route. For open-ended research, another hard problem follows generation: how to find the useful result when there is no simple fixed objective to rank everything.

He suggests one latency optimization for code-driven tools: begin independent tool work while code is still being generated when it can be predicted safely. He describes speculative programmatic tool calling as an idea to explore.

Research bets before there is a product

Zhang argues that academic researchers should use their relative freedom to investigate questions that initially appear trivial or unimportant. He recalls skepticism toward RLMs and points to SWE-bench as an example of a benchmark that became more salient when systems appeared capable of attempting the task. A simple idea can matter because it opens a different set of questions; visibility or early enthusiasm alone does not prove the idea works.

One such question is whether a language model must always produce a sequence of text. Zhang sees fast classifiers as a way to change the output space for a narrower task, potentially avoiding the latency of a full generative call. He says he does not know the training details of the system discussed, and does not suggest that a classifier replaces a general model for every problem.

His longer-term bet is that a simple interface could eventually conceal a richer system of models and agents. He also says today’s frontier models do not reliably perform a job over a month, while suggesting that better harnesses might close some of that gap. That remains a research hypothesis. In empirical science, he adds, feedback loops can be long, so choosing where to invest effort is itself a difficult decision.