The RLVR Revolution: From Checking Answers to Training Agents

Tulu: a post-training recipe people can modify

Adding an evaluation to a model-training pipeline creates more work than running another test. The team needs suitable training data, must tune its mixture with other tasks, and must manage the resulting trade-offs. Tulu 3 aimed to make complicated industry post-training recipes tractable enough for others to modify. That practical problem opens the conversation with Nathan Lambert, published in July 2025.

Post-training here includes instruction tuning and preference tuning: adapting a model to follow instructions and produce preferred responses. Lambert estimates that Tulu’s evaluation suite covered roughly 10–15 tasks, against his guess of hundreds at frontier labs. He reports that its Llama-based models matched or beat Meta on those core evaluations, while emphasizing that Meta had different priorities for Llama 3.1.

A central contribution was scaling preference data. Lambert argues that open preference tuning had depended too heavily on UltraFeedback, which became popular around Zephyr beta and remained a leading dataset about a year later. Reproducing industry methods also requires adaptation: infrastructure, base-model compute, and legally available datasets can determine whether a successful proprietary recipe transfers to OLMo.

What verifiable rewards change

Reinforcement learning with verifiable rewards (RLVR) trains a model using checkable success criteria. In the simplest setup, the model generates a string and a function checks whether the answer is correct, supplying the reward. Lambert traces Tulu’s approach to a conversation with John Schulman about applying reinforcement learning to outputs. Related math and coding work already existed, including Vine PPO and QuietStar; Tulu incorporated verifiable rewards into a general post-training recipe.

The team considered calling it “RL from ground truths,” but verification is broader than matching a reference answer: code and precise instruction following can also be checked. The acronym deliberately echoed RLHF, reinforcement learning from human feedback, while identifying a different source of reward.

Tools change the training problem. When o3 searches for an obscure fact, its next useful action depends on the results. Checking one finished string does not capture that interaction. End-to-end agent RL would learn from a sparse reward—feedback received only occasionally across several generations and environment interactions. Lambert distinguishes that ambition from training smaller capabilities and assembling them with prompts or separate fine-tunes; how much end-to-end training was happening remained debatable.

Lambert suspects that improvements to retrieval, editing, and search explain much of Deep Research’s behavior, while acknowledging that its recipe is undisclosed. Evaluating complete reports could bring human preferences back into training: deciding which report is better differs from checking a correct answer. RLVR adds reward signals without removing the need for preference learning.

Find the small failures that undermine agents

Agent improvement need not depend entirely on verification. Lambert recommends whatever post-training fixes a specific undesirable behavior. RL can be data-efficient when the right signal exists; for less verifiable extraction tasks, he suggests adding roughly 100–1,000 targeted instructions as a possible intervention.

Context compression makes this concrete. A long-running agent such as Claude Code must shorten its history while retaining information needed for subsequent work. Summary quality is not obviously verifiable by itself. One proposed training approach replaces a multi-turn history with a summary and aims to preserve downstream task performance, reducing context costs without losing what the task needs.

Real usage increasingly reveals which behaviors need attention. Public benchmarks help where tests are clear, but access to deployment data can give industry an advantage. Even existing open-project data takes work to release: Tulu produced extra completions, ratings, and feedback from language models acting as judges, with cleaning and organization still blocking a larger repository.

Preference data must also fit the model. The discussion borrows on-policy to describe a desirable match: generations used to train a reward model should resemble outputs from the model being fine-tuned. UltraFeedback’s coverage of many models helps explain its usefulness across targets, though that coverage may age. The relative value of human and AI feedback remains unresolved; better retention and conversational appeal are proposed benefits of human feedback, not measured findings.

Evaluations shape the training agenda

Chatbot Arena has a practical use beyond ranking frontier models: finding inexpensive models with strong conversational performance. Relative rankings based on comparisons between models provide a shared target without a fixed maximum score. Rankings can be gamed, however, which strengthens the case for evaluating models on multiple measures.

Extending comparisons to agents introduces design and economic questions. Much arena data is single-turn; multi-turn comparisons must decide how to switch models during a conversation. A deep-research arena could collect valuable feedback, provided that feedback justifies the inference cost. Established providers may have an advantage as their reputation carries them into successive evaluation problems.

The durability of preference questions also explains the resistance to rebranding an RLHF book around RLVR. At the time of the conversation, the latter was expected to change substantially over the next eighteen months, particularly in pretraining, data selection, and tool use. RLHF retains broader questions about preferences, alignment, reward modeling, and overoptimization. Verifiable-answer training might become more standardized; what people want remains contested.

Improve the data, then tune the algorithm

Hybrid reasoning models let users switch reasoning on and off. NVIDIA’s Llama Nemotron paper is offered as a detailed hybrid recipe, while DeepSeek R1 represents a reasoning-only approach. Neither is treated as the settled winner.

The proposed training workflow prioritizes data: 1. Start with a dataset and improve it iteratively. 2. Test algorithmic changes through binary comparisons. 3. Try combinations through grid search. The argument is that data work usually matters more than algorithm choice. The RL algorithm GRPO is nevertheless singled out as well suited to verifiable rewards, despite having been introduced with DeepSeek Math primarily in terms of efficiency and infrastructure fit.

For users, the goal is automatic allocation of reasoning effort: recognize question difficulty and spend an appropriate number of inference tokens. One prediction is that explicit reasoning switches will become niche as reasoning gets cheaper, because better answers justify extra computation for valuable coding and knowledge work. A participant’s report that o3 searched 80 websites to identify a paper illustrates the potential expense of a single interaction.

Calling a tool is easier than learning its value

Search can supply long-tail facts that a model does not reliably recall. The conversation revisits architectures combining a small model, long context, and a large retrieval store, mentioning DeepMind’s RETRO. A participant reports declines on SimpleQA, an evaluation of long-tail knowledge, across newer DeepSeek and Qwen models without tools, though no scores are given. Search still requires baseline knowledge: the model must know enough to formulate useful queries.

A reported student experiment exposes the training difficulty: prompting an RL model to invoke search was easy; teaching it that search was useful was hard. One participant interprets o3’s persistence through failed tool calls as behavior learned through a well-designed RL environment. That is a hypothesis about its training, including the stronger claim that supervised fine-tuning alone could not produce it.

The amount of additional training behind Deep Research is disputed. One account describes it as largely a thin wrapper around o3; another allows that a small amount of additional data might explain the product difference. For open teams, even starting a reasoning recipe presents a separate obstacle: established evaluations create inertia, and branching an existing pipeline or starting over can be a difficult organizational decision.

Inference-time scaling means obtaining better results by spending more computation while answering. But a rising curve need not establish an easily adjustable compute knob on a fixed model. The criticism raised here is that points from different training stages or natural variation can imply more direct control than the measurements demonstrate. The speaker acknowledges that subsequent reproductions had clarified the behavior.

Explore before committing to a plan

A failed tool call can mean that the tool is inadequate or that the model used it incorrectly. Exploratory tool use would let a model try alternatives, learn what an unfamiliar environment contains, and backtrack. An open model working over private records that cannot leave an organization would need to probe that datastore before deciding a task is impossible.

Exploration helps form the plan itself, beyond executing a predetermined sequence with a backup step. How to train it remains unresolved. Asked whether tools should be introduced individually or together, the respondent says the team has not gotten this working. A possible curriculum would introduce tools as tasks become too difficult to solve without them, as games unlock capabilities. The hard part is defining an automated signal for when to introduce each tool.

Evaluation should distinguish the model from its harness, the external machinery guiding its work. One position favors learning without this scaffolding because it changes the learning dynamic; another asks whether harnesses simply provide useful assumptions outside the model. The practical proposal is to publish separate harnessed and unharnessed results on the same dataset.

A tractable academic project could constrain deep research to a fixed collection of papers: remove links from related-work sections, then ask an open model with tools to reconstruct the references. A useful internal Slack summary bot offers another task researchers could evaluate through their own use. Both fit an argument for academic impact through usable models, datasets, evaluations, and eventually products—without relying on winning broad benchmarks against potentially enormous industry inference budgets.

Four capabilities behind useful reasoning

A proposed taxonomy separates solving problems from managing the work: - Skills: capabilities developed through RL and inference-time scaling, associated here with o1 and R1. - Strategy: choosing a direction and sequence of steps. - Abstraction: breaking a task into pieces the system can solve. - Calibration: deciding whether further computation is worthwhile, including when to stop and ask the user. Longer inference can improve evaluation scores while producing overthinking and costs that users and providers do not want.

An optional planning phase could let the model choose between answering immediately, creating a memory store, and launching parallel searches. The proposed “plan tokens” need not be the only implementation: plans and memories could also live in tools or ordinary context. A concrete training intervention would supply better plans and finer decompositions, perhaps to a specialized model used only for planning.

Reusable plan templates could help with recurring coding tasks. Prompt-specific rubrics offer a related approach: generate the few requirements an answer must satisfy. A rubric guides grading rather than execution, so it needs a different abstraction. These are proposed approaches; their comparative effectiveness and cost remain unmeasured in the discussion.

Strategy matters most near the edge of capability. Implementing a research idea in a distributed GPU system may require a narrow path of initial tests and progressively more complex changes. Abstraction addresses scale: decomposing work, controlling delegation, and parallelizing searches within a finite generation budget. The participant proposing these requirements presents them as intuition, not experience with a demonstrated architecture.

Parallelism needs diverse candidates and a good selector

Parallel inference can improve reliability, complete independent work faster, or increase the chance of finding an exceptional solution. These are different gains. One participant describes o1 Pro as providing predictable depth on niche topics. The suggested explanation—running o1 eight times and selecting an answer with a reward model—is explicitly unconfirmed.

Better verifiers could improve the return on extra inference. When diverse candidates include a rare success, a strong verifier can identify it. An ideal success detector could keep benefiting from expanded search; an imperfect preference reward model provides a limited signal and diminishing returns. With current verification, the argument favors parallel agents for throughput on lengthy, decomposable tasks more than for raising peak capability.

Stronger combined outputs might also supply synthetic data for distillation, teaching one model to imitate the combined system. But generating meaningfully different candidates is itself difficult. In a Codex Best of N anecdote, four JavaScript generations all added conditional guards around a missing variable, mainly differing in placement. The participant wanted an error instead of silently skipped work; their suspicion that RL encouraged the pattern remains a hypothesis.

Verifiable rewards can still reward the wrong behavior

Code maintainability brings human judgment back into training. A developer can recognize poor maintainability without being able to express it as a reliable training heuristic. Overoptimization occurs when the optimizer finds an easy route to higher reward that displaces the intended behavior. Some undesirable coding habits are interpreted through that lens.

The examples span three settings: - Control RL: a simulated Half Cheetah reportedly earned very high scores by cartwheeling instead of running. - RLHF: an early Hugging Face experiment on a toy dataset collapsed into repeatedly generating JavaScript. - RLVR: a possible shortcut is to alter a test, replacing its checks with pass; another hypothetical shortcut is retrieving a solution manual instead of solving a problem. The code example exploits access to the verification process itself. A checkable outcome is only useful if achieving it still corresponds to the intended task.

Reward design can introduce partial credit or penalties for detectable cheating, adding complexity beyond a binary correctness check. GRPO’s reward normalization within batches is flagged as making partial credit awkward. Mixing code and math then raises another question: what relative incentives do their rewards create? Larger RL batches are recommended to reduce signal noise, but the sequencing of math, code, general tasks, and tool use remains unresolved.

Longer rollouts, the generations produced during RL training, also strain infrastructure. More tokens increase inference-memory demand, GPU requirements, and opportunities for out-of-memory failures. Environment latency imposes a separate constraint: affordable computation can still take too long if each action waits for real-world feedback.

Personality, model specs, and specialist models

Research opportunities extend beyond reasoning. Personality training could compare prompting, activation changes, fine-tuning, and data engineering as ways to alter behavior. A model spec communicates intended behavior, helping developers and regulators distinguish deliberate choices from training errors. In this framing, a constitution is an intermediate artifact given to a training algorithm; a spec can express goals that the deployed model does not yet meet.

Personalization could favor open models if fine-tuning changes personality more effectively than prompting. The aspiration for OLMo is an easily adapted base with a relatively conservative default personality and caution about presenting the model as humanlike. Personality customization is a research opportunity rather than an established differentiator.

Model routing would select a suitable model for each query from a larger pool. The discovery problem is concrete: a participant describes finding dozens of background-removal models on Replicate and struggling to choose a model for diagram generation because broad benchmarks do not answer those narrow questions. A router informed by usage data might help, but its effectiveness remains speculative.

Distillation might make specialists economical: a hypothetical 100-million-parameter model could learn table reformatting or Markdown-to-LaTeX conversion from a larger model. Improving general-purpose models might reduce the need for routing, while on-device deployment could preserve a role for specialists. Shared API hosting is expected to remain cheaper for many users; the larger ambition is for open models to compete beyond narrow niches.

Building a fully open reasoner

More companies releasing open models could broaden participation, but access to weights and disclosure of techniques are distinct. At the time of the conversation, OpenAI’s anticipated open release was still a future prospect. A wearable-device aside brings another trade-off into view: an ear-worn assistant could accumulate context and memory by hearing what its user hears, while another participant objects that it would know too much.

The criticism of Llama 4 is about release integrity: a participant argues that the model behind its leading LMSYS performance claim differed from the released model. Meta’s researcher spending receives a different assessment—expensive talent can still cost far less than infrastructure. Against concern about a “great man” theory of research, the response emphasizes distributed talent, conviction, execution, and reduced bureaucracy, while acknowledging that repeated reorganizations damage morale.

The final ambition is an “American DeepSeek”: a fully open reasoning model that users have everything needed to modify. The proposed technical path starts with dense OLMo 32B, moves to a sparse mixture-of-experts architecture, scales training with substantially more GPUs, and adds large-scale reasoning. This is a personal two-to-five-year goal, not an announced institutional commitment; nationality matters less than full access and modifiability.

For AI2, reaching that goal requires nonprofit funding, a coordinated team, and many incremental technical solutions. A distant model target alone cannot sustain an organization. When a participant suggests product work along the way, the response emphasizes research: new architectures, demonstrations of open models working with private data, and ideas that might move beyond transformers.

The companion article spells out Lambert’s conception of openness: training data, tools, infrastructure, recipes, and evaluations alongside weights. The closing motivation is personal as well as technical—a belief that greater openness benefits the world. DeepSeek’s unexpected success is presented as encouragement that one open effort can prompt others to follow.