RLHF 201: From Preference Labels to Model Behavior

RLHF works—but whose economics apply?

The Llama 2 researchers were wary of reinforcement learning’s instability, yet found it effective in both cost and time. Nathan Lambert begins the technical discussion with that reversal, then questions its portability: efficiency at Meta’s scale does not establish efficiency for a smaller team. The comparison lacks a resource baseline, and Meta had substantial staffing and funding.

Reinforcement learning from human feedback (RLHF) uses judgments about model outputs to guide changes in model behavior. It can substantially change a language model, but connecting particular feedback to the eventual behavior remains difficult. This conversation with Nathan Lambert, published January 11, 2024, explores that gap as major companies build dedicated RLHF teams and open researchers work with fewer resources.

Lambert treats reinforcement learning as a framework for learning decisions through trial, error, feedback, and consequences. Moving from control theory’s costs to reinforcement learning’s rewards, then to human preferences, changes the assumptions. Optimizing a reward is one problem; deciding whether people’s judgments can be measured and aggregated into a useful reward is another.

A learned scorer replaces ongoing human interaction

A conventional reinforcement-learning agent takes an action, receives a new state and reward from its environment, and repeats. Its policy, which determines how it acts, learns to increase rewards. In the language-model setup here, the prompt supplies the state, an entire completion counts as one action, and a reward model scores it. Training then moves to another prompt. This is a bandit formulation: an action receives feedback without the continuing environmental transitions of a longer interaction.

The reward model makes human feedback reusable during optimization. Directly generating tens of thousands of responses, showing them to people, collecting reactions, and feeding those reactions back into training would require substantial infrastructure. Process reward models provide finer feedback by scoring individual reasoning steps, but that granularity alone does not create an interactive environment.

Calling the scorer a preference model can imply more than it establishes. A robot’s dynamics model is intended to mirror physical behavior; a preference scorer has no comparably clear account of what a preference is. Claude’s constitution makes training priorities explicit, but training a model to prioritize principles does not establish that it believes them. Whether specified values reliably become model behavior remains unresolved.

Comparing outcomes can be easier than producing them

Human-feedback learning predates chatbots. The history begins with TAMER, dated in the conversation to 2008: people score an agent’s actions, and a learned model of those scores guides its policy. Paul Christiano and coauthors’ 2017 work, Deep Reinforcement Learning from Human Preferences, instead asks people to compare pairs of agent trajectories. That structure anticipates the chosen and rejected completions used in language-model training.

The 2017 work is recalled as achieving higher rewards in some simulated control environments and learning faster relative to the number of environment interactions. One proposed explanation is that trajectory comparisons give humans richer information than a single state supplies. That interpretation does not establish a universal advantage for preference feedback.

For language, the useful premise is evaluation can be easier than generation. Someone who struggles to write a good poem may still choose between two generated poems. A comparison can express attraction to a good answer or rejection of a bad one. Safety training uses the same mechanism when a benign response is preferred over a harmful response, with harm defined by the organization collecting the data.

Language-model RLHF work in 2019 led into OpenAI’s 2020 summarization experiments. Long Reddit posts, including explanations and career advice, supplied inputs for comparing ordinary model outputs with trained summaries. The remembered change was qualitative: repetitive baseline text gave way to much better summaries, within that restricted task. The summarization and later InstructGPT work support a practical habit: read the outputs alongside the plots. Aggregate scores can miss how different a model feels to use.

Start with instruction tuning, then constrain optimization

For most practitioners, the recommendation is to start with instruction tuning: train on examples of the behavior you want. Hand-written examples, modest compute, and a straightforward loss can accomplish many customization goals. RLHF becomes more compelling when preference optimization could create a specific competitive advantage and the team can afford uncertain experimentation.

Vicuna demonstrated how suitable instruction data could transform conversational behavior, though claims of ChatGPT equivalence drew pushback. Multi-turn conversation does not itself require RLHF: even a little training data containing several turns can help. Model size also matters. The discussion attributes greater nuance, coherence, and robustness when moving from roughly 7 billion to 70 billion parameters partly to instruction tuning and scale alone.

The conventional workflow follows a dependency order: 1. Instruction-tune a useful starting model. Formatted question–answer pairs teach response behavior. Chat templates establish conversational structure, including system prompts and requested personas. 2. Collect preferences and train a scorer. Generate candidate responses from an instruction-following model and collect judgments about which are better. 3. Optimize against the learned reward. Retain the chat template and account for how closely the preference data matches the starting model’s distribution.

The objective is to increase learned reward while limiting departure from a reference model. A divergence constraint, called KL in the discussion, penalizes changes in the output distribution; token log probabilities provide quantities used to estimate that divergence. This limits overfitting to incidental features in a small preference dataset, such as punctuation, unusual tokens, or favored formatting. Its importance is presented as practitioner experience, without a controlled comparison establishing that optimization always fails without it.

A winning response may still be bad training data

The Bradley–Terry model connects pairwise judgments to reward scores: annotators choose between completions, and the model relates the probability of that choice to their rewards. Direct zero-to-ten ratings are described as having been less successful at producing ChatGPT-like behavior than pairwise training. This is a practical observation, not a controlled result showing that ratings generally cannot work.

Annotators receive extensive instructions ranking factuality, helpfulness, honesty, and harmlessness, with worked examples. Organizations rank those qualities differently. When a participant challenges preference aggregation, the response argues that practical collection concentrates on correctness and style while avoiding contentious preferences. Even so, an annotation policy does not prove the trained model exhibits its intended qualities. Testing honesty and attributing behavior to particular instructions remain difficult.

A labeling interface may show two responses and ask which is better. One example uses a 1–8 scale: 1 strongly favors A, 8 strongly favors B, and 4–5 indicate near ties. Behind that simple control are lengthy reading tasks, workload allocation, repeated judgments, and disagreement tracking.

Relative preference also conceals absolute quality: - Good versus good: potentially informative distinctions between useful answers. - Good versus bad: a clear reason to prefer the useful response. - Bad versus bad: a winner that may still be unsuitable training material. A rushed Hugging Face collection effort reportedly produced enough bad–bad comparisons that the team was unsure whether the data was usable.

Quality metadata can help separate these cases. InstructGPT’s fields included task failures, hallucinations, dangerous content, moral judgments, and expressed opinions. Such flags make an ablation possible—remove a category of data, retrain, and measure the change. Removing hallucination-flagged examples is proposed as an experiment, with no reported outcome. Richer annotation also costs more, because each field adds work and collection costs scale roughly with task duration.

Safety training and deployment filtering have different roles. Llama 2 is cited as using helpfulness and safety reward models during training. A separate output classifier could filter responses at deployment. The suggestion that major services use such classifiers, and that much smaller models could filter better than RLHF alone, is an expectation rather than a confirmed implementation or measured comparison.

Feedback costs money—and may lose usefulness

A correction to an earlier Llama 2 spending estimate exposes an accounting trap: prompts and labeled turns are different units. Each turn in a multi-turn conversation can count as a data point, so the cited 1.5 million data points could represent roughly 400,000 prompts. The revised estimate was $6–8 million, possibly more with other purchases and discarded data; this was not disclosed Meta spending. Similarly, a published compute figure for one run omits the other experiments needed to develop a model.

Reuse depends partly on how closely the data matches the model. The discussion calls comparisons originating from the instruction-tuned model on-policy preference data. Fresh comparisons may be needed for continued improvement with proximal policy optimization (PPO), the reinforcement-learning optimizer examined next. Early batches may cease to expose useful distinctions as the model improves. The limits of transfer across models and the rate of this decline remain unknown.

Synthetic preferences offer a cost advantage: GPT-4 is presented as a source of reasonably good data at low cost relative to engineering salaries and compute. A hypothetical comparison—$10 for a human data point versus one-tenth of a cent for a generated one—illustrates the incentive, rather than reporting measured prices. Human-written instructions are judged especially valuable, while the weaker signal attributed to human preference judgments makes those judgments attractive to automate.

That incentive comes with terms-of-use and contamination concerns when large volumes of model outputs enter training workflows. At recording time, a participant cited reports that ByteDance had lost OpenAI access after training on GPT-4 data. Broader use of Constitutional AI for generating preferences was still a prediction for 2024.

LMSYS’s arena offered another potential source of human feedback, but access to prompts is different from access to preference labels. A provider serving an arena endpoint can receive its prompts; that does not establish access to users’ comparisons. Limited staffing, safety processing, and legal concerns complicated preference-data releases.

Train the scorer, then debug the flow of rewards

A reward model modifies a language model to output a single scalar score. Chosen and rejected examples pass through the same model, typically in a batch, and the loss updates shared parameters to favor a higher score for the chosen completion. Reward-model agreement measures how often that ordering matches held-out preference labels. Roughly 65–75% is described as common, without specified datasets. This measures ordering agreement, not absolute answer quality.

Alternative losses, ensembles, and different models are being explored to improve agreement, without an expectation of eliminating all disagreement. Optimization nevertheless proceeds: the language model generates a completion, the reward model scores it, and reinforcement learning uses that score to update the language model. Subtracting a divergence penalty makes the effective reward combine preference scoring with a constraint on model change.

PPO requires coordinating several models and updates: the trainable policy, a reference model for the divergence constraint, reward scoring, and a learned value function used in optimization. Understanding the components is easier than making the combined system produce a good chatbot. Practical diagnosis should inspect the reward signal, the actual generated tokens, and how outputs change during training. Data quality, infrastructure, and signal flow deserve attention before fine distinctions among objective terms.

Behavioral change does not guarantee capability gains. Public evaluations in the GPT-4 technical report showed mixed effects from RLHF: some metrics improved and others declined. Multiple-choice tests may poorly capture the behavior preference tuning targets, while stronger safety refusals may change a model in ways some users dislike.

Use reward models for selection and more specific feedback

Rewards can guide model use and training without a PPO loop: - Best-of-n sampling: generate multiple responses—10–20 in the example—score them, and return the highest-scoring answer. Extra inference compute buys a chance of a better response. - Rejection sampling for training: retain highly ranked responses and instruction-tune on them. Then optionally collect more preferences, retrain the scorer, and repeat selection. - Offline learning: use an existing dataset to reduce generation and scoring costs during optimization.

Llama 2 reportedly began its RLHF process with rejection sampling. Selected responses can use the usual autoregressive training loss and existing instruction-tuning infrastructure, making this easier to implement than PPO. Offline reinforcement learning is raised as another direction, but its brief explanation does not specify an algorithm.

Feedback can also become more specific through written critiques, multiple scores, multiple comparisons, or supervision on individual reasoning steps. OpenAI’s Let’s Verify Step by Step is the process-reward example discussed. Choose feedback to suit the domain: stepwise reasoning supervision appears well suited to mathematics, while its usefulness for poetry is questioned.

Constitutional AI carries principles through training data

Constitutional AI changes how instruction and preference data are produced. The published approach from around 2022 has two stages: 1. Critique and revise responses using principles. Model-generated critiques guide improved instruction responses for fine-tuning. 2. Generate preference judgments using principles. Give a model two completions and a sampled constitutional principle, then ask which completion is preferable. Save the preference pair without the principle context.

Principles influence behavior statistically through the data. Sampling principles across many prompts carries their influence into critiques and preferences; it does not provide an explicit checklist proving that every answer satisfies every principle. This makes the link between training priorities and model behavior an empirical question.

That question also requires evaluating intermediate checkpoints. A strong optimizer can produce poorly understood changes between a run’s endpoints, motivating evaluations every epoch or half epoch. A subsequent exchange connects AI feedback with the problem of controlling systems that exceed human capabilities, but leaves the technical relationship to weak-to-strong generalization unresolved.

DPO simplifies training while changing its controls

Direct preference optimization (DPO) starts with the same kind of chosen–rejected data but uses an implicit reward instead of a separately trained scalar scorer. That reward depends on the policy’s generation likelihood relative to a reference model; differences between log-probability ratios supply the preference-training signal. DPO belongs under the broad RLHF umbrella while remaining a distinct algorithm family.

Simpler implementation helps explain DPO’s open-model adoption, but the tradeoff concerns control: - Separate reward and policy training allows different prompt distributions and objectives at each stage. - DPO couples preference learning to the policy update, removing that separation. The separation might matter at very large scale, though its importance remains unknown and may be irrelevant to many practitioners. Claims that PPO has a higher performance ceiling are explicitly rumors, without comparative measurements.

Code execution makes the distinction concrete. Whether generated code runs can directly supply a reward in a PPO-style loop; DPO requires additional work to construct suitable preference data. At recording time, reported DPO successes were concentrated on UltraFeedback, with improvements on AlpacaEval, MT-Bench, TruthfulQA, and qualitative behavior. A successful dataset–method combination does not isolate the method’s contribution.

[Tulu 2](https://blog.allenai.org/camels-in-a-changing-climate-enhancing-lm-adaptation-with-tulu-2-0eb692698a78) tested the Zephyr recipe at 70 billion parameters. The team used Hamish Ivison’s JAX DPO implementation and research-access TPUs. The released model reportedly came from the first run, using the same data and copied training settings, with no initial ablations. It performed well on benchmarks and in use according to the account, though no scores were supplied. Data ablations were underway to identify which choices mattered.

Tulu 2 fit the Allen Institute’s expansion toward model releases alongside papers and policy work. DPO releases were also becoming routine elsewhere. Adoption can outpace comparative understanding: a method can dominate published releases while its tradeoffs against other optimizers remain unsettled.

Chat evaluations establish a range, not complete usefulness

The Hugging Face leaderboard began as a way to automate evaluation and centralize comparable scores. It helps people discover models, but developers can overfit to its tests and mistake leaderboard success for overall quality. Broader evaluation also needs funding. Direct interaction remains part of model evaluation, even when it feels peripheral to an engineering or research task.

Model versions matter when evaluating changing services. In the Chatbot Arena comparison discussed, GPT-4 March 14 scored about 40 Elo points above GPT-4 June 13, a difference described as outside the error bars. The evaluators and their prompts were unknown, limiting the comparison. GPT-4 Turbo was also reported ahead of earlier versions. The suggestion that preference improved while conventional benchmark scores stayed similar remained conjecture.

Two chat evaluations use GPT-4 feedback with different procedures: - MT-Bench tests two-turn responses. It asks a prompt and a follow-up question, scores both responses, and averages the scores. Its custom prompts reflect its designers’ view of good chat behavior. - AlpacaEval measures a judged win rate. In the setup discussed, GPT-4 compares a candidate model’s responses against DaVinci 003, an older OpenAI instruction model. Its larger prompt collection draws from existing instruction and assistant datasets.

Strong results on both place a model in a competitive range; actual conversations establish practical usefulness. Zephyr’s reported adoption supplied that additional signal: You.com experimented with it, and a writer used it for writing feedback instead of ChatGPT. Conversely, Gemini Ultra’s practical quality remained unresolved at recording time, pending release and hands-on use.

The next experiment—and the loop beyond one training run

The closing research questions suggest concrete experiments: - Base-model sensitivity: hold the preference dataset and training framework fixed across different base models, then compare AlpacaEval improvements. - Annotator context: compare professionally collected labels with community-collected labels to test whether the collection setting changes data quality. - Qualitative alignment: train on preference batches with different priorities and examine whether downstream behavior changes accordingly. These remain proposed studies. Qualitative alignment also exposes a weighting decision: should feedback about healthcare count the same as feedback about joke writing? Aggregation at scale does not by itself establish a fair or desirable outcome.

Beyond individual experiments lies an outer loop: observe behavior, revise reward data, and rerun training. John Schulman’s public talk on RLHF proxy objectives is cited in connection with verbosity, self-doubt, and refusals. OpenAI is described as monitoring behavior, combining curated data with user prompts, training new reward models, and repeating RLHF across model versions.

The speculative concern is a Whac-A-Mole cycle: fixing one behavior introduces another problem. Understanding and automating that cycle could make model improvement resemble reinforcement learning over a longer horizon. This is a proposed direction, offered with limited firsthand access to the work.

The conversation closes on the services business around feedback. Human-feedback and AI-feedback services are expected to coexist; if human data remains a decisive advantage, its capital requirements could support specialized, venture-backed providers. Nazneen Rajani, described as a former Hugging Face colleague, was reported to be starting a company in the space. Whether human or synthetic feedback ultimately provides the advantage remains open.