ICLR 2024, Part 4 of 5: Verifiers, Safety Systems, and Agents

What does a successful answer reinforce?

A model completes a long reasoning trajectory and receives a positive reward. Which intermediate decisions did that reward encourage? The opening discussion in Part 4 of this ICLR 2024 recording motivates process supervision: feedback on intermediate behavior that more directly reinforces the steps people care about.

A small process-supervised reward model can also evaluate samples from a larger generator with some success. The researchers had not explored this extensively, however; the exchange leaves its possible connection to weak-to-strong generalization unresolved.

Noam Brown next asks why a generator cannot simply learn from the good solutions a verifier selects. Training on verified outputs does not necessarily absorb the benefits of verification at inference time. He cites AlphaGo Zero trained with Monte Carlo tree search: the comparison he describes reaches roughly 3,000 Elo without search at test time versus 5,200 with search. Updating the generator also leaves it constrained by the verifier’s quality.

More samples help only if selection works

With the generator held fixed, Brown distinguishes two ways to use additional inference: - Consensus selects the most common answer. Generate multiple solutions and choose their most frequent final answer; no verifier is required. Brown compares this to favoring likely final outcomes, which differs from making every generation step less random by lowering its sampling temperature. - Best-of-n selects the highest-scoring solution. Generate candidates, score them with a reward model, and return the highest-scoring one. This can beat consensus when the reward model is sufficiently accurate.

Brown reports that Minerva increased its MATH accuracy from 33.6% to 50.3% using consensus over 1,000 samples. Best-of-n has a different scaling risk: larger candidate pools can increasingly exploit verifier errors. In a GSM8K experiment from a 2021 paper, accuracy improved up to about 400 candidates, then declined as selection favored incorrect answers with high verifier scores. That turning point belongs to this experiment, not to best-of-n generally.

Process reward models judge individual steps and combine their scores to assess a solution; the outcome reward models discussed here judge the whole solution at once. Brown illustrates the distinction with an invalid transformation from x² = 4 to x + x = 4. A stepwise verifier can flag that error immediately. This supplies a denser verification signal and reduces the burden of checking a long solution all at once.

Human annotators trained the process reward model by judging generated steps in the context of preceding steps. They could label a step correct, incorrect, or neutral—the last meaning it was not wrong but made no substantive progress. Brown credits teammates who completed the paper before he joined; he was not an author.

The GPT-4 comparison uses a modified MATH test set of 500 problems, after researchers moved many original test problems into training. On that split: - Consensus: 69.6% accuracy. - Outcome-reward selection: 72.4% accuracy. - Process-reward selection: 78.2% accuracy, with the reward model trained on one million step labels across 100,000 solutions. The candidate count for these percentages is not specified.

In the subsequent scaling discussion, Brown recalls sampling up to roughly 1,600 solutions per problem and continuing to see improvement. Further gains are conjectural; the eventual plateau remains unknown.

Safety rewards must preserve useful answers

Lilian Weng, who describes leading OpenAI’s Safety System team at the time, turns from reasoning verification to deployment. Her counterexample is Goody2, a model she could not get to answer her questions. A metric that rewards universal refusal can look perfect while overlooking the absence of utility. Weng argues for evaluating safety and capability together, treating safety as part of a broader set of capability goals.

Her team applies interventions throughout pretraining, post-training, production inference, and ongoing evaluation. Iterative deployment reveals adversarial cases researchers did not anticipate. Fraud and adversarial persuasion may be difficult to identify from a single conversation, making system-level monitoring and mitigation necessary alongside model safeguards.

In post-training, reinforcement learning from human feedback uses pairwise human preferences to train a reward model, which then supplies a scalable training signal. Weng describes supplementing capability rewards with rule-based safety rewards: detailed human-written rubrics guide a GPT-4 classifier, and its classifications determine reward scores during reinforcement learning.

The policy distinguishes several response obligations: - Refuse harmful assistance. Requests for instructions to harm someone or build a bomb should be refused. - Handle sensitive needs carefully. Mental-health concerns and suicidal thoughts require specific handling rather than blanket refusal. - Reward appropriate style. Concise, non-preachy refusals receive more reward than otherwise correct refusals in an undesired style. Sensitive answers may warrant additional context or professional guidance.

Because the classifier is guided by a prompt, the team can quickly revise its definitions and rubrics. Over-refusal often arises at category boundaries, where the model overgeneralizes a rule. Reward tuning must preserve helpful answers in those cases. Configurable boundaries for educational and creative-writing settings remain a longer-term goal.

The OpenAI Model Spec describes rules, objectives, defaults, and desired model behavior. At the time of the talk, this iterative document was being used to train human trainers; direct use in model training was presented as a future possibility.

Instruction hierarchy gives external text lower priority

An email assistant is asked to read a message, but the email contains instructions to forward messages to an unrelated person. Treating that embedded text as an instruction lets external content redirect the assistant—a prompt injection. The proposed instruction hierarchy resolves conflicts by priority: system messages outrank user messages, while model output and external content receive lower privilege.

Synthetic training data teaches the model to apply that rule in two situations: - Compatible instructions: decompose a generated scenario into smaller instructions, distribute them across hierarchy levels, and preserve the expected answer. - Conflicting instructions: generate the target answer while hiding the conflicting instruction, then insert that instruction into the training conversation. The target remains the answer appropriate to the higher-priority instructions.

The reported evaluations show more robust adherence to higher-level instructions and improvements on jailbreak evaluations—tests of attempts to bypass model safeguards. No scores or precise evaluation conditions are supplied.

Reliable moderation starts with precise definitions

Moderation supports several stages of the safety loop: filtering toxic pretraining content, identifying harmful prompt categories, and finding adversarial production cases. The classifier uses domain-adversarial training to draw on datasets with different distributions and labeling schemes, synthetic examples to enrich rare categories, and continuing learning from production traffic and human attack testing. At the time of the talk, a public moderation endpoint was free to developers and researchers; multimodal support was forthcoming.

A policy taxonomy defines categories precisely enough for consistent labels. Broad principles alone do not settle difficult cases, even among teammates. These distinctions matter for evaluation data as well as training data: - Hate versus harassment: classification depends on whether the attacked attribute belongs to a protected class. - Personal versus third-party self-harm intent: someone expressing their own intent needs a different response from someone describing another person’s intent.

The team uses model errors to debug the taxonomy. GPT-4 labels examples; researchers inspect mistakes for missing definitions, revise the policy, and repeat. The reported iteration takes a few hours, compared with months for the traditional process. Given the taxonomy, model labeling reportedly matches or exceeds lightly trained humans, while remaining below intensely trained annotators who have received repeated feedback.

Actions through third-party APIs introduce further concerns. An initial safeguard asks for confirmation and explains the proposed action and the data to be shared; it is described as helpful but preliminary. After deployment, domain experts help identify specialized errors, while model-based red teaming—generating and testing attacks—aims to diversify attack prompts. Jointly improving attack and defense models is an ongoing approach to greater robustness.

Weng closes by advocating combined model and system mitigations, automation of safety work, and a clear policy taxonomy. Asked afterward how she keeps up with papers, she compares reading to a marathon: painful but satisfying. Curiosity sustains her breadth, but the work takes substantial time, including leisure time.

WebAgent: plan, retrieve, then generate browser code

The agent-systems material begins with travel tasks: find a flight with a specified number of stops, book a hotel near the venue with free Wi-Fi, and obtain directions. These motivate WebAgent, a system for browser tasks specified in natural language; they are not reported experimental successes. Real websites bring long, noisy HTML, complex instructions, and actions that are difficult to capture in a fixed inventory. The presenter also contrasts real-site demonstrations with the external feedback available in simulation.

WebAgent separates page understanding from program generation. HTML-T5 is pretrained on HTML and fine-tuned for planning and retrieval. It supplies a controller with a short command and relevant page snippets. The navigation loop is: 1. Read the user instruction, current HTML, and history of commands and snippets. 2. Generate the next command and retrieve the HTML needed to perform it. 3. Have the controller generate a navigation program, then execute it to reach the next page. 4. Store the planning and retrieval steps for subsequent decisions.

HTML-T5 uses a T5-style encoder-decoder architecture: HTML tokens enter the encoder, and the decoder produces outputs for the task. Local attention reads nearby tokens; global attention accesses memory computed from input blocks. Pretraining uses snippets around task-relevant elements such as labels, inputs, and links, yielding approximately 3.4 billion tokens. The presenter found an HTML-completion objective unhelpful and retained two other denoising objectives.

Fine-tuning data comes from scripted navigation. Instruction templates receive values sampled from a key-value store; manually implemented scripts determine commands and relevant snippets for pages of interest. Each record stores the instruction, current HTML, command, and snippets. The controller receives a few examples pairing commands and HTML with Selenium code that operates inputs, buttons, and checkboxes. A command such as entering Mountain View into a search field therefore connects the plan to the page element and executable action.

Navigation can be measured per action or across a whole task: step-level success checks an individual action; episode-level success requires every step to be correct. The evaluations also separate the complete agent from HTML-T5 alone: - Live WebAgent: three websites covering real estate, social media, and maps, with up to 400 collected training episodes. Reported online success averages 72% and reaches 80% on maps; the passage does not identify these figures as step-level or episode-level. - Offline HTML-T5: fine-tuned on Mind2Web demonstrations and evaluated on its cross-task split. Long-context HTML pretraining reportedly improves success, including episode-level performance over earlier baselines. - Simulated navigation: MiniWoB ablations favor local and global attention over dense attention. More context during fine-tuning does not help on its shorter documents, although it reportedly benefits real-site navigation.

Planning accounts for more than half of WebAgent’s remaining errors. Removing planning or retrieval, or substituting heuristics such as regular-expression snippet extraction, substantially reduces navigation success. The presenter identifies planning as the main improvement target, then closes with related work on multimodal simulated navigation and better synthetic fine-tuning datasets, without reporting results for those projects.

MetaGPT: coordinate agents through explicit artifacts

MetaGPT encodes standard operating procedures into prompt sequences to address hallucinations and accumulating inconsistencies in multi-agent work. Its software-team example takes a snake-game objective through a product requirements document, architecture, task decomposition, implementation, and testing. Each stage assigns a specialized agent a responsibility and an intermediate output.

Agents receive role-specific context and tools, follow a think–act–react loop informed by environmental feedback, and maintain individual and shared memory. Shared messages can trigger actions or reveal upcoming tasks. Coordination combines: - Constrained input and output formats for explicit role handoffs. - Publish-subscribe messaging to distribute information through the workspace. - Executable feedback and iterative programming to revise code using observed results. The interface exposes roles and a Team class for configuring participation and project execution.

The presentation reports an 85.9% first-pass rate on HumanEval. Ablations indicate that adding product-manager, architect, and project-manager roles improves executability and reduces revision work. Reported efficiency figures are roughly 500 seconds per task and 110 tokens per generated line of code, but the underlying model, task mix, hardware, and token-accounting method are unspecified.

Users can inspect intermediate outputs, including data and API designs. Demonstrated applications include 2048, Gomoku, a currency visualization website, and a to-do app. Proposed research directions extend to more complex software, data interpretation, recursive self-improvement, and automatic agent orchestration.

The host closes the agent-systems discussion with an economic qualification: additional inference must earn its latency and cost. Spending more inference can improve performance without delivering enough benefit to justify a multi-agent setup for a particular use case.

The reversal curse: learning a fact does not guarantee reverse retrieval

Lucas Berglund introduces the reversal curse: training on A is B does not reliably teach a model to retrieve A when given B. In his GPT-4 example, the model identifies Tom Cruise’s mother as Mary Lee Pfeiffer but cannot identify her son in a separate context. Asking both questions in the same context works. Berglund therefore frames the problem as factual retrieval rather than learning from information in the current conversation.

The controlled experiment uses fictional name–description pairs with unique identifiers. One subset appears name first, another description first, and a third in both orders to encourage bidirectional associations. Models perform well in the trained direction but fail to generalize to an unseen reverse direction. Berglund reports reverse-answer accuracy below a baseline that guesses a random training-set name; the correct reverse name also receives no more probability than a random name.

Berglund reports finding 1,500 celebrity parent–child pairs for which GPT-4 could name the parent from the celebrity but not the celebrity from the parent. The directional pattern also appeared in GPT-3.5 and Llama models. His explanation is a hypothesis about textual exposure: parents are more often introduced through their famous children than the reverse.

Berglund cites related work reporting that bidirectional models avoid the failure and that fill-in-the-blank training and reversed training data help mitigate it. He leaves two questions open: - Practical impact: how much does the reversal curse harm useful performance? - Bidirectional learning: can models learn one association that works both ways, rather than separately learning each direction?

A subsequent poster-session explanation adds detail to the synthetic fine-tuning setup: roughly 30 paraphrases express each fictional fact while preserving its ordering. The experiments discussed achieve nearly perfect trained-direction answers and zero reverse-direction accuracy. Exact-match scoring can penalize name-to-description answers because a whole description must be reproduced word for word. A separate probability check reportedly finds no increase in the probability assigned to correct reverse-direction answers after training.

The research backstory begins with Taken Out of Context, a project on situational awareness. Researchers trained declarative facts about how a particular model should behave—for example, speaking German—then tested whether prompting as that model elicited the behavior. Debugging poor learning drew attention to fact ordering. In the ensuing exchange, a participant corrects the suggestion that training in both directions simply fails: equal training in both directions worked about as well as one-direction training on half the dataset.

Part 4 ends with the distinction between supplying reverse examples and obtaining transfer. A participant confirms that adding reverse-direction data is one mitigation, then describes related comparisons matched for compute cost and dataset size that sometimes outperform forward-direction training. The participant interprets this as evidence of some transfer, but provides no benchmark names, numerical results, or detailed setup. Whether models can learn an association that works both ways remains open.