Nathan Lambert

Nathan Lambert is a machine-learning researcher and the writer of Interconnects. He helped build an RLHF research team at Hugging Face and later led post-training at Ai2, co-leading work on OLMo and Tülu models. His collaborative contributions include Zephyr, RewardBench, and Tülu 3: methods and tools for teaching assistants to follow instructions, learning from answer preferences, and testing the signals used to guide training. He advocates publishing training data, code, evaluations, and recipes so others can investigate and adapt these systems.[2][7][9][10][4]

Lambert left Ai2 in June 2026. As of September 2026, he describes himself as a co-founder of an unnamed nonprofit focused on open post-training recipes, how models acquire their character, and the open-weight model ecosystem.[4][2]

Lambert joined Hugging Face in May 2022. With Louis Castricato, Leandro von Werra, and Alex Havrilla, he coauthored Illustrating Reinforcement Learning from Human Feedback, published on December 9, 2022. It explains an existing training sequence: begin with a pretrained language model, collect comparisons between answers, train a reward model to predict those preferences, and optimize the language model against that reward. Comparisons provide a workable signal when people would assign inconsistent numerical scores. In his career retrospective, Lambert recalls that the article’s reception after ChatGPT’s release prompted Hugging Face to support him in forming an RLHF team.[4][6]

Lambert coauthored Zephyr, submitted in October 2023, with Lewis Tunstall, Edward Beeching, and other Hugging Face collaborators. The team adapted Mistral-7B through supervised fine-tuning on synthetic dialogues, then used AI-ranked answer pairs for distilled Direct Preference Optimization (DPO). DPO learns directly from preferred and rejected answers, avoiding a separate learned reward model and sampling during preference fine-tuning. The approach required no human annotation for those stages; this does not describe the entire upstream training history of its base and teacher models.[7][8]

The team reported that skipping supervised fine-tuning sharply reduced performance on MT-Bench and AlpacaEval, with the resulting model struggling to learn the chat template. Zephyr-7B scored above Llama2-Chat-70B on MT-Bench, but the paper notes limitations of GPT-4 judging, including bias toward distilled or verbose responses. Its alignment work specifically concerns helpfulness; harmful outputs and illegal-advice safety considerations were outside its scope.[7][8]

Lambert joined Ai2 in October 2023 after, he recalls, meeting Luca Soldaini at ICML in Hawaii and learning the institute was hiring. In his January 2024 Latent Space interview, he described working with Hamish Ivison to try the Zephyr recipe at 70-billion-parameter scale for Tülu 2, using Ivison’s JAX DPO implementation and available TPUs. He recalled that the first run produced the released model without initial hyperparameter changes or ablations, leaving substantial room for further experimentation.[4][12]

Lambert is first author of the collaborative RewardBench paper, submitted in March 2024, with collaborators including Valentina Pyatkin, Jacob Morrison, LJ Miranda, and Bill Yuchen Lin. RewardBench tests whether a reward model assigns a higher score to a chosen answer than to a rejected answer across chat, reasoning, and safety. Pairs include subtle but verifiable defects, such as code bugs or incorrect facts. It evaluates both classifier reward models and implicit DPO rewards, exposing reasoning, instruction-following, and refusal weaknesses. The paper leaves the relationship between benchmark scores and usefulness in downstream training as an unresolved question.[9]

In summer 2024, Lambert recalls assembling a larger post-training effort that became Tülu 3. Released on November 21, 2024, its initial 8B and 70B models built on Llama 3.1 base models. Core contributors included Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, and Lester James V. Miranda. Lambert received Ai2’s internal “Cat Herder” award, which he connects to coordinating Tülu 3 and OLMo 2 post-training. He explicitly credits these projects as team efforts and says his public recognition can exceed his individual contribution.[4][10][11][3]

Tülu 3 combines prompt curation, supervised fine-tuning, DPO, and reinforcement learning with verifiable rewards (RLVR), supported by standardized development and unseen evaluations and training-data decontamination. The team selected DPO for experimental simplicity and efficiency after comparing preference algorithms and tuning hyperparameters and length normalization. It reported better aggregate downstream DPO performance when preference data included generations from its own supervised fine-tuned model, rather than relying entirely on responses from other models.[10][11]

RLVR replaces a learned reward model with a verification function. For mathematics, it can match a final answer against a known answer; for precise instruction following, it can check specified constraints. These checks supply binary rewards for verifiably correct responses. Tülu 3 integrates this training into a generalist pipeline after DPO, while acknowledging earlier reasoning and execution-feedback approaches.[10][11]

The team released model weights, intermediate checkpoints, datasets, training recipes, code, and curation and evaluation tools. Its new synthetic supervised fine-tuning datasets nevertheless used GPT-4o or Claude 3.5 Sonnet responses, with Sonnet used for coding, and preference scoring used GPT-4o-2024-0806. The released artifacts let researchers train with supplied data and adapt the workflow; they do not expose the proprietary teachers’ internals or independently reproduce their generation and judging.[10][11]

Before his language-model work, Lambert studied reinforcement learning for robotics and control. He earned a Cornell BS in Electrical and Computer Engineering in 2017 and a Berkeley EECS PhD in 2022, advised by Kristofer Pister and Roberto Calandra. His doctoral period included research at Facebook AI in 2019–2020 and a DeepMind internship in 2021. Lambert recalls arriving at Berkeley to study microelectromechanical systems, then seeking AI research opportunities.[3][4]

His dissertation, Synergy of Prediction and Control in Model-based Reinforcement Learning, examines how learned environment models interact with decision making. Recursive use of a one-step model can accumulate prediction errors; accurate transition predictions also need not yield good downstream policies, a problem termed objective mismatch. Working primarily with sample-based model predictive control, he studied a trajectory-based dynamics model that represents time directly to mitigate these difficulties.[5]

Lambert’s public explanations emphasize the foundations and limits of assistant training. In January 2024, he recommended instruction tuning as a practical way to adapt models with modest compute, cautioning startups against RLHF unless it offered a clear advantage in their domain. He also questioned whether models satisfy the priorities specified for human annotators: “why don't we check if the models actually do these things that we tell the data annotators to collect?” He stressed the difficulty of connecting preference data to explicit values or measured behavior.[12]

Alongside Interconnects, Lambert wrote the RLHF and Post-Training Book as an accessible reference for toy implementations and entry into the research literature. Its introduction reports a July 2026 Manning print edition, with the web version continuing to receive corrections. His ATOM Project, introduced in August 2025, advocates fully open models for US research and industry, including training data, intermediate checkpoints, code, and permissive licenses.[2][14][15]

In his June 2026 farewell essay, Lambert described public training recipes as “research infrastructure”: resources that let researchers ask specific questions about training processes beyond what one institution could investigate alone. He favored useful medium-sized open models complementary to frontier systems and identified improving fully open post-training recipes as a priority for his next work.[4]