Archit Sharma

Archit Sharma is an AI researcher and coauthor of Direct Preference Optimization (DPO), a method for aligning language models with human preferences using a classification loss. His work also examines what drives improvements from AI feedback and how physical robots can learn through autonomous practice. In his personal biography, he describes an interest in autonomous decision-making agents for both digital and physical worlds, with greater reasoning and planning capabilities.[3][4][6][2]

Sharma coauthored Direct Preference Optimization: Your Language Model is Secretly a Reward Model with Rafael Rafailov, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. First submitted in May 2023, the paper reparameterizes the reward model used in reinforcement learning from human feedback so that the corresponding optimal policy can be recovered in closed form. This lets the team solve the standard RLHF problem with a simple classification loss, eliminating the need to sample from the language model during fine-tuning. Sharma also participated in the DPO poster-session segment of Latent Space’s NeurIPS 2023 recap.[3][1]

In February 2024, Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora and Thomas Kollar published A Critical Evaluation of AI Feedback for Aligning Large Language Models. They questioned whether the complexity of the reinforcement-learning step was warranted in the AI-feedback pipelines they studied. The team reported that its gains were virtually entirely explained by using a weaker teacher, such as GPT-3.5, to collect supervised fine-tuning data and a stronger critic, such as GPT-4, to generate feedback. The finding challenges the interpretation of those comparisons; it does not establish that reinforcement learning is generally unnecessary.[4]

The official implementation, hosted in Sharma’s repository and built on the DPO implementation, includes a synthetic bandit experiment to explain the limitation. With no prompt and only a small set of possible responses, it illustrates how improvement from AI feedback can still leave a sufficiently weak initial model behind a strong teacher. This supplies a concrete mechanism for the paper’s comparison with supervised fine-tuning rather than a general prediction about deployed models.[5]

Before these language-model studies, Sharma worked on learning skills and enabling robots to practice tasks autonomously. That earlier research provides context for the physical-agent side of his work.[8][6]

Sharma coauthored Dynamics-Aware Discovery of Skills (DADS) with Shixiang Gu, Sergey Levine, Vikash Kumar and Karol Hausman. Published at ICLR 2020, the method learns skills without rewards while also learning a dynamics model of those skills. That model can then support model-based control on a downstream task. The open-source implementation names Sharma as its contact.[8]

In March 2023, Sharma, Ahmed M. Ahmed, Rehaan Ahmad and Chelsea Finn introduced MEDAL++ in Self-Improving Robots. Starting with a small set of expert demonstrations, the robot learns both to perform and to undo a task, enabling repeated autonomous practice while inferring the reward function from the demonstrations. The policy and reward function learn directly from high-dimensional visual inputs, bypassing explicit state estimation and task-specific visual-encoder pretraining. The project page specifically credits Sharma and Ahmed with substantial contributions to real-robot experimentation.[6][7]

Sharma’s personal biography describes doctoral research at Stanford advised by Chelsea Finn and an earlier AI residency at Google Brain. He graduated from IIT Kanpur with a bachelor’s degree in electrical engineering and minors in machine learning and linguistic theory, worked there with Piyush Rai, and undertook a research internship at MILA with Yoshua Bengio.[2]