Rylan Schaeffer

Rylan Schaeffer researches the evaluation and predictability of AI systems. With Brando Miranda and Sanmi Koyejo, he coauthored Are Emergent Abilities of Large Language Models a Mirage?, which challenged whether apparent sudden jumps in model capabilities reflect changes in the models or choices in how they are measured. The paper received an Outstanding Main Track Paper award at NeurIPS 2023.[4][5]

Schaeffer is an AI/ML computer science PhD student at Stanford in Sanmi Koyejo’s STAIR lab, supported by a Stanford Data Science PhD Scholarship. His personal website lists previous research roles in Google Gemini pretraining and data, reached through the Character AI acquihire, Gemini post-training, and Meta GenAI’s Llama evaluations.[2][3]

First submitted in April 2023, the Mirage paper proposed an alternative explanation for abilities that appear absent in smaller models but suddenly present in larger ones. For a particular task and model family, the researchers held model outputs fixed and examined how scoring choices affected the apparent scaling behavior. They argued that nonlinear or discontinuous metrics can produce abrupt-looking transitions, while linear or continuous metrics reveal smoother, more predictable improvement. The distinction concerns how performance is measured, even when the underlying outputs remain the same.[4]

Schaeffer, Miranda and Koyejo supported their explanation with a mathematical model and three complementary analyses: tests on InstructGPT/GPT-3 tasks with claimed emergence, a meta-analysis of BIG-Bench results, and demonstrations that metric choices could create seemingly emergent abilities in vision networks. Their conclusion was qualified: alleged emergence can disappear with different metrics or better statistics and may not be a fundamental property of scaling AI models. Schaeffer discussed this work in Latent Space’s NeurIPS 2023 recap, published December 23, 2023.[4][1]

Schaeffer also coauthored DecodingTrust, a broader collaborative evaluation of GPT-4 and GPT-3.5 covering toxicity, stereotype bias, robustness, privacy, machine ethics and fairness. The work received the NeurIPS 2023 Outstanding Datasets and Benchmarks Paper award in the benchmark category. Its authors reported vulnerabilities including toxic or biased outputs and leakage of private information; they also found that GPT-4’s stronger standard-benchmark performance did not consistently translate into greater resistance to jailbreaking prompts.[5]

In Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?, first submitted in June 2024 and revised in February 2025, Schaeffer worked with Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman and Sanmi Koyejo. Across five model families and twelve multiple-choice benchmarks, the authors reported that transformations from negative log likelihoods into benchmark scores progressively weakened the statistical relationship between performance and scale. Their explanation was concrete: predicting whether a model selects the correct answer requires tracking both its probability for that answer and fluctuating probabilities for the specific incorrect alternatives.[6]

A January 2026 project explanation describes Schaeffer’s joint work with Noam Levi, Miranda and Koyejo on scaling laws for generative evaluations, including GSM8K and MATH. They proposed three ways to fit and forecast pass-at-k: using pretraining compute, model parameters plus pretraining tokens, or the log likelihoods of gold reference solutions. They tested forecasting by fitting laws on cheaper models and predicting the performance of more expensive models.[7]

The authors reported that the compute and parameters-plus-tokens laws required models within approximately two orders of magnitude of the target. Parameters for the gold-reference law converged using models up to approximately five orders of magnitude cheaper. That result concerns fitted-parameter stability in their backtesting study, rather than a general guarantee of forecasting accuracy.[7]

Before his work on frontier models, Schaeffer studied neuroscience-inspired AI. His earlier training spanned statistics, computer science and cognitive neuroscience: he double-majored in Computer Science Engineering and Computational Statistics at UC Davis, then pursued a master’s in Cognitive Neuroscience at University College London, researching metacognition. He spent a year as an Uber data scientist before returning to academia for a second master’s in computer science at Harvard, followed by a Google DeepMind internship before beginning his PhD. His earlier NeuroAI research examined neural representations involved in mammalian spatial problem-solving, including localization, mapping and navigation.[8]

In a November 13, 2024 Stanford interview, Schaeffer emphasized continuity in machine learning’s basic workflow: “We’ve always collected large datasets, trained models on them, and refined our models based on the results.” He also described increasingly rigorous analysis of data and model behavior. In his view, better tools, coding assistants and chatbots accelerate experiments and analysis, while greater speed and complexity increase the risk of polished but fundamentally flawed work.[8]

In the same interview, Schaeffer recalled advice received from an unnamed computer science PhD student before he joined Stanford: a successful researcher should accelerate when the work becomes miserable. Remembering that every project contains unpleasant parts had helped him persist. “That’s been incredibly true for me, and I think it’s a valuable lesson for everyone,” he said.[8]