Ronen Eldan

Ronen Eldan is a mathematician whose AI research includes TinyStories, developed with Yuanzhi Li to study coherent language generation in very small models, and approximate unlearning, developed with Mark Russinovich to change an already trained model’s recall of selected material. His mathematical contributions include a neural-network depth theorem with Ohad Shamir and the creation of stochastic localization, recognized by the 2023 New Horizons in Mathematics Prize.[2][3][4][6]

Eldan and Li introduced TinyStories in a paper first submitted on 12 May 2023. They used GPT-3.5 and GPT-4 to generate short stories containing only words typically understood by children aged three to four. This deliberately restricted domain let them investigate whether coherent English required models with hundreds of millions of parameters and many attention layers.[2]

The authors reported that models with fewer than ten million parameters, or simpler architectures with only one transformer block, could generate fluent, consistent stories spanning several paragraphs and demonstrate reasoning capabilities. These were results within the restricted story domain. The parameter count and single-block architecture described alternative ways of making models smaller or simpler.[2]

Eldan and Li also proposed using GPT-4 to assess generated stories as a teacher would grade students’ writing, assigning separate scores for grammar, creativity and consistency. Their stated hope was that TinyStories would support language-model development and analysis, especially in low-resource or specialized domains, while helping researchers study the emergence of language capabilities.[2]

In October 2023, Eldan and Mark Russinovich published Who’s Harry Potter? Approximate Unlearning in LLMs. They addressed the problem of unlearning selected training material without retraining a language model from scratch, framing the work around the legal and ethical challenges posed by copyrighted content in training corpora.[3]

Their method had three components. First, they further trained a model on the target material and compared its next-token scores, or logits, with those of a baseline model to identify target-related tokens. Second, they replaced distinctive expressions with generic counterparts and used the model’s predictions to generate alternative token labels. Those labels aimed to approximate predictions from a model never trained on the target material. Finally, they fine-tuned the model on the alternative labels.[3]

Testing on Harry Potter material in Llama2-7b, Eldan and Russinovich reported suppressing the model’s ability to generate or recall the target content with about one GPU hour of fine-tuning, while performance on Winogrande, Hellaswag, ARC, BoolQ and PIQA remained almost unaffected. This was an experiment in approximate unlearning of a particular corpus, rather than a guarantee of complete deletion under every prompt.[3]

Before these 2023 language-model projects, Eldan had worked on mathematical questions about neural-network representation and high-dimensional geometry and probability. That earlier work provides the theoretical context for his career in AI research.[4][6]

Eldan earned a mathematics BA from the Open University of Israel in 2004 and a PhD in pure mathematics from Tel Aviv University. He subsequently became a visiting researcher at Microsoft Research’s Theory Group in Redmond, Washington, and joined the Weizmann Institute’s Department of Mathematics in April 2015.[5]

With Ohad Shamir, Eldan published The Power of Depth for Feedforward Neural Networks at the 2016 Conference on Learning Theory. They constructed an approximately radial function representable by a small three-layer network that a two-layer network could not approximate beyond a specified constant accuracy unless its width grew exponentially with the dimension. The result covered activation functions including rectified linear units, sigmoids and thresholds. It established that an additional layer can offer an exponential representational advantage for some functions; it was not a universal claim about practical training performance.[4]

Eldan received the 2023 New Horizons in Mathematics Prize for creating the stochastic localization method. The award citation credited the method with significant progress on open problems in high-dimensional geometry and probability, including Jean Bourgain’s slicing problem and the KLS conjecture.[6]