Yuanzhi Li

Yuanzhi Li is an AI researcher whose collaborations on TinyStories, phi-1 and phi-1.5 investigated smaller language models trained on synthetic stories and textbook-quality data. With Ronen Eldan, he introduced TinyStories; he subsequently coauthored models for code and common-sense natural-language reasoning. A July 2023 Cognitive Revolution episode introduced Li and Eldan as researchers from Microsoft Research.[1][2][3][4]

Li and Eldan's TinyStories paper, first submitted on May 12, 2023, examined how small a language model could be while still producing coherent English. They introduced a synthetic dataset of short stories generated by GPT-3.5 and GPT-4, using vocabulary that typical three- to four-year-olds would understand. This restricted story domain provided a setting for studying language capabilities in small models.[2]

Eldan and Li reported fluent, consistent, multi-paragraph stories from models with fewer than 10 million parameters, or simpler architectures containing only one transformer block. These were results in the simplified story domain. They also proposed using GPT-4 to grade generated stories as a teacher might grade students' writing, separating grammar, creativity and consistency rather than relying on a single score.[2]

Li was one of 19 authors of Textbooks Are All You Need, first submitted on June 20, 2023. The collaboration included Suriya Gunasekar, Sébastien Bubeck and Ronen Eldan. Its phi-1 model used six billion tokens of selected textbook-quality web data and one billion tokens of GPT-3.5-generated textbooks and exercises, followed by fine-tuning on coding exercises.[3]

In the paper's revised October 2023 version, the authors reported training the 1.3-billion-parameter Transformer for four days on eight A100 GPUs. They reported pass@1 accuracy of 50.6% on HumanEval and 55.5% on MBPP—results on those coding benchmarks.[3]

Li coauthored the September 11, 2023 phi-1.5 technical report with Bubeck, Eldan, Allie Del Giorno, Gunasekar and Yin Tat Lee. They extended the textbook-data approach to common-sense reasoning in natural language with another 1.3-billion-parameter model. The team reported performance on natural-language tasks comparable to models five times larger, alongside abilities such as step-by-step reasoning and rudimentary in-context learning.[4]

Li and his coauthors also acknowledged hallucinations and the potential for toxic and biased generations. They reported improvement on toxicity and bias associated with the absence of web data, while retaining those limitations. Their stated reason for open-sourcing phi-1.5 was to promote further research into these problems.[4]