Joel Hestness

Joel Hestness is a machine-learning researcher and coauthor of Cerebras-GPT, an open language-model family built to study efficient training and predictable scaling. His work also includes a 2017 empirical study of how model error changes with training-data size and a practical guide to Maximal Update Parameterization (μP). In May 2024, he was principal research scientist and lead of the core machine-learning team at Cerebras.[4][3][5][6]

Hestness coauthored Cerebras-GPT with Nolan Dey, Gurpreet Gosal, Zhiming (Charles) Chen, Hemant Khachane, William Marshall, Ribhu Pathria, and Marvin Tom. Announced on March 28, 2023, the family comprised seven models ranging from 111 million to 13 billion parameters. The team trained them on the public Eleuther Pile dataset using the Chinchilla recipe of 20 training tokens per parameter; Hestness submitted the paper to arXiv on April 6.[2][4]

The experiment varied training-data quantity with model size rather than giving every model the same number of tokens. Under the chosen recipe, a one-billion-parameter model receives 20 billion training tokens. This allocation targets accuracy for a fixed pre-training compute budget: spending too much on data for a small model or too little on data for a large one can use that budget inefficiently. The team established a scaling law on Pile and compared training efficiency on both pre-training and downstream language tasks.[2][4]

All seven models were trained on Andromeda, a cluster of 16 Cerebras CS-2 systems. Its weight-streaming architecture enabled data-parallel training without partitioning the model across devices. In their release, the authors described this as allowing researchers to focus on machine-learning design while reducing distributed-system engineering and model-parallel tuning.[2]

Hestness, Dey, and Cerebras cofounder Sean Lie jointly presented openness as part of the project's purpose. They argued for models that were “open, reproducible, and royalty free for both research and commercial applications”. The release made the models, weights, and checkpoints available under Apache 2.0, alongside descriptions of the training methods and performance results.[2]

In September 2024, Hestness, Nolan Dey, and Quentin Anthony published The Practitioner’s Guide to the Maximal Update Parameterization. They argued that expensive hyperparameter tuning can distort comparisons between research ideas: a method may succeed because it suits an existing training recipe. They called this the “Parameterization Lottery” and advocated μP as a way to make comparisons more robust by keeping optimal hyperparameters stable across model widths.[5]

Their guide explains μP through control of training dynamics: weight initialization controls the forward and backward passes, while learning-rate scaling controls weight updates as model width changes. This allows practitioners to tune a small proxy model and transfer the resulting settings to a wider model, reducing the need for expensive searches at large scale.[5]

The authors recommend two implementation checks. A coordinate check trains models of different widths for ten steps and measures activation magnitudes to test whether they remain independent of width. A subsequent μTransfer test checks whether optimal hyperparameters remain stable as width varies. Their guidance retains practical limits: μP does not eliminate all training instability, and architecture or training-procedure changes warrant renewed tuning.[5]

Hestness's work on predictable scaling predates Cerebras-GPT. His 2017 research examined learning curves as training datasets grew, providing earlier empirical context for the later study of compute-efficient language-model scaling.[3][4]

Submitted on December 1, 2017, Deep Learning Scaling is Predictable, Empirically was coauthored by Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. The team tested machine translation, language modeling, image processing, and speech recognition, reporting power-law scaling of generalization error and sublinear growth of model size with training-data size.[3]

Within the tested settings, the authors found that model improvements shifted error levels but did not appear to change the power-law exponent—the steepness of the learning curve. They identified practical uses for these measured relationships: debugging models, setting accuracy targets, deciding how much to grow a dataset, and guiding computing-system design.[3]