Pratyush Maini

Pratyush Maini is a machine-learning researcher and founding member of DatologyAI whose work examines how training data shapes foundation-model behavior, memorization, and reliability. His contributions include synthetic web-data rephrasing, tests for detecting training-data use, benchmarks for machine unlearning, and methods for building safety into pretraining.[2][4][6][7][8]

Maini received his PhD in Machine Learning from Carnegie Mellon University, advised by Zico Kolter and Zachary Lipton. On his personal website, where he also uses the handle @pratyushmaini, he announces that he will become an Assistant Professor of Computer Science at Cornell Tech and the Cornell Ann S. Bowers College of Computing and Information Science starting in spring 2027.[2]

Maini and Vineeth Dorna conducted the core research for BeyondWeb, presented by DatologyAI in August 2025. The framework generates synthetic pretraining material by transforming existing web documents into different formats, styles, and structures—for example, question–answer pairs or passages with a more pedagogical tone. Existing documents supply the knowledge; the rephrasing model changes how that knowledge is presented.[4]

The research tested which choices make synthetic data useful over long training runs. The team found that high-quality source documents help even when their information also appears in the original training text. Matching deployment-relevant styles helps too, but its benefits diminish. Diversity across generation strategies sustained improvements where more uniform approaches plateaued. No single factor was sufficient, and stronger general-purpose generators did not consistently produce better training data.[4]

DatologyAI reported that an 8-billion-parameter model trained with BeyondWeb exceeded Nemotron-Synth—the high-quality synthetic subset of Nemotron-CC—by 2.6 percentage points in average accuracy across 14 tasks using zero-shot and five-shot prompting. Synthetic-dataset comparisons mixed 60% RedPajama tokens with 40% synthetic tokens. The BeyondWeb model reached RedPajama’s 180-billion-token baseline performance after 23.2 billion tokens, and Nemotron-Synth’s after 66.2 billion: reported token-efficiency gains of 7.7× and 2.7× for those comparisons.[4]

In his May 8, 2025 Stanford NLP seminar, Maini argued that memorization research offers a different lens on model safety. His abstract connected measuring memorization, detecting unauthorized training-data use, and evaluating unlearning to the need for “natively safe models”—rather than adding alignment patches after a model has learned harmful patterns.[3]

Maini coauthored Safety Pretraining with Sachin Goyal, Dylan Sam, and other collaborators. First submitted in April 2025, it proposes four interventions: classify web data as safe or unsafe; rephrase unsafe material into safer narratives; teach refusal and moral reasoning through dedicated pretraining datasets; and tag harmful content with a special token that supports steering at inference. The September 2025 revised abstract reports attack success falling from 38.8% to 8.4% on the evaluated LLM safety benchmarks without degradation on general tasks.[8]

A 2026 follow-up with Dylan Sam, Sachin Goyal, Alexander Robey, and Zico Kolter asked when to introduce these interventions. Holding data sources and interventions fixed, the team compared starts after 0%, 20%, or 60% of pretraining tokens. With standard top-k decoding, delayed interventions after an initial period of safe-only pretraining often produced the strongest robustness, especially after benign fine-tuning. With safety-aware inference, interventions beginning immediately improved steerability toward safer outputs. The result makes timing a conditional curriculum choice rather than a rule that earlier is always better.[9]

Before these pretraining studies, Maini worked on identifying the knowledge a model acquires from its training data. His 2021 Dataset Inference paper with Mohammad Yaghini and Nicolas Papernot, an ICLR spotlight paper, addressed model stealing. Their method combined statistical testing with estimates of training points’ distance to a model’s decision boundary, using private training-data knowledge as evidence of copying. In experiments on CIFAR10, SVHN, CIFAR100, and ImageNet, the authors reported confidence above 99% while exposing only 50 training points.[10]

In January 2024, Maini, Zhili Feng, Avi Schwarzschild, Zachary Lipton, and Zico Kolter introduced TOFU: A Task of Fictitious Unlearning. Its central question is whether a model can be tuned to behave as though it had never learned selected information. TOFU uses 200 synthetic author profiles with 20 question–answer pairs each, designates some profiles as a forget set, and evaluates forgetting through a suite of metrics. None of the baselines tested in the paper achieved effective unlearning under that evaluation.[7]

Later that month, Maini and Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly proposed Web Rephrase Augmented Pre-training (WRAP), an earlier foundation for BeyondWeb’s source-rephrasing approach. WRAP jointly trains on original web text and paraphrases generated by an instruction-tuned model in styles such as “like Wikipedia” or “question-answer format.” The authors reported approximately 3× faster pretraining on C4. At matched compute, they reported more than 10% average perplexity improvement across Pile subsets and more than 2% improvement in zero-shot question-answer accuracy across 13 tasks.[5][4]

Maini’s June 2024 LLM Dataset Inference paper with Hengrui Jia, Nicolas Papernot, and Adam Dziedzic returned to detecting training-data use in language models. The team showed that apparent success in identifying individual training passages can reflect distribution differences—for example, comparing older training articles with newer non-training articles. Most tested membership-inference methods performed no better than random guessing when the distributions matched. Their alternative selected useful membership signals and aggregated them into a dataset-level statistical test, shifting the question toward collections of documents. On the evaluated Pile subsets, the authors reported distinguishing training and test sets with p-values below 0.1 and no false positives.[6]