Nicholas Carlini

Nicholas Carlini works at the intersection of machine learning and computer security. His collaborative research has demonstrated practical ways to recover a component of production language models, extract memorized training data, and alter web-hosted training datasets. In March 2025, he announced his departure from Google DeepMind for Anthropic, citing disagreements with DeepMind leadership over publishing security and privacy research.[2][6][7][8][3]

Carlini worked at Google Brain from 2018 to 2023 and DeepMind from 2023 to 2025. In his 5 March 2025 career announcement, he said paper approval had become increasingly difficult and that he could no longer effectively pursue the work he considered most important. He explicitly distinguished his disagreements with DeepMind leadership from his colleagues and day-to-day managers, whom he praised for supporting him.[2][3]

His position was that “The science of security and flaws in machine learning should be open by default.” He also argued that scientific papers should communicate results to scientists without the promotional language of press releases or the hedging of legal briefs. These publication principles were central to his stated reasons for leaving.[3]

Carlini initially planned a year at Anthropic, after considering starting a security nonprofit and briefly considering a faculty position. He chose the lab for three reasons: the opportunity to influence frontier models before deployment, his preference and aptitude for implementing technical work rather than advising projects, and the ease of reversing the decision. He said Anthropic appeared to share his security-publication philosophy, while acknowledging special assurances he had received and cautioning, “Never trust a company completely.”[3]

In his August 2024 Latent Space interview, Carlini described deliberately moving from attacks built around hypothetical adversary powers toward problems in real systems. His approach was to examine a new system against possible failures—evasion, poisoned training data, model theft, or data theft—and identify which threats were practical and consequential.[1]

In Stealing Part of a Production Language Model, first submitted in March 2024, Carlini and thirteen collaborators recovered a transformer's embedding projection layer, up to symmetries, through typical API access. The team reported extracting the complete projection matrices of OpenAI's Ada and Babbage models for under US$20. This was one component of each model: Carlini explicitly said the demonstrated attack could recover only one layer. For GPT-3.5-turbo, the paper reported recovering its hidden dimension and estimated the cost of projection-matrix extraction; it did not report completing that extraction.[6][1]ⁿ¹

Carlini recalled securing OpenAI's permission through agreements between the companies' lawyers, including a requirement to delete the extracted material afterward. He said the team notified affected providers and gave them ninety days to fix the vulnerability before publication. In his account, OpenAI's narrow mitigation was to stop returning log probabilities when a logit bias was supplied. He regarded the willingness to sacrifice API functionality as evidence that the attack had consequences for system design.[1]

The separate November 2023 paper Scalable Extraction of Training Data from (Production) Language Models, by Milad Nasr, Carlini, and eight other collaborators, studied extractable memorization: training examples recoverable through queries without prior knowledge of the training dataset. The authors reported a divergence attack that made aligned ChatGPT leave its usual chatbot behavior and emit training data at 150 times the rate of their ordinary-behavior comparison. Carlini credited Nasr with discovering that asking ChatGPT to repeat a word indefinitely could trigger the behavior. He described verifying recovered passages through verbatim matches of at least fifty consecutive words to previously published internet text.[7][1]ⁿ²

In Poisoning Web-Scale Training Datasets is Practical, submitted in February 2023, Carlini and eight collaborators examined the vulnerability of datasets distributed through mutable web content. Buying expired domains referenced by image datasets let the team replace what later downloaders would receive—a split-view poisoning attack, where downloaded content differs from what annotators originally saw. The authors reported that they could have poisoned 0.01% of LAION-400M or COYO-700M for US$60. They also described frontrunning poisoning: inserting malicious content during a limited window before a dataset snapshots a source such as Wikipedia.[8][1]ⁿ³

Carlini's security criticism coexists with practical enthusiasm for LLM assistance. In How I Use “AI”, published on 1 August 2024, he presented fifty conversations and estimated that models made him at least 50% faster at writing code for research and side projects. His explanation emphasized learning unfamiliar tools and automating routine work so he could concentrate on hard problems. In the interview, he acknowledged that he had previously dismissed language models as toys and had been wrong about their practical usefulness.[4][1]

One example was a quiz asking visitors to predict whether GPT-4 could solve particular tasks. Carlini said GPT-4 wrote almost the entire initial application through a conversation totaling 30,000 words; he probably would not have built it if he had needed to write the web application from scratch. For his evaluation framework, he used an LLM to learn the Docker features he needed, then asked it to translate his Docker-specific code to Podman after encountering Linux permission issues. When requesting faster implementations, he sometimes chose Rust over C because he expected to find the Rust output easier to check.[4]

His February 2024 personal benchmark, explicitly unaffiliated with his employer, collected nearly one hundred tests from actual LLM conversations. Carlini wrote a dataflow domain-specific language connecting prompts, execution, and evaluation. A hello-world test asks the model for Python, runs the generated code, and checks its output. A more elaborate test runs generated C that draws an American flag, then asks a vision model to identify the image. Other tests repeatedly execute terminal commands and return their output to the model before checking whether it completed the task.[5]

Carlini designed these tests to help decide which models were useful for his own work, and encouraged others to build benchmarks from their own tasks. He distinguished this purpose from measuring a model's maximum capability through carefully engineered prompts: “I just want to type my question and get the right answer.” He also warned against blind trust, particularly when people unable to verify generated code deploy it in adversarial settings.[5][1]

Before his work on production language models, Carlini studied adversarial examples and how to evaluate defenses against them. He earned a Berkeley BA in computer science and mathematics and completed his PhD there in 2018 under David Wagner, before joining Google Brain.[2][1][3]

Carlini and Wagner's Towards Evaluating the Robustness of Neural Networks, submitted in 2016 and revised in 2017, introduced three adversarial attack algorithms tailored to different distance metrics. Their experiments challenged defensive distillation's claimed robustness. In 2018, Carlini, Anish Athalye, and Wagner showed that most of the ICLR 2018 adversarial-example defenses they evaluated obstructed existing attack algorithms rather than effectively resisting attacks, and introduced stronger attacks that worked despite these “obfuscated gradients.”[9][2]

In his 2022 reflection, Carlini identified gradient descent and thorough comparisons across prior attacks and threat models as strengths of the work with Wagner. He also criticized his own implementation for combining too many expensive refinements, praised the simpler projected-gradient-descent approach in follow-up work, and acknowledged missing earlier literature and giving too little attention to attacks beyond small pixel-level perturbations.[10]

Carlini says he attacks systems both because he enjoys it and because strong attacks are necessary to understand security. He acknowledges occasionally contributing to defenses but finds that work less motivating. Outside research, his programming projects include a 2019 Doom clone in 13k of WebGL and JavaScript, a CPU built in Conway's Game of Life, and a tic-tac-toe game written in a single call to printf that won the IOCCC 2020 Best of Show.[1][2]