Matt Fredrikson

Matt Fredrikson is CEO and cofounder of Gray Swan AI and an associate professor at Carnegie Mellon University. His AI-security work includes joint research on transferable language-model jailbreaks and indirect prompt injection against agents. At Gray Swan, he works with cofounder Zico Kolter on testing AI vulnerabilities and enforcing policies on model and agent behavior.[3][4][8][6][7][1]

In his June 2026 Latent Space interview, Fredrikson said Gray Swan grew out of his and Kolter’s Carnegie Mellon research into deep-learning vulnerabilities, adversarial testing, and more robust inference. He described a recent shift in the company’s work for frontier labs: earlier testing concentrated on chat interactions that circumvented content policies; the focus had moved toward agents, tool use, and downstream applications. Its approach combined prize-based community red teaming with models trained for automated adversarial testing.[1]

Fredrikson and Kolter were among the 31 coauthors of a March 2026 study of indirect prompt injection. The competition tested attacks across tool calling, coding, and computer use, including whether an agent could execute a harmful action while leaving no sign of compromise in its final response. The authors reported 272,000 attack attempts from 464 participants against 13 frontier models, yielding 8,648 successful attacks across 41 scenarios. Every tested model was vulnerable, and capability and robustness showed weak correlation. These results describe the competition’s tested models and scenarios, rather than the probability of compromise in ordinary deployment.[7]

Fredrikson’s preferred enforcement point is what the agent is about to do. He described Cygnal, Gray Swan’s guardrail model, as checking tool calls against an organization’s custom data-usage policies. Encountering a prompt injection should not necessarily end a coding task that has already run for three hours, he argued: “The real question is whether the agent’s planned action violates a policy. If it does, stop it there.” He also emphasized system-level controls. In his bank example, AI-layer safeguards should operate on a platform that grants agents the access they need without exposing everyone’s banking information or the organization’s most sensitive assets.[1]

When Gray Swan announced its $40 million Series A on May 28, 2026, Fredrikson said security had not kept pace with AI. He connected the funding and Snowflake partnership to bringing security capabilities used by frontier labs to the platforms where enterprises were already building applications on their data.[8]

Fredrikson qualified claims that Shade, Gray Swan’s automated red-teaming system, outperformed human attackers. In June 2026, he said automated techniques could find more breaks within a fixed time window and a specific task set, while adding: “I don’t think we’re quite to superhuman levels of red teaming yet”. His qualification kept the comparison tied to the conditions of the test.[1]

A browser robustness challenge illustrated why he resists broad comparisons between human and AI safety. Fredrikson described assigning browser tasks to human participants or agents, then allowing attackers to choose phishing or prompt injection. Attackers knew which kind of participant they were targeting and adapted their techniques. He reported that skilled attackers phished participating humans with 60–70% success, while a couple of models suffered only a handful of successful breaks. The results surprised him, but he emphasized different failure modes rather than concluding that agents were generally safer. As a counterexample, he described frontier models falling for an email that called itself a simulation and instructed them to forward future messages to a random address.[1]

Before this work on tool-using agents, Fredrikson studied attacks on language models and privacy risks in machine learning. That earlier research supplies context for his later emphasis on testing AI under adversarial conditions.[6][3][1]

In July 2023, Fredrikson coauthored Universal and Transferable Adversarial Attacks on Aligned Language Models with Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, and J. Zico Kolter. Their method automatically searched for suffixes to append to objectionable requests, using greedy and gradient-based techniques to increase the likelihood of an affirmative response instead of a refusal. The authors reported that suffixes trained across multiple prompts and Vicuna-7B and Vicuna-13B transferred to the then-public ChatGPT, Bard, and Claude interfaces. The findings concern those tested 2023 systems.[6]

Fredrikson earned a B.S. in Computer Science from Duquesne University in 2007 and a Ph.D. in Computer Sciences from the University of Wisconsin–Madison in August 2015, advised by Somesh Jha. He began as a Carnegie Mellon assistant professor in September 2015. His earlier joint publications include the 2014 pharmacogenetic privacy study of personalized warfarin dosing, with Eric Lantz, Jha, Simon M. Lin, David Page, and Thomas Ristenpart, and the 2015 paper Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures, with Jha and Ristenpart.[4][5][3]

His background also includes writing secure code whose properties could be formally verified by an algorithm. Returning to that earlier experience in the June 2026 interview, Fredrikson proposed that coding agents could make secure implementation backends more practical if they became good at generating such code: “People can still code in English; the agent can generate the secure implementation.” He presented this as a possibility, while describing model robustness and policy enforcement as problems that still require research.[1]