Zico Kolter

Zico Kolter is a machine-learning researcher and Gray Swan cofounder whose contributions include automated attacks on aligned language models, provable defenses against adversarial examples, and deep equilibrium models. He combines collaborative technical research with AI safety governance: OpenAI announced his appointment to its board on August 8, 2024.[1][7][8][9][5]

Kolter is professor and head of Carnegie Mellon University's Machine Learning Department, cofounder and Chief Scientist of Gray Swan AI, and chair of OpenAI's Safety and Security Committee (personal biography). He cofounded Gray Swan with Matt Fredrikson.[4][1]ⁿ¹

Kolter coauthored Universal and Transferable Adversarial Attacks on Aligned Language Models with Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, and Matt Fredrikson. First submitted in July 2023, the work automated the search for text that could make aligned models answer objectionable requests rather than refuse them.[7]

The team's method used greedy and gradient-based search to generate adversarial suffixes appended to requests. Its objective was to increase the probability of an affirmative response. Training a suffix across multiple prompts and both Vicuna-7B and Vicuna-13B, the researchers reported that it transferred to black-box public interfaces including ChatGPT, Bard, and Claude. These were results for the interfaces tested in the 2023 study.[7]

Before this language-model security work, Kolter had collaborated on methods for bounding adversarial behavior in classifiers and reducing the memory required by deep sequence models. These earlier contributions provide context for the breadth of his machine-learning research.[8][9][7]

With Eric Wong, Kolter coauthored Provable defenses against adversarial examples via the convex outer adversarial polytope, first submitted in November 2017. For deep ReLU classifiers, their method constructs a convex outer approximation of the activations reachable under a norm-bounded input perturbation, then minimizes worst-case loss over that region. Representing the associated linear program's dual as a network makes robust-loss bounds efficient to optimize.[8]

The guarantees apply within that bounded threat model. The authors reported a convolutional MNIST classifier with provable test error below 5.8% for attacks with an ℓ∞ perturbation bound below ε = 0.1. Their approach also guarantees detection of adversarial examples on previously unseen inputs within the threat model, while allowing some non-adversarial examples to be flagged.[8]

In 2019, Kolter, Shaojie Bai, and Vladlen Koltun introduced Deep Equilibrium Models. Rather than explicitly traversing a stack of hidden layers, the architecture finds an equilibrium hidden state through root-finding and backpropagates through it using implicit differentiation. The authors characterized this as equivalent to an infinite-depth, weight-tied feedforward network.[9]

The approach requires constant memory with respect to effective network depth. In language-modeling experiments involving self-attention transformers and trellis networks, the team reported memory reductions of up to 88%, with computational requirements similar to the comparison models. That percentage was an experimental maximum, rather than a universal saving.[9]