Jessica Rumbelow

Jessica Rumbelow is an AI interpretability researcher who designs algorithms to explain what machine learning models learn, how they make predictions, and where they fail. Her work includes joint research with Matthew Watkins on glitch tokens, the Leap Laboratories team's prompt-attribution library PIZZA, and her Exemplar Partitioning method for finding interpretable structure in language-model activations.[2][4][5][6]

In May 2026, Rumbelow introduced Exemplar Partitioning, an unsupervised alternative to learning feature dictionaries with sparse autoencoders. It streams a model's activations and groups them around real, observed examples: an activation joins an existing region if an exemplar lies within a calibrated distance threshold; otherwise, it becomes the exemplar of a new region. The threshold controls resolution, while the data determine dictionary size. Construction requires no gradient descent, and exemplars can serve as directions for interventions in the model.[6][7]

Rumbelow reported mean AUROC of 0.881 across 500 concepts on AxBench latent concept detection at layer 20 of Gemma-2-2B-it, compared with 0.755 for the canonical GemmaScope sparse-autoencoder entry and 0.911 for SAE-A. She reported approximately 1,000 times less build compute in the stated comparison. These are construction and benchmark results, rather than a general inference-speed claim; she also identified sensitivity to activation streaming order and the need to test additional models and datasets.[6][7]

Her refusal experiment distinguished identifying a behavior-associated direction from reproducing the behavior. Removing the exemplar direction of a refusal-associated region sharply suppressed refusal in the tested Gemma configuration. Adding the direction produced degenerate refusal-like outputs and, at higher strengths, token loops. As she put it in her explanation of the experiment, the direction was “not causally sufficient on its own.”[7]

In her June 29, 2025 article, Scientific Discovery in the Age of Artificial Intelligence, Rumbelow argued that interpreting predictive models can reveal patterns beyond those researchers already know to investigate. She described Leap's approach as gathering scientific data, training a model to predict something about the phenomenon, and interpreting what it learned. She acknowledged dependence on data quality and quantity, assumptions embedded in model architecture, and the expense of training fresh models to avoid importing unrelated patterns from a pretrained model.[8]

Rumbelow supported accelerated discovery but warned that plausible, subtly erroneous LLM-generated papers could worsen the reliability of scientific literature. “The point is not to speed up irreplicable science,” she wrote. Her recommendations included preregistration, requirements for usable code and data, better-supported expert review, and incentives for publishing high-quality datasets. She advocated using LLMs for language, familiar code, and some reasoning tasks while considering other data-driven approaches for scientific work.[8]

These later methods and judgments followed earlier work on explaining model outputs without access to their internals, and on investigating unexpected language-model behavior.[2][5][4]

On August 3, 2024, Rumbelow published Why did ChatGPT say that?, introducing PIZZA and crediting Leap's research and engineering team, including @Arush, @sebastian-sosa, and Robbie McCorkell. The open-source library attributes outputs to prompt inputs by removing or replacing tokens and measuring changes in completion probabilities or meaning. It can calculate how each input token affects individual output tokens. For closed models, it uses GPT-2 tokenization and embeddings as proxies, which Rumbelow cautioned could obscure important subtleties.[5]

Rumbelow explained hierarchical perturbation as starting with large, overlapping prompt chunks and subdividing those with sufficiently high attribution. This avoids testing every token in regions that contribute little to the output. She declined sweeping efficiency claims because performance depends on saliency, thresholds, and chunk configuration. Detecting hallucination, deception, or steganography through attribution patterns remained proposed research directions.[5]

Earlier, Rumbelow and Matthew Watkins conducted research at SERI-MATS that led to their February 5, 2023 SolidGoldMagikarp post. They optimized prompts toward target completions in continuous embedding space, using regularization and constraints to keep the results close to valid tokens. While investigating embedding clusters, they encountered unusual strings such as SolidGoldMagikarp and cloneembedreportprint, then tested whether models could repeat them.[4]

They chose GPT-3 davinci-instruct-beta at temperature zero for controlled tests rather than relying on the changing ChatGPT interface. Their screening of 50,257 tokens, followed by ChatGPT list-repetition tests, narrowed the set to 374 problematic tokens. They distinguished ordinary word fragments that models tended to complete from stranger tokens that prompted stalling, substitutions, evasion, or insults. The candidate list was explicitly nondefinitive. A February 14, 2023 update said ChatGPT appeared to have been patched, while unusual behavior remained in the playground models then tested.[4]

Before her later language-model and scientific-discovery work, Rumbelow built and interpreted models using histopathology data for cancer diagnostics. Her doctoral research concerned model-agnostic interpretability—explaining a model's decisions without needing access to its internal structure. The University of St Andrews records her PhD thesis, Model agnostic interpretability, with Ognjen Arandjelović and David James Harrison as supervisors.[2][3]