The Utility of Interpretability, Part 3 of 3: From Experiments to Explanations

Understanding risks before capabilities advance

Publishing safety research can help researchers understand risks before models become too capable; publishing capability improvements can accelerate the systems they are trying to understand. This tension opens Part 3 of The Utility of Interpretability. The researcher offers it as a reason safety publications matter, while declining to make a broader policy judgment.

The conversation then turns to a practical publishing problem: how much work does it take to make results about model internals understandable? The team's process starts with experimental validation before visual polish. Researchers gather results, have colleagues test them, perform interventions, and validate the observed behavior. Only then do they iterate on its presentation.

Automatic graphs, hand-built explanations

An automatically generated graph and a carefully arranged explanatory illustration can represent the same underlying data. The manual work lies in arranging boxes and arrows so readers can follow the result. Producing the data and explaining it clearly are separate tasks.

The illustrations are assembled in-house, with substantial effort at two levels: - Reusable visualization tools: experienced data-visualization colleagues build tools researchers can use to construct explanations. - Individual illustrations: researchers arrange and refine the presentation of each experiment, down to aligning its arrows.

That work can help draw people into the field. At a team meeting, everyone reportedly raised a hand when asked whether reading one of the team's papers had contributed to their decision to join. The anecdote illustrates a broader argument: difficult mathematics makes clear exposition more valuable. Readers who might avoid an equation-heavy paper can still grasp its core takeaways—and become curious enough to investigate similar behavior in their own models.

Infrastructure makes subsequent investigations faster

The publication effort was unusually large: the team released a substantial methods paper alongside another containing ten case studies. When an interviewer suggests a month as a production estimate, the researcher declines to generalize. A two-paper release is a poor baseline for a follow-up investigation.

Once the investigation infrastructure exists, moving from a prompt to an account of what happened takes roughly minutes, according to the researcher. That makes many investigations possible, while reusable diagram tools also shorten the presentation work. A follow-up investigation of a jailbreak case, for example, uncovered additional nuance. The researcher estimates that the experiment and visualization took a few days, with the diagram alone probably taking less—but did not work on it and is uncertain about the timing.

The team also open-sourced its graph-inspection interface, linked from Circuit Tracing. The researcher describes it as an integrated development environment (IDE) for inspecting model behavior: the working interface where the team conducts its investigations. Though less polished than the explanatory illustrations, it required more work to build. Other researchers who produce graphs can reuse it without reimplementing the interface.

Investigation does not require a publication-quality rendering. Outside researchers can work with the inspection interface; reusable infrastructure and the conceptual work already done should reduce the initial burden of further investigations.

Scaling succeeded; clear gaps remain

Asked what still blocks a deeper understanding of model internals, the researcher favors faster progress with more people building on existing research footholds. Three open directions are named: - Understanding attention. - Understanding longer prompts. - Exploring different replacement architectures.

This optimism follows an earlier concern: would interpretability results on tiny models ever transfer to models of practical interest? The researcher reports that scaling succeeded through substantial work. The field now has methods that work on relevant models, with clear gaps still to fill.

That leaves room for newcomers to challenge existing methods. A questionable design choice in a paper can become the starting point for an experiment, and alternatives can be tested locally on smaller models. The invitation is to participate in a young field whose methods remain open to improvement.

Start with behavior that puzzles you

Everyday use of Claude and other models supplies concrete research questions: - Arithmetic: how does a model perform calculations, and why does it make mathematical mistakes? - Behavior across languages: why might a bilingual user perceive a different personality in French and English?

These are starting points for investigation. The language example is a personal impression to probe, rather than an established finding. The closing advice is to follow the questions you find fun: notice something puzzling in a model you already use, then investigate how that behavior arises.