The Utility of Interpretability — Part 1 of 3

Does the answer go through Texas?

Ask a model for the capital of the state containing Dallas, and it may answer Austin. Does it use Texas as an intermediate step? In this Latent Space conversation, Emmanuel introduces circuit tracing: a method for exposing intermediate representations involved in a particular next-token prediction. He reports that Gemma 2 2B and Claude 3.5 Haiku show extremely similar circuits on this task, despite substantial differences in their broader capabilities.

That interpretation suggests an experiment: suppress the Texas representation and check whether the answer deteriorates. Emmanuel proposes the test without reporting a result here. This distinction runs through the conversation: a readable graph suggests a mechanism; changing an internal state and observing the predicted consequence provides stronger evidence that the mechanism contributes to the answer.

At the recording, Emmanuel works on Anthropic’s interpretability circuits team. Its papers examined private models; the code release, built mostly by Anthropic Fellows, extends exploration to supported open models. Fellow Michael Hanna’s tutorial notebook covers Dallas–Texas–Austin and includes three precomputed, unresolved or unlabeled cases. There are three ways to contribute: - Investigate a behavior: inspect representations and test an explanation of something a model does. - Improve graph construction: modify the open-source method that produces the graphs. - Support another model: train the required replacement model using the available code. The invitation reflects the scale of the problem: models exhibit more behaviors than any one group can investigate.

Why “having me on” suggests a podcast

Given Thanks for having me on the Latent Space, Gemma’s most likely continuation is podcast; other candidates include stop tokens, blog, and show. The demonstrated tools target base models, which expect text to continue rather than a user–assistant exchange. Emmanuel recommends composing a passage as if it appeared in a book or article, stopping before the word you want to investigate.

The tutorial connects to Neuronpedia’s interactive circuit explorer. Its nodes include features—intermediate model representations—and let users work backward from a prediction: 1. Generate a graph: enter the prefix, name the graph, and start generation, leaving parameters alone initially. 2. Select an output: choose podcast and inspect the features feeding into it. 3. Check feature meanings: examine other text where each feature activates, alongside its label. 4. Trace and simplify: follow influences backward from the output or forward from an interesting input, prune less important nodes, and group related features. The demonstrated path includes podcast-related features, an interview feature, and a contribution from on: the phrase having me on slightly favors a podcast context.

Gratitude supplies another clue. The graph associates thanks with an interview or podcast context, suggesting an experiment: remove or reverse the gratitude and see whether the interpretation changes—perhaps toward something unpleasant, such as being on trial. That test is proposed, not performed. Trying related prompts can refine a hypothesis before deeper interventions in the notebooks, which demonstrate turning nodes off and injecting a node from one prompt into another. The notebooks are reported to run on Google Colab’s free tier without an expensive GPU.

What a clean graph leaves unexplained

A host questions the graph’s neatness: if representations overlap, why are its nodes so clean? Three limits matter: - Features are extracted directions. A displayed feature is provisionally described as a direction extracted from overlapping representations. The clean display does not mean that overlap has disappeared. - Error nodes mark missing computation. Replacement models reproduce some of the original computation and miss some. If errors dominate a graph or a node’s inputs, that portion remains unexplained. - Attention is not decomposed. The method decomposes multilayer perceptron (MLP) computation, one component of the model. Attention heads can perform substantial work along graph edges without that work being explained. Interventions test whether an interpretable path matters. Successful tests support the mechanism’s participation in the computation; they leave open how much the explanation covers and how often the method succeeds.

An informal dog-breed exploration makes the distinction concrete. A participant describes a Pomsky as a small dog mixed from a Husky and another breed, aiming for Pomeranian. After several prompt attempts and substantial guidance, the most probable output is Pom. A few minutes of exploration reveal features associated with dogs, breeding, animal welfare, animal science, and stubbornness. An evocative label does not establish an explanation of breed knowledge. The proposed next step is to change the breed and swap or clamp candidate features—forcing their activations higher or lower—to test predicted outputs. No intervention result is reported.

The demonstration closes with a broader research agenda: - Explain response style: investigate how a model judges whether a query calls for concision or detail. - Compare training stages: examine base versus chat models, early versus late pretraining checkpoints, fine-tuned versus unfine-tuned models, or different model families. - Trace failures: investigate incorrect riddles or letter counting to find representations and intermediate computations behind the error. These are proposed investigations. The team invites experiments, questions, and reports of broken functionality.

Research depends on the experimental loop

The studio conversation turns to Emmanuel Amiesen’s route into the field. He moved from applied machine learning and fine-tuning Anthropic’s production models into interpretability as improving capabilities made understanding those models increasingly compelling. His earlier book addresses practical AI-product work, such as shipping a random forest for fraud classification and avoiding implementation mistakes; he considers that advice useful but distinct from his interpretability research.

Amiesen also distinguishes the publications: Transformer Circuits is the publication; Circuit Tracing is the methods paper on which he is an author; the companion “biology” paper covers findings inside the model; and Tracing the Thoughts is the blog post. He recommends approaching the subject from a high-level account before moving into technical detail.

When a host suggests research is more valuable than engineering, the respondent pushes back. In fine-tuning and parts of interpretability, the hard work includes building an experiment, getting it to run, and interpreting its results correctly. Finding the shortest path from an idea to a useful test transfers across research domains. The empirical character of model training—where scaling compute and data often outperforms elaborate human-inspired priors—makes engineering and systems execution central to research.

Some interpretability projects are accessible without a large training cluster, using open models small enough to run on a MacBook CPU. The field’s relative youth also means fewer accumulated abstractions to learn. Features and dictionary learning are suggested foundations; Chris Olah’s blog, Distill, and Transformer Circuits are recommended starting points.

Superposition packs concepts into shared dimensions

Mechanistic interpretability seeks to understand internal representations and computations. A hypothetical fraud decision tree illustrates directly inspectable logic: a branch might flag perfume transactions above $10,000, giving an engineer a rule to assess. Neural networks instead produce weights and activations whose meanings are not immediately apparent. Many feature and circuit ideas emerged from vision research, where individual neurons can sometimes be recognized as edge, curve, or spatial-frequency detectors. Language-model neurons are often harder to interpret.

The proposed explanation is superposition: representing more features than there are available dimensions. One neuron can then participate in several concepts. In the conversation’s toy example, two neurons represent five things: their activations define a plane containing five distinct directions, spread apart like spokes without being mutually perpendicular. Under this view, locating a concept such as red means finding its direction in activation space rather than a dedicated red neuron. These represented concepts are called features.

Sparse autoencoders learn to unpack candidate feature directions. In the illustrative setup, a learned model expands two neuron activations into five latent dimensions and reconstructs the original activations from them. Training encourages only a few latent features to activate at once—this is the sparsity constraint. If successful, the resulting dictionary separates features whose meanings researchers can analyze. Although described as unsupervised, the method has a training target: reconstruct the original activations. The target comes from the data itself, much as the next token supplies the target in language-model pretraining.

From steering a feature to tracing an algorithm

Suppressing or amplifying a candidate feature tests its influence on behavior. Golden Gate Claude kept a discovered Golden Gate Bridge feature active, reportedly introducing the bridge into unrelated answers, including arithmetic. The interpretation was later refined to something closer to awe at the bridge’s beauty. A feature’s initial label can miss part of its meaning, even when steering produces a striking effect.

A collection of activated features still leaves their connections unexplained. In a travel illustration, a request involving San Francisco and Sonoma might lead through intermediate processing to a Golden Gate Bridge representation. Detecting that feature alone omits the path. Circuits connect inputs, intermediate features, their influences, and outputs to explain the algorithm producing an answer.

Other anecdotes illustrate the range of steering: - Golden Gate Gemma: a participant reports increasing weights associated with proper nouns, places and people, and the term golden, without providing precise settings or an evaluation. - Sycophantic praise: increasing a feature with that description reportedly made Claude lavish praise on an ordinary poem. Golden Gate Claude’s public release itself is recalled as an organic outcome of an amusing internal discovery and demo, rather than a choice from a ranked shortlist.

These examples raise distinct practical questions. On detecting excessive flattery in the GPT-4o update under discussion, a respondent argues that ordinary conversation already made the problem apparent. Whether steering can improve coding without trade-offs remains unanswered. Behavioral influence alone does not establish a general capability-improvement dial.

Current utility: identify causes and test predictions

Interpretability is framed as long-term, high-risk, high-reward research relative to fine-tuning. Reducing hallucination or bias through deeper understanding is an anticipated benefit. Its current practical use is to investigate particular behaviors and identify causes, including jailbreak susceptibility and unexpected planning.

A reported jailbreak investigation shows the experimental loop at work. Its circuit suggested that completing a grammatical sentence competed with stopping a harmful response. This led to a prediction: preventing sentence completion would make the jailbreak more effective. The speaker reports that it did, without giving an effect size. Here, an internal mechanism produced a tested behavioral prediction.

Making models easier to interpret from the outset could address a limitation of replacement layers: they capture only part of the original computation. Training differently, or adding a post-training step to disentangle representations, might reduce that gap. Both the direction and the possibility of only a small capability cost remain early aspirations.

Naive pruning may miss the difficulty. With superposition, a neuron that looks unimportant on individual examples may contribute to many capabilities. Small contributions spread across language understanding, integration, or code cracking could matter collectively. These are illustrative possibilities, not measured contributions.

Understanding mechanisms can guide reliance

A hypothetical workflow in which Claude writes code, reviews it, and deploys it illustrates why understanding should improve alongside capability and deployment breadth. Interpretability could help determine where a model’s method deserves trust. Understanding how it performs arithmetic, for example, might inform whether to rely on its calculations or require a calculator tool.

The analogy is a car delivered to someone in the sixteenth century: learning which controls accelerate, brake, and steer would not reveal hidden failure conditions. Likewise, good evaluation performance leaves generalization unresolved when evaluation material may have appeared in training. Observed behavior and an understood mechanism provide different grounds for deciding where a system will keep working.

The closing discussion extends this uncertainty to training recipes. A participant recalls the C4 text corpus outperforming Common Crawl and helping a training mix despite aggressive obscenity filtering, including filtering the word gay. With no model, benchmark, mixing ratio, or measured improvement specified, the anecdote illustrates an effective but poorly understood recipe. Training is compared to an inherited soup recipe: ingredients accumulate through experiments and tradition without a clear explanation of why the combination works.

Part 1 closes by introducing induction heads: attention heads described as finding a previous occurrence in the context and enabling repetition. They offer a concrete account of how a model can repeat text without dedicating too much capacity to that behavior. The speaker suggests they seem to occur in every model—a broad observation, but also an example of the explanatory goal: identifying how a behavior arises.