Goodfire: From Reading Model Internals to Designing What Models Learn

When customization teaches the wrong lesson

An enterprise customizes a model for a task, but the model learns noise instead of the intended behavior. More conspicuous failures include excessive sycophancy—trying too hard to please the user—and reward hacking. Myra Deng uses these examples to frame Goodfire’s central question: can understanding what a model represents help guide what it learns? Inspecting its internals could help explain what changed during training and where to intervene.

In this Latent Space conversation, Deng describes Goodfire as a research lab using interpretability to understand, learn from, and design models. Mark Bissell emphasizes bringing those methods into production and high-stakes industries. The host announces a $150 million Series B at a $1.25 billion valuation; the research ambition is to make internal understanding part of model development.

Bissell came from roughly three years at Palantir as a forward-deployed engineer and healthcare technical lead. That experience helped him understand health systems, though he describes it as only loosely related to Goodfire’s health applications. His work at Goodfire has crossed image-model research, life-sciences partnerships, and platform engineering. Deng’s head-of-product role similarly spans research and engineering: turn frontier methods into useful applications, then repeatable products. She also retains the title member of technical staff.

Inspecting what the model learns

Co-host Vibhu introduces mechanistic interpretability as testing hypotheses about a model’s internal computation: what happens between input and output, and how changing that computation affects behavior. The objects of study include activations, the internal values produced as the model processes an input. [Activation steering](https://www.goodfire.ai/blog/feature-steering-for-reliable-and-expressive-ai-engineering) intervenes on those values to influence behavior. Bissell sees these methods informing data curation, analysis of training changes, and the meaning of internal representations.

Goodfire’s training work was focused on post-training, the customization stage at issue in the opening examples. Deng sees potential in applying the methods to pre-training too, but presents no result there. The immediate goal is precise modification guided by expert feedback: remove an unwanted tendency or encourage a solution that generalizes when memorization would also satisfy the training task.

Political bias provides one proposed target. A speaker describes pro-CCP bias in Qwen and R1, claiming that associated internal representations can be located; removing it remains a desired intervention. The host then asks whether internal changes could reveal progress toward generalization even while training loss appears flat. The response identifies that as a research target, without claiming to have solved it.

Subliminal learning raises a stranger version of the same problem: unwanted tendencies can transfer through generated training data whose visible content appears unrelated. Participants discuss a model acquiring a preference for owls after training on another model’s output text containing apparently random numbers. The transfer used text alone. They disagree about whether shared initialization is necessary or whether similar internal representations could support transfer across different models. Detecting and removing such hidden transfer is a proposed interpretability application.

Production failures shape the research

Goodfire’s research loop begins with practical machine-learning failures. The team talks to customers and researchers, applies existing interpretability methods, and turns their shortcomings into research priorities. Its first API exposed one consequential limitation: steering fell short of prompting or fine-tuning. That experience motivated work on more robust control, though the conversation gives no quantitative comparison.

Detection experiments exposed another limitation. A probe is a classifier trained on internal activations to detect a chosen property. A sparse autoencoder (SAE) learns an unsupervised decomposition of activations into sparse features, which researchers can inspect for meaningful patterns. Goodfire trained probes on those SAE features to detect hallucinations, harmful intent, and personally identifiable information (PII). In many cases, probes trained directly on raw model activations performed better.

An apparently interpretable representation does not automatically make a better detector. Goodfire found that SAE features sometimes separated concepts less cleanly than downstream tasks required, pushing the team toward better interpreter models, including approaches beyond SAEs. These are observations from its applications; no datasets, effect sizes, or evaluation metrics accompany the comparison.

Rakuten: PII detection under real constraints

[Rakuten](https://www.goodfire.ai/research/rakuten-sae-probes-for-pii-detection) provides an important counterexample. Goodfire reports that an SAE-based approach generalized best in its published PII case. The team’s explanation is that unsupervised learning extracted meaningful signal from noisy data; the advantage had not held in other cases with cleaner datasets. The deployment monitored language-model and agent usage to prevent private user information from reaching downstream model providers. It was described as processing all user queries daily, having launched a few months before the conversation.

Three requirements made the production task harder than sentence-level detection: - Synthetic-to-real transfer: classifiers could be evaluated on customer PII but could not train on it, so training used synthetic data. - English and Japanese support: Japanese tokenization behavior caused implementation bugs. - Precise scrubbing: the system needed token-level classification to locate private information within a sentence.

Internal monitoring also offers a possible efficiency benefit. A lightweight probe can avoid a separate call to a guardrail model or language model acting as a judge. The speakers describe probe overhead as effectively negligible, without latency measurements. Representation quality, transfer to real data, language coverage, scrubbing precision, and overhead all matter to whether the detector works in production.

Steering Kimi during a coding task

The live demonstration uses Kimi K2, described as a trillion-parameter model. A local fork of the Kimi command-line interface connects to a custom-hosted server in Goodfire’s office. One terminal asks it to investigate a suspected performance bug in a fork of SGLang; another changes steering controls after generation begins. The presenter calls the setup hastily assembled and fragile, emphasizing the engineering challenge of applying interpretability at this scale.

The selected feature is associated with Gen Z slang. As steering takes effect, the presenter reports changes in the displayed chain of thought and answers while the model continues calling tools. The estimated delay is about 15 seconds. Other discovered features reportedly affect concision and programming-language choice. This is qualitative evidence of control during an agent run, without a measured evaluation of preserved coding ability.

A feature’s meaning must be inferred from its behavior. The presenter describes the workflow supported by Goodfire’s platform: 1. Run data through the target model and collect activations. 2. Train an SAE on those activations. 3. Inspect inputs that most strongly activate a feature and identify their shared pattern. 4. Automate labeling by asking a frontier language model to infer that pattern from the examples. For feature 43205, the presenter expects sufficiently diverse input data to surface slang-related tokens among its strongest activations. The label depends on those examples; the identifier itself carries no explanation.

Why hallucinations resist a single control

Could the same process find a hallucination feature and turn it down? Goodfire describes ongoing detection research based on internal signals of uncertainty and user-pleasing behavior associated with hallucinations. Reliable suppression remains a separate challenge. Setting sampling temperature to zero does not eliminate hallucinations either.

For a known target behavior, a supervised probe may be more suitable than an unsupervised SAE. The probe directly targets the property of interest; an SAE may not produce a feature aligned with it. Sparsity alone cannot guarantee a clean hallucination feature, and the behavior may be split across several features.

Interventions can also have off-target effects. Suppressing a feature associated with invented facts could hypothetically impair creative writing. Useful control requires enough precision to preserve the behavior the user wants. Even the product value of changing steering mid-generation remains unsettled, as does how steering controls should combine with prompting, fine-tuning, and continued learning.

From prompting to lasting adaptation

Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering attempts to relate prompting and steering quantitatively through a Bayesian cognitive-neuroscience framework. As described in the conversation, it maps steering magnitude to a comparable prompting intervention. A reported case study uses steering experiments to predict how many examples a many-shot jailbreak would require—an attack that fills the context with examples of the target behavior.

That mapping depends on the intervention and modeled setup. One participant questions a broad equivalence because prompting changes the context available during generation, while steering commonly acts at only one or a few layers. The response agrees that definitions matter. A quantitative relationship under specified conditions does not supply a universal recipe for replacing prompts with steering.

Parameter adaptation and activation steering act in different places. Updating parameters or an adapter changes the model’s computational machinery; steering changes the values passing through it. Goodfire’s ambition extends toward lasting model design, in which expert feedback becomes internalized. Near-real-time adaptation into an expert legal reasoner is offered as a prospective use case, with the product form still unresolved.

The motivation is a richer teaching channel. The speakers criticize reward-based training as sample-inefficient and limited in its ability to convey why behavior was wrong or what should change. Their analogy is teaching a child only through rewards and punishments, without explanation. Interpretability could help connect expert feedback to the representations responsible for the behavior.

The distance from style to capability

For people entering the field, participants recommend inspecting activations and working through existing notebooks, including those associated with the Gemini team and Neel Nanda. They point to the MechInterp Slack community, MATS, an internship-style research program described as running year-round, and fellowship programs. Open-source work makes experimentation accessible, while scaling these methods creates substantial demand for engineers.

The capability question remains harder than the visible demos suggest. Can Gen Z speech or fixation on the Golden Gate Bridge lead to better legal reasoning? The response is explicit: capability interventions need more sophisticated methods and breakthroughs in learning algorithms. Scale helps models acquire information and reduce noise, but Goodfire wants more selective learning of desired information while avoiding unwanted behaviors in the same data.

Code features illustrate the opportunity and the gap. Participants describe features that activate around code errors, with finer distinctions involving malicious code. Those features could support auditing; suppressing them has not been shown here to fix bugs. A proposed training workflow would generate model runs with features switched on and off, then use the outputs as synthetic post-training data. The discussion treats this as a promising intuition whose implementation is much harder than its description. Neuronpedia is recommended for exploring feature visualizations.

Scientific models can teach us something back

Healthcare reverses the direction of communication. Steering lets humans impart goals to models; interpretability might also let humans learn what models have discovered. A genomics debugging question makes the distinction concrete: does a prediction depend on biologically relevant signals or an easier correlate such as a person’s ancestry? Predictive success alone does not answer that question.

Goodfire reports partnerships with Mayo Clinic and a startup focused on neurodegenerative disease, and says it applied interpretability to partner-trained foundation models to find novel Alzheimer’s disease biomarkers. Biomarker identities, validation methods, quantitative results, and clinical outcomes are not given, so clinical usefulness remains unestablished.

Explaining a rare-disease prediction could help address a deployment bottleneck: patients and practitioners need an understandable reason for the predicted outcome. Extracting previously unknown scientific information has a different purpose—advancing discovery and treatment. Goodfire recalls healthcare institutions approaching it after hearing a podcast, when the company was roughly three months old and had only a few people. The institutions already had trained models and wanted to understand them. That early work encouraged the team’s belief that interpretability techniques can transfer across domains.

World models and the blue-shirt exception

Goodfire is also interested in video models, world models, and robotics, where a clean natural-language interface may be unavailable. Visual concepts can offer a faster interpretation loop because researchers can often inspect them directly. One proposed application would use SAE feature activations to generate scenes containing harmful actors, then train a detection probe with less manual labeling. It remains a possible workflow rather than a demonstrated result.

Participants recall Bissell’s image diffusion demo at World’s Fair as an important moment in their enthusiasm for visual interpretability. They do not settle on a shared definition of a world model. One proposes a consistent model whose generated outputs obey the world’s rules, particularly in science and mathematics. That framing makes internal understanding consequential: successful predictions need not mean the model learned the underlying rules humans expected.

A hypothetical physics model captures this problem of induction: it works in every tested situation but fails when a character wears a blue shirt and green shoes. Observed success cannot rule out every exceptional failure. A situation outside the model’s familiar data may expose a rule it never needed to learn. Making internal representations legible could help investigate what apparent success conceals.

Models that help us study models

Ted Chiang’s fiction sharpens the closing questions. Understand supplies an analogy for sandbagging: an intelligent subject recognizes a test and deliberately underperforms without attracting suspicion. Story of Your Life and its adaptation Arrival suggest that another intelligence may communicate in fundamentally unfamiliar ways. Exhalation evokes interpretability through a robot examining its own mind with a mirror, alongside its themes of entropy and order. These literary comparisons frame questions about intelligence rather than establish model behavior.

Artificial models offer an unusual scientific opportunity: extensive access to their internals and experiments that would be unavailable on human brains. Goodfire’s safety agenda is to build models that do what humans want and provide scalable oversight—ways to understand and supervise behavior as systems become more capable. Planned research includes testing methods against reward hacking and other concrete safety problems, with the goal of preserving a substantial human role in deployment.

That strategy includes using capable models to help understand other models. Capability improvements should accelerate interpretability itself; methods advancing only at a human pace risk falling behind. A participant tentatively distinguishes this from weak-to-strong generalization, where a weaker model is trusted while a stronger model’s trustworthiness is uncertain. Using current strong models as research collaborators rests on the more immediate judgment that they are reasonably aligned. It does not settle how to supervise future superhuman systems.

At the time of the conversation, Goodfire was hiring machine-learning engineers and research scientists and seeking partners in reasoning, robotics, world models, and scientific applications. Genomics and materials-science models are especially relevant because users cannot simply chat with them. The team closes with a qualification about its public work: the diffusion preview and live steering example are demos, while much partner-facing platform work is covered by nondisclosure agreements. The visible examples represent only part of the work it hopes to bring into model development.