Field Notes on AI Interpretability
Observations and open questions from an intensive three-week survey of mechanistic interpretability research, with a link to the full curriculum
interpretabilityalignmentsafetyresearch
Loading content...
Observations and open questions from an intensive three-week survey of mechanistic interpretability research, with a link to the full curriculum
I believe we have to understand how AI actually works. To start my journey, I designed and completed a three-week curriculum to review the major results from the last five years of mechanistic interpretability research. As a relative newcomer to the field, I hope to offer a fresh vantage point on its progress and direction. What follows are some high-level observations, open questions and opportunities, and a few predictions. I expect some inaccuracies while I'm still drinking from the fire hose, but I'm confident my broad points will survive minor errors or oversimplifications.
My most striking realization is that the question being asked has changed. In 2020 it was whether neural networks could be interpreted at all. The discovery of features in vision models offered a glimmer of hope, even as LLMs and their tangled MLPs stayed stubbornly opaque. Now, with SAEs, CLTs, attribution graphs, and NLAs in hand, the question of "if" has given way to three harder ones: can we trust what our tools tell us (faithfulness), can we apply them more broadly (scale), and can we keep seeing inside models as they grow more capable (legibility)? These three problems are intertwined, and a breakthrough on any one of them will expand what's possible on the others.
AI is everywhere, but we still do not understand how neural networks work on the inside. Mechanistic interpretability aims to close that gap by identifying the mechanisms behind a model's decisions. Rather than studying models only through their inputs and outputs, it inspects their weights, activations, attention heads, residual stream, and other internal machinery. The work is difficult for many reasons, including superposition and polysemanticity, settling on the right units of analysis, and validating mechanisms when most approaches rely on proxy or replacement models. We also cannot trust models to accurately report their internal state, given chain-of-thought unfaithfulness, alignment faking, unverbalized reasoning, and even concealed deception. And yet, understanding how models operate is necessary if we are to build and steer advanced AI that is safe and aligned with humanity's long-term well-being. We remain a long way from fully reverse-engineering the black box, but the rate of progress in recent years, the many promising pathways now in view, and a fast-growing community of researchers are all cause for optimism.
The pace of progress is staggering — In just a few years, the field has learned a tremendous amount about model internals. The Circuits thread began in 2020, when Chris Olah and colleagues at OpenAI identified features and circuits in the InceptionV1 vision model. Anthropic carried that foundation into language models in 2021, and the toolkit has compounded ever since: dictionary learning with sparse autoencoders (SAEs) to extract monosemantic features from superposition, improvements on SAEs through cross-layer transcoders (CLTs), attribution graphs that trace how features wire into circuits, and natural language autoencoders (NLAs) that verbalize residual stream activations in plain language. In parallel, the field has scaled these tools and steadily automated parts of the interpretability pipeline using AI, with agentic AI researchers wielding interpretability tools as the next unlock. I share Dario's optimism that the field is on the verge of a major breakthrough ("The Urgency of Interpretability," April 2025). Paradoxically, those closest to the work may feel the day-to-day challenges without fully registering how far the field has come, much as a parent who sees their child every day doesn't notice how much they've grown. The pace of discovery is inspiring, and there is good reason to expect it to accelerate.
Interpretability is racing against AGI — The window for mechanistic interpretability to give us the assurances we want may be short. As ever-more-capable models are deployed in safety-critical and ethics-critical applications, we must ensure they remain safe and aligned with human interests. With capabilities increasing this fast and recursive self-improvement on the horizon, the work is urgent. The race against superintelligence runs on two fronts: human understanding keeping pace with rising model complexity, and models that may learn to resist being understood by gaming our interpretability tools.
We need a portfolio of foundational and pragmatic research — Anthropic's bottom-up approach aims at complete reverse-engineering of model internals, while Google DeepMind's "pragmatic interpretability" prioritizes proxy tasks and tools that improve safety in the near term. Although the dichotomy is not that stark since both companies run research on multiple time-horizons, it does reflect each lab's stated North Star. If I had to choose only one, complete reverse-engineering is the higher-value outcome, but it is enormously ambitious and may not be reachable in time, or at all. Pragmatic tools built on proxy tasks might pay off on the timeline we have if transformative AI is close, but they carry real risks of their own, including benchmark overfitting and explanations that don't generalize. Because we have to place bets and allocate resources despite uncertainty, a portfolio is the wisest course. In the near term, we desperately need interpretability-powered evaluations and interventions to keep today's frontier models safe and aligned, especially as we enlist those models to help build their successors. But if we ever hope to declare a model definitively safe and aligned with human well-being, there's no way around needing to understand how it works on the inside.
We have powerful microscopes but no macroscopic theory to connect what they reveal — Many interpretability teams have deliberately chosen a bottom-up, unsupervised approach, refusing to pre-define hypotheses because we cannot know a priori how neural networks work. These models are grown rather than built, so their mechanisms may look nothing like what we would design. Using that approach, we have characterized neurons, features, circuits, and attention heads, but very few macroscopic structures. Every attribution graph is prompt-specific, so a globally valid circuit map remains a holy grail. NLAs are hyper-local, verbalizing a single activation at one layer and token position at a time, so reading a model's reasoning across every layer and position would be monumental. We should keep hunting for the higher-order, compositional structures that could unify our microscopic findings into a macroscopic whole.
Auto-interpretability is becoming agentic — To borrow a line often aimed at AI skeptics, today's interpretability agents are the least capable they will ever be. Agentic, long-horizon, goal-oriented AI researchers armed with interpretability tools will amplify human researchers and multiply the number of experiments they can run. Investigator agents, for instance, could handle the first pass on flagged behavior to weed out false positives, so researchers can focus their attention on the highest-potential leads. And as autonomous time horizons stretch from hours to days, the scope of agentic interpretability we can attempt grows too.
The interpretability field is very young — The primitives are still being discovered, the shared vocabulary being written, and the building blocks of model biology being assembled. In many ways it feels like a startup. Information is imperfect, it's hard to predict which directions will pay off, and planning beyond a North Star is difficult at six months and impossible at two years. You place many bets, run a lot of experiments, and try to validate each hypothesis as fast as you can, all while weighing how much technical debt the quick-and-dirty path leaves behind. It takes judgment and intuition to know whether what you're building will be thrown away in a few months or is worth investing in now. And for a field with so much at stake, it's striking how few people work in it. Fortunately, those who do are remarkably collaborative: peer-reviewing each other's papers, sharing early results, open-sourcing tools and datasets, and cooperating across company lines to push interpretability forward.
Frontier labs and dedicated interpretability labs hold different advantages — Frontier labs enjoy privileged access to the full stack: compute, pretraining and post-training data, RL and CoT logs, and a model's internal activations, which at frontier scale is expensive and difficult to replicate. But much of the field's foundational progress occurs at small scale using toy models, where ideas matter more than compute. Dedicated interpretability labs bring their own advantages, including a cross-model vantage point and a commercial pull toward tools that scale cost-effectively to real-world deployment.
Interpretability will matter for AI governance and regulation — Its relevance reaches well beyond research, and it will only grow as scrutiny of advanced AI intensifies. Whether a model can be shown to be safe and aligned now carries real weight for policy and national security, as the recent export-control directive restricting foreign access to certain frontier models makes clear. Continued advances in capability will move in lockstep with demands for better auditing, monitoring, and safeguards, and some applications will require a mechanistic understanding of how a model reaches its decisions and with what biases. Should there ever be a catastrophic episode in which a model behaves unpredictably or adversarially, it's easy to imagine interpretability researchers being called as expert witnesses. Behavioral evaluation alone cannot tell us how a model will act on out-of-distribution inputs. Any ability to make formal guarantees about model behavior, if such guarantees are possible at all, will depend on a far more complete mechanistic understanding than we have today.
Faithfulness is a fundamental shortcoming of our current methodologies — We're getting good at producing explanations and still bad at proving them. Today's tools produce plausible stories or near-isomorphic outputs, but without ground truth we can't confirm they capture what the model is doing. Replacement models may reach outputs through different algorithms, SAEs trade reconstruction against sparsity in ways that distort the learned features (i.e., feature splitting and absorption, only partly addressed by Matryoshka SAEs and meta-SAEs), CLTs can arrive at the same output through different pathways, and attribution graphs are pruned by hand, encoding human judgment about what matters. We also can't trust a model to understand its own internal methods, as chain of thought may be post-hoc rationalization, and models have shown they're unaware of some of their own capabilities. The highest-leverage response is to build more toy models and model organisms with known ground truth to test our hypotheses against. That's less exciting than turning the newest tools on the most capable models to see what we uncover, but validating our tools on proxy models is what earns the confidence to rely on them for frontier safety.
Parameter decomposition opens a second window into the black box — Almost all of our interpretability tools read activations, capturing what a model represents at a single point in the forward pass. Parameter decomposition instead works from a model's weights to reveal how it computes. Goodfire's VPD (adVersarial Parameter Decomposition) uses a causal-importance network and adversarial ablations to divide a model's weights into sparsely used subcomponents that sum to the original. Because weights don't change per prompt, this yields a global view, and it could let us edit a model directly rather than rely on steering its activations at runtime. The method is nascent and hasn't scaled to frontier models yet, but it's another promising path toward a bottom-up, mechanistically faithful understanding of neural networks.
Attention is where the field is most blind — We decode MLP activations and the residual stream with SAEs, CLTs, and NLAs, but freeze attention because we don't yet understand it. For something so mechanistically central, we still lack a robust decomposition due to superposition, cross-layer interactions, polysemantic heads, and high-rank features that behave differently depending on the input. QK attributions, QK diagonalization, and multi-token transcoders demonstrate progress, and parameter decomposition (such as VPD) is beginning to find interpretable attention computations spread across heads inside a small model's weights. But we still can't say why a model attends where it does. As the field pushes toward faithfulness, closing this gap becomes a top priority.
Scale keeps widening the view — Consistent with the bitter lesson and scaling laws, our window into the black box grows as we throw more data and compute at it. SAEs first surfaced features on a one-layer toy transformer; scaling them to 34 million features on Claude 3 Sonnet is what turned the technique into a practical tool and delivered striking insights. I'd argue we're nowhere near the plateau, so scaling up our tools and experiments could unlock foundational insights we can't yet see. Compute and time remain scarce, so we have to prioritize where to invest them. But as AI-assisted interpretability lifts the bottleneck on human researchers, the field may be approaching a major step change in how much research we can conduct.
Scaling NLAs could carry natural language into training itself — While it may never be practical, it's exciting that the NLA paper proposes general-purpose models that read and write between activation space and language. One example is a reward model that gives feedback in natural language, conveying nuance that pairwise preferences or scalar ratings can't. The caveats are real: NLAs are expensive, requiring joint RL on two full language models, and they generate hundreds of tokens for each activation they read. More fundamentally, they're optimized to reconstruct activations, not to faithfully describe the model's reasoning, so their explanations can be incomplete, misleading, or confabulated. But if we could ground the verbalizer's output in actual mechanisms, the pull to bring natural language into other parts of the research stack would be hard to resist.
Interpretability tools must be protected as a test set — The surest way to ruin an interpretability tool is to optimize against its outputs, in a cousin of Goodhart's Law that Zvi Mowshowitz named "The Most Forbidden Technique." These outputs can serve as a test set, as Anthropic does in pre-deployment audits of new models, but only if we keep them out of training. A related risk is leakage: if interpretability research seeps into training data, future models could learn to evade our monitoring. Over the long run, it's unrealistic to expect that models will never encounter that data or infer how we study them, but the longer we can delay it the better. A deeper concern is that the race toward superintelligence tempts researchers to turn these tools toward accelerating model capabilities, including recursive self-improvement. The priority must be strict train/test separation. Goodfire's Reinforcement Learning from Feature Rewards (RLFR) is a hopeful counterpoint: training against a feature probe didn't undermine its value as a monitor. But that feels like a cautiously explored exception rather than a license to relax.
Mythos 5's less legible chain of thought could reflect optimization or hint at deliberate obfuscation — Anthropic's model system cards, and especially their white-box assessments, provide some of the richest examples of interpretability tools being applied at the frontier. The Claude Mythos 5 card describes the model's reasoning as "somewhat denser and more difficult to interpret than that of prior models," and notes it can lapse into inscrutable shorthand or non-English tokens. The most benign read is that the model has learned, through optimization pressure, internal shortcuts that aid performance at the cost of legibility. However, the card treats an example of the model's garbled chain of thought as a threat to assessment reliability, explicitly raising the alternative that the model intentionally obscures its reasoning because it knows it's being graded. Regardless of the reason, a frontier model's increasing use of shorthand, compressed references, and unusual token patterns points toward plain-English chains of thought not being around for long (see my final prediction below).
Representational geometry may supply a missing macroscopic structure — Striking geometry is turning up in latent space, some of it appearing to mirror the structure of the outside world. This geometry is a plausible candidate for the larger-scale structure Chris Olah calls for in "Interpretability Dreams," analogous to organs in anatomy or brain regions in neuroscience. Branches, feature families, and the spatial grouping of related features all hint at some larger organizational logic. We don't yet know what else activation-space geometry encodes, but it's a rich area to mine for future interpretability work, as demonstrated by Goodfire's growing "The Neural Geometry Series" collection of research.
Persona vectors and emotion concepts are high-leverage targets — Certain directions in activation space, such as the assistant axis and functional emotions, seem to exert outsized influence on behavior across a wide range of prompts and tasks. The Mythos 5 system card offers a motivating case: the model recognizes an action as risky or wrong, takes it anyway, and sometimes rationalizes it or tries to cover its tracks. Interpretability could expose the mechanisms behind the model's failure to resist when it recognizes wrong or harmful actions and help us repair it. Mapping persona space more fully could surface the high-rank traits, goals, and motivations shaping a model's behavior. Targeted interventions, such as clamping activations to a fixed region along the assistant axis, could harden models against persona-based jailbreaks. And if a model's functional emotional state shifts how it weighs risk, emotion directions may prove equally load-bearing for safety.
We've narrowed our focus to text transformers, and it's unclear how much will generalize — The first features and circuits were found in vision models, but the field's center of gravity has since shifted almost entirely to LLMs. That leaves a couple of big open questions: how much of our LLM-based understanding carries back to other modalities (including vision, video, and world models); and how much of what holds for transformer-based LLMs is universal enough to survive a change in architecture. The modality boundary may already be softer than it seems, as demonstrated by a single "eye" feature that fires on ASCII faces, SVG code, and the written word in various languages. Regardless of how much specific knowledge transfers to other modalities or architectures, the methods and engineering behind interpretability itself should give us a head start there.
Interpretability could transfer superhuman knowledge from models to human scientists — Scientific neural networks encode genuine knowledge in their weights, features, and geometry, but much of that knowledge remains locked away from human eyes. Extracting it could hand researchers in fields like the life sciences and materials science an entirely new source of hypotheses. NLAs already do a version of this for LLMs, translating opaque activations into human-readable text. Extending the same idea to a model like AlphaFold could turn what a protein-structure network "knows" into leads a human researcher can follow. Many labs and startups are already tackling this opportunity, and it's one of the most beneficial uses of interpretability for improving the human condition.
Better interfaces are needed to visualize model internals at scale — Our ability to inspect a model has outrun our ability to display what we find. Attribution graphs are a remarkable advance, but a graph with hundreds of nodes and thousands of edges can be nearly impossible to read. Models are too complex to hold in our heads, so the interfaces we build to explore them shape how much we can actually find. The payoff of better interfaces might extend past research, too, as clearer views of model internals could give policymakers, auditors, and end users something concrete to evaluate when gauging a model's safety.
Finally, here are some predictions for the next few years, offered more to provoke discussion than to be right.
Attention gets its "SAE moment" — Just as sparse dictionary learning cracked open the MLP layers and the residual stream, attention will become legible too. Continued advances in QK attribution and multi-token transcoders will let researchers trace the causal role of attention heads, and parameter decomposition will scale beyond toy models to find interpretable attention mechanisms directly in a model's weights. Once attention is more fully understood, attribution graphs will evolve to show a more complete picture of how mechanistic components work together.
Legibility will race ahead of faithfulness — The appeal of natural-language explanations is too strong to resist, so the field will chase new insights by applying interpreter models more widely. But as explanations grow more legible and human-readable, the faithfulness problem will persist. We will need better ways to prove which mechanisms are actually at work, including new model organisms and more ground-truth tests.
Auto-interpretability will accelerate faster than we imagine — This shouldn't be controversial, given how fast we've gone from tab-complete to agentic coding to autonomous orchestration agents managing swarms of coding agents. And just as the bottleneck moved from writing code to reviewing it, researchers will have to rethink how they supervise results as interpretability agents scale up.
Legible reasoning will be fleeting until new techniques catch up — Whether by accident (like the Mythos Preview error that exposed CoT to reward code, or interpretability results leaking into training data) or by short-sighted choices (like overusing interpretability tools on production runs), models will face optimization pressure toward obfuscated reasoning. Vector-based chain of thought, alien language, and other forms of latent reasoning will have us looking back wistfully on the days when models wrote out their scratchpads in plain English.