Mechanistic interpretability

What Is Mechanistic Interpretability?

Why this matters:

Safety: Understanding failure modes requires knowing the algorithm, not just the activations
Scientific understanding: Neural networks may implement novel algorithms we can learn from
Debugging: Find where computation goes wrong, not just where output is wrong
Alignment: Ensure models reason the way we think they do

Start with a transformer layer computation. Real transformers apply LayerNorm (either pre-norm or post-norm). Writing the common pre-norm form: $h_{\text{mid}} = h_l + \text{Attn}(\text{LN}_1(h_l))$ $h_{l+1} = h_{\text{mid}} + \text{MLP}(\text{LN}_2(h_{\text{mid}}))$

Why include LayerNorm explicitly? The placement of normalization changes where nonlinearities act and is essential to the residual-stream picture — it normalizes what each sublayer reads from the residual stream while the stream itself accumulates raw sums. Dropping it (as a bare $h_{l+1} = h_l + \text{MLP}(h_l + \text{Attn}(h_l))$ ) misrepresents the actual computation.

Now decompose the attention sublayer. Each of the $H$ heads computes a value-weighted sum, then all head outputs are concatenated and a single output matrix $W_O$ is applied: $\text{Attn}(x) = W_O \cdot \text{concat}_{i=1}^{H}\left(\text{softmax}\!\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i\right)$

Equivalently, splitting $W_O$ into per-head blocks $W_O = [W_O^{(1)} \; W_O^{(2)} \; \cdots \; W_O^{(H)}]$ acting on each head's slice, we can write the additive-over-heads form: $\text{Attn}(x) = \sum_{i=1}^{H} W_O^{(i)} \cdot \text{softmax}\!\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i$

Here the $W_O^{(i)}$ are not independent projection matrices — they are column-blocks of the single shared $W_O$ . Why the $1/\sqrt{d_k}$ factor? It rescales the dot-product logits so their variance stays $\approx 1$ regardless of head dimension $d_k$ , keeping the softmax out of saturation. Why this decomposition? Because empirically, individual heads often perform distinct operations (e.g., "previous token head", "induction head").

A circuit is a subset of these heads + MLP neurons that:

Have strong causal effect on the output
Implement a coherent algorithm
Are minimally sufficient (removing components breaks functionality)

2. Activation Patching & Causal Analysis

Why patching over just ablation?

Ablation shows if a component matters
Patching shows what information it computes and where that information flows

Derivation of the patching method:

Consider two inputs $x_{\text{clean}}$ and $x_{\text{corrupted}}$ where the model succeds on clean but fails on corrupted.

Let $a_i^{\text{clean}}$ be the activation of component $i$ on clean input. The patching experiment: $\text{model}(x_{\text{corrupted}}) \quad \text{but with} \quad a_j \gets a_j^{\text{clean}}$

If performance recovers, component $j$ causally mediates the correct computation.

Why this works: If $j$ computes the key information needed for success, transplanting that computation restores functionality.

3. Superposition & Polysemanticity

Derivation from information theory:

Available capacity: $n$ dimensions Required features: $m$ features, each active with probability $s$

If features were orthogonal, we'd need $m$ dimensions. But with sparsity:

Expected simultaneous features: $\approx ms$
If $ms< n$ , we can pack $m > n$ features via superposition

Cost: Interference between features when multiple activate simultaneously.

Mathematical model (simplified toy example):

Optimize feature vectors $\mathbf{v}_i \in \mathbb{R}^n$ to minimize interference: $\min_{\{\mathbf{v}_i\}} \mathbb{E}_{f \sim \text{Sparse}(s)}\left[\left\| \sum_{i=1}^m f_i \mathbf{v}_i - \sum_{i=1}^m f_i \mathbf{v}_i^* \right\|^2\right]$

where $\mathbf{v}_i^*$ are orthonormal basis vectors (the "ideal" non-interfering representation).

Consequence: Individual neurons are polysemantic — they respond to multiple unrelated features.

Key Techniques

Logit Lens & Tuned Lens

Why this works:

In transformers, residual stream $h_l$ accumulates information: $h_L = h_0 + \sum_{l=1}^{L} \Delta h_l$

If information is linearly represented, early layers should already contain partial "answers" readable by the unembedding matrix $W_U$ .

Derivation:

The final prediction is: $p(w) = \text{softmax}(W_U \cdot \text{LN}(h_L))$

For intermediate layer: $p_l(w) = \text{softmax}(W_U \cdot \text{LN}(h_l))$

Comparing $p_l$ to $p_L$ shows how predictions evolve across layers.

Attention Pattern Analysis

Common interpretable patterns:

Previous token head: $A_{i,i-1} \approx 1$
Induction head: $A_{i,j}$ high when token at $j-1$ matches token at $i-1$ (pattern completion)
Duplicate token head: $A_{i,j}$ high when tokens at $i$ and $j$ are identical

Why analyze patterns? Attention is the main mechanism for moving information between positions. The pattern reveals the routing algorithm.

Worked example Worked Example: Finding an Induction Circuit

Task: Understand how a model completes repeated sequences like "A B ... A B"

Step 1: Hypothesis Model uses two heads: one to detect "previous token", one to complete the pattern.

Step 2: Test with activation patching

Input: "cat dog mouse cat dog [?]"
Clean run: Model predicts "mouse"
Corrupt position "cat": Model fails
Patch activations from specific attention head: Performance recovers Why this step? Confirms that head causally mediates the completion.

Step 3: Analyze attention patterns Head1 (layer 1): Position 5 attends to position 4 (previous token) Head 2 (layer 2): Position 5 attends to position 2 (previous occurrence of current token)

Why this step? Shows the algorithm: Head 2 searches for previous "cat", copies what came after it.

Step 4: Verify composition Head 1 output → Head 2 query. Patch this path: circuit breaks.

Result: Identified two-head induction circuit implementing pattern: "Find previous occurrence of current token, copy what followed it."

Worked example Worked Example: Polysemantic Neuron Decomposition

Setup: MLP neuron in vision model activates for both "car wheels" and "text with letter O".

Step 1: Measure activation patterns Neuron fires on: wheels (0.8), letter O (0.7), circles (0.6), ovals (0.5) Why this step? Identify all contexts where neuron is active.

Step 2: Sparse dictionary learning Learn overcomplete basis: $\mathbf{h} = \sum_{i=1}^{m} \alpha_i \mathbf{d}_i$ where $m > n$

Train to minimize: $\|\mathbf{h} - \sum_{i} \alpha_i \mathbf{d}_i\|^2 + \lambda \|\boldsymbol{\alpha}\|_1$

Why L1 penalty? Enforces sparsity — each input uses few dictionary features.

Step 3: Analyze learned features Dictionary feature $d_{23}$ : wheels only (no false positives on text) Dictionary feature $d_{67}$ : round text characters only

Result: Single polysemantic neuron was superposition of two interpretable features.

Why this matters: Can't understand computation by looking at neuron level — need to decompose to feature level.

Common mistake Common Mistake: Confusing Correlation with Mechanism

Wrong approach: "This neuron activates on images of cats, so it detects cats."

Why this feels right: High correlation between neuron activation and cat images.

Why it's wrong:

Correlation doesn't imply the neuron computes cat-ness
Could be spurious correlation (cats often outdoors → neuron actually detects grass)
Doesn't explain HOW the neuron contributes to the final decision

The fix: Use causal methods (activation patching, ablation) to verify the neuron's computational role:

Does ablating it hurt cat classification?
Does patching its activation transfer "cat-ness" to non-cat images?
What information does it compute that downstream layers use?

Steel-man the mistake: Activation patterns are useful starting point for hypothesis generation. The error is stopping there instead of testing causal claims.

Connections to Other Interpretability Methods

vs. Feature visualization:

Feature viz shows WHAT a component responds to
Mechanistic interpretability shows HOW it computes and WHY that matters for the output

vs. Saliency methods:

Saliency shows importance for a single decision
Mechanistic interpretability finds the general algorithm

vs. Probing:

Probing: "Is information X present?"
Mechanistic: "How is X computed and used?"

Applications & Implications

AI Safety: Detect deceptive reasoning circuits before deployment
Capability analysis: Understand limits of what model can learn
Scientific discovery: Networks might implement novel algorithms humans didn't design
Model editing: Surgically modify behavior by targeting specific circuits

Recall Explain to a 12-year-old

Imagine you found a super advanced alien computer that can play chess really well, but you don't understand how it works inside. You have two ways to figure it out: Old way (regular interpretability): Watch it play and notice "oh, when there's a queen on the board, this light blinks." You're just noticing patterns from the outside.

New way (mechanistic interpretability): You carefully open it up and trace the wires. You discover there's a group of parts that work together to "look for pieces that can be captured", another group that "evaluates if the king is safe", and these groups are connected in a specific way to make decisions. You've figured out the actual step-by-step recipe the computer uses!

Mechanistic interpretability is like being a detective who doesn't just watch what the AI does, but figures out the actual "recipe" or "algorithm" it's following, step by step, wire by wire. This is really important because if we're going to trust AI with important decisions, we need to understand not just WHAT it decides, but HOW it thinks!

Connections

Feature Visualization — complementary: shows WHAT, mechanistic shows HOW
Attention Mechanisms — attention patterns are primary object of study
Sparse Autoencoders — tool for decomposing polysemantic neurons
Transformer Architecture — residual stream structure enables mechanistic analysis
Probing Classifiers — related but different: presence vs. computation
Model Editing — applications of mechanistic understanding

#flashcards/ai-ml

What is mechanistic interpretability and how does it differ from post-hoc interpretability? :: Mechanistic interpretability reverse-engineers the learned algorithms neural networks implement by analyzing individual components and circuits. Post-hoc interpretability explains decisions after they're made; mechanistic interpretability explains the computational mechanism that produces decisions.

What is the Circuits Hypothesis?

The hypothesis that neural networks are composed of circuits — sparse subgraphs that implement specific, interpretable algorithms. Each circuit performs a human-understandable computation like edge detection or pattern completion.

Why does the attention score include a 1/√d_k scaling factor?

It rescales the query-key dot products so their variance stays approximately 1 regardless of the head dimension d_k, keeping the softmax out of its saturated regime and preserving useful gradients.

In multi-head attention, is there a separate output matrix W_O per head?

No. The H head outputs are concatenated and a single shared output matrix W_O is applied. The per-head blocks W_O^(i) are just column-slices of that one W_O, which lets you write attention as an additive sum over heads.

What is activation patching and why is it more powerful than ablation?

Activation patching replaces activations from one run with activations from another run to test if behavior transfers. It's more powerful than ablation because ablation only shows IF a component matters, while patching shows WHAT information it computes and WHERE that information flows.

What is superposition and why does it cause polysemanticity?

Superposition is when networks represent more features than they have dimensions by encoding features as non-orthogonal directions. This causes polysemanticity (neurons responding to multiple unrelated features) because features interfere when they're not orthogonal.

What is the logit lens technique?

A technique that projects intermediate layer activations directly to vocabulary logits to reveal what the model "thinks" at each layer. It works because the residual stream accumulates information linearly, so early layers contain partial "answers" readable by the unembedding matrix.

What is an induction head and what algorithm does it implement?

An induction head is an attention pattern that completes repeated sequences. It implements: "Find the previous occurrence of the current token, then copy what followed it." Example: in "A B ... A B", it predicts the next token should be what followed the first "A B".

Why is correlation between neuron activation and a concept insufficient to claim the neuron "detects" that concept?

Correlation doesn't prove the neuron COMPUTES that concept — it could be spurious correlation, and it doesn't explain HOW the neuron contributes to decisions. Must use causal methods (patching, ablation) to verify computational role.

How does dictionary learning help decompose polysemantic neurons? :: Dictionary learning finds an overcomplete basis (more features than dimensions) with L1 penalty for sparsity. This decomposes a single polysemantic neuron into multiple interpretable features that were in superposition, separating out the distinct computations.

What are the three criteria for identifying a circuit in a neural network?

1) Strong causal effect on output (via ablation/patching), 2) Implements a coherent algorithm that's human-understandable, 3) Minimally sufficient (removing any component breaks functionality).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, yaha core idea ye hai ki neural networks ko hum aksar "black box" maante hai — matlab input daala, output aa gaya, par andar kya ho raha hai pata nahi. Traditional interpretability sirf itna batati hai ki "ye neuron cat detect karta hai." Lekin mechanistic interpretability ek step aage jaati hai — ye poochti hai ki network kaunsa algorithm implement kar raha hai. Jaise ek compiled program ko reverse-engineer karke uske actual computational steps samajhna, na ki sirf input-output mapping dekhna. Isko aise socho jaise ek machine ke gears aur wiring khol kar dekhna, sirf uske panel ka reading padhna nahi.

Ab isme sabse important concept hai circuits hypothesis — matlab network ke andar chhote-chhote subgraphs (circuits) hote hai jo specific, samajhne-layak kaam karte hai, jaise "edge detection" ya "previous token copy karna." Transformer me har attention head alag kaam karta hai, aur inko combine karke poora computation banta hai. Yaha LayerNorm aur $1/\sqrt{d_k}$ scaling jaise details important hai kyunki ye actual computation ka hissa hai — inhe ignore karoge to galat picture banegi. Aur activation patching ek tarika hai jisme hum ek component ko ablate (band) karke dekhte hai ki output pe kitna asar padta hai — isse pata chalta hai ki wo component kis kaam ke liye causally responsible hai.

Ye matter kyu karta hai? Kyunki safety ke liye humein pata hona chahiye ki model kyun fail hota hai, sirf kaha fail hota hai nahi. Debugging me bhi ye help karta hai — galti kaha compute ho rahi hai wo pinpoint kar sakte ho. Aur alignment ke liye ye ensure karta hai ki model us tarah reason kare jaisa hum sochte hai, na ki kisi hidden shortcut se. Basically, jitna better hum model ka andar ka mechanism samajhenge, utna zyada bharosa aur control humein powerful AI systems pe milega — jo aaj ke daur me bahut zaroori hai.

What Is Mechanistic Interpretability?

Core Principles & Approach

1. Circuits Hypothesis

2. Activation Patching & Causal Analysis

3. Superposition & Polysemanticity

Key Techniques

Logit Lens & Tuned Lens

Attention Pattern Analysis

Connections to Other Interpretability Methods

Applications & Implications

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Interpretability & Explainability

Connections