Worked examples — Mechanistic interpretability

This page is the "run the machine on every kind of input" companion to Mechanistic interpretability. The parent note gave you the tools — activation patching, the circuits hypothesis, superposition, the logit lens. Here we use them on every case class that can come up, so you never meet a scenario you have not seen worked out.

Every symbol used here was defined in the parent note. If we re-introduce one, we re-earn it in plain words first.

The scenario matrix

Think of "mechanistic interpretability in practice" as a small number of distinct situations. Each row below is one case class — a genuinely different kind of question — and each is covered by at least one worked example.

Case A — Patching recovers behaviour

Forecast: guess — is 0.74 "most of the way back"? Roughly what fraction of the gap did we close?

1. Define the gap. The behaviour we care about is the probability of the right token. The full drop caused by corruption is $p_{\text{clean}}-p_{\text{corr}}=0.80-0.10=0.70$. Why this step? Recovery only makes sense relative to how much was lost; the gap is our denominator. (It is positive, so the formula is valid — see the mistake callout above.)
2. Measure how much the patch restored. Restored amount $=p_{\text{patch}}-p_{\text{corr}}=0.74-0.10=0.64$. Why this step? We compare against the corrupted baseline, not zero — the patch's job is to climb back up from the corrupted floor.
3. Recovered fraction. $\frac{0.64}{0.70}=0.9142\dots \approx 91.4\%$. Why this step? A normalised fraction (0–1) is comparable across tasks; a raw probability difference is not.
4. Decide. $91\%$ recovery from a single head is a strong positive — this head causally mediates the completion. (Case A.)

Verify: sanity check the endpoints. If the patch had done nothing, $p_{\text{patch}}=p_{\text{corr}}$ and the fraction would be $0$. If it had fully recovered, $p_{\text{patch}}=p_{\text{clean}}$ and the fraction would be $1$. Our $0.914$ sits sensibly between them. ✓

Case B — The null case: patching does nothing

Forecast: what fraction do you expect when the number doesn't move at all?

1. Gap unchanged: $0.80-0.10=0.70$.
2. Restored amount: $p_{\text{patch}}-p_{\text{corr}}=0.10-0.10=0.00$. Why this step? This is the degenerate input to the recovery formula — the numerator is exactly zero (but the denominator is still the positive $0.70$, so the formula stays valid).
3. Fraction: $\frac{0}{0.70}=0$. Why this step? Zero over a nonzero gap is a clean zero; there is no ambiguity, no division-by-zero.

Interpretation: this head is causally irrelevant to the completion. Do not confuse "we measured it" with "it matters" — the honest answer here is that the component carries none of the needed information. (Case B.)

Verify: the fraction is $0$, matching the endpoint we derived in Example 1's Verify. The two examples share a denominator ($0.70$) and only the numerator changed, which is exactly what "isolate one component" should mean. ✓

Case C — Ablation and patching disagree

Forecast: if a head is important, must patching it recover the behaviour? Guess yes/no.

1. Ablation score (from parent's formula $\text{Effect}(C)=\text{Perf}-{\text{Perf}}_{C\leftarrow \text{baseline}}$): ${\Delta }_{\text{abl}}=0.50$. Large. Why this step? Ablation asks "does removing it hurt?" — here, strongly yes.
2. Patching recovered fraction: $\frac{0.05}{0.70}=0.0714\dots \approx 7.1\%$. Small. Why this step? Patching asks "does its specific value carry the task information across inputs?" — here, barely.
3. Reconcile. The head is doing generic work every input needs (so removing it hurts everything — high ablation), but it is not where the clean-vs-corrupt distinguishing information lives (so transplanting it recovers little — low patch). The answer to the forecast is no: importance ≠ information. (Case C.)

Verify: the two numbers are on different scales by design — $0.50$ is a raw performance drop, $0.071$ is a normalised recovery fraction. They should disagree; that disagreement is the diagnostic, exactly the "Why patching over ablation?" point in the parent note. ✓

Case D — Superposition arithmetic

Forecast: 1000 features into 100 dimensions sounds impossible. Is it — and even if the average is fine, could a rare crowded moment still overflow?

1. Orthogonal budget. If features had to be orthogonal, we'd need one dimension each: $1000>100$ — impossible. This is why we need superposition at all.
2. Expected simultaneous features. From the parent's derivation, the number active at once is $ms=1000\times 0.05=50$. Why this step? Superposition only fails when too many features fire together; the expected co-activation count is what competes for the $n$ directions.
3. Feasibility test (average). Compare $ms$ to $n$: $50<100$, so on average there is room. Headroom $=n-ms=100-50=50$ spare directions. Why this step? $ms<n$ is exactly the packing condition; the margin tells us how comfortably it holds on average.
4. The variance / worst-case burst. The count of active features is a sum of $m$ independent yes/no draws (a binomial), so its variance is $ms(1-s)=1000\times 0.05\times 0.95=47.5$, giving a spread (standard deviation) $\sqrt{47.5}=6.89\dots$. Why this step? The average being safe is not enough — a rare crowded input could still overflow the $n$ dimensions, causing extra interference. We need to know how wide the fluctuations are.
5. How risky is an overflow? Overflow ($\text{count}>n=100$) means being $\frac{100-50}{6.89}=7.25$ standard deviations above the mean. That is astronomically unlikely. Why this step? Expressing the danger threshold in standard deviations tells us whether the headroom is comfortable ($\gg 3\sigma$, safe) or fragile ($\lesssim 2\sigma$, risky). Here $7.25\sigma$ means bursts essentially never overflow.

Interpretation: yes — we can pack $10\times$ more features than dimensions, and not just on average: even the crowded tail of inputs stays under budget because $n$ sits over $7$ standard deviations above the mean load. Shrink $n$ toward $ms$, and this margin collapses — that is the true edge of feasibility. (Case D.)

Verify: units are consistent — $m$ (count) $\times s$ (dimensionless probability) $=$ count $=50$; variance $ms(1-s)$ is also a count${}^2$, so $\sigma$ is a count $=6.89$. And the sanity limit: if $s=1$ then $ms=1000>100$ and packing fails immediately, matching intuition. ✓

Case E — Interference (the cost of D)

What to look at in the figure below: the two black solid arrows are the stored feature directions ${\mathbf{v}}_1$ (along the horizontal axis) and ${\mathbf{v}}_2$ (tilted $60^{\circ }$ up). The black dashed arrow is their sum — the thing actually sitting in activation space when both fire. The red arrow is the readout: the shadow the stored sum casts onto ${\mathbf{v}}_1$'s direction. Notice the red arrow overshoots ${\mathbf{v}}_1$'s tip — that overshoot is the leaked interference from ${\mathbf{v}}_2$.

Forecast: if the vectors were $90^{\circ }$ apart there'd be zero interference. At $60^{\circ }$, more or less than the signal itself?

1. Set up the stored vector. stored $=f_1{\mathbf{v}}_1+f_2{\mathbf{v}}_2={\mathbf{v}}_1+{\mathbf{v}}_2$. Why this step? Superposition stores a sum of active feature directions (parent's ${\sum }_i{f}_i{\mathbf{v}}_i$) — this is the black dashed arrow.
2. Read out feature 1 with the ideal reader ${\mathbf{v}}_1$ (unit): readout $={\mathbf{v}}_1\cdot ({\mathbf{v}}_1+{\mathbf{v}}_2)={\mathbf{v}}_1\cdot {\mathbf{v}}_1+{\mathbf{v}}_1\cdot {\mathbf{v}}_2$. Why this step? The dot product is the natural "how much of this direction is present?" measure — it is the length of the red projection arrow. Splitting the sum lets us see the true signal and the leak as separate terms.
3. Evaluate. ${\mathbf{v}}_1\cdot {\mathbf{v}}_1=1$ (unit vector, the true signal). ${\mathbf{v}}_1\cdot {\mathbf{v}}_2=\cos 60^{\circ }=0.5$ (the leaked interference). Why this step? The dot product of two unit vectors is the cosine of their angle — this is the exact reason non-orthogonality causes leakage, and why the red arrow is longer than ${\mathbf{v}}_1$.
4. Report. Signal $=1.0$, interference $=0.5$. The readout is $1.5$: the true $1$ plus half of feature 2 bleeding in. Why this step? Separating the number into "what we wanted" ($1.0$) and "what leaked" ($0.5$) is the whole point — it quantifies the tax, so a downstream layer (or an SAE) knows how much noise to expect.

Interpretation: interference ($0.5$) is smaller than the signal ($1.0$) but not negligible — this is the tax superposition pays. Sparsity (case D) keeps such collisions rare. (Case E.)

Verify: if the angle were $90^{\circ }$, $\cos 90^{\circ }=0$ and interference vanishes (orthogonal = no cost). If $0^{\circ }$, $\cos 0^{\circ }=1$ and the features are indistinguishable — maximal collision. Our $60^{\circ }$ gives the intermediate $0.5$. ✓

Case F — Logit lens across layers (limiting behaviour)

Two symbols from the parent note appear here; we re-earn them in plain words before using them:

Forecast: does the correct answer appear suddenly at the last layer, or grow gradually?

1. Layer 0: $p_0=\frac{e^{-1}}{e^{-1}+1}=\frac{1}{1+e^1}=0.2689\dots$ Why this step? Two-outcome softmax collapses to the logistic sigmoid $\sigma (\ell )=1/(1+e^{-\ell })$ — the cleanest way to turn one relative logit into a probability.
2. Layer 2: $p_2=\sigma (0)=0.5$. The model is undecided. Why this step? A logit of $0$ means the target and competitor are tied; the sigmoid must return exactly $0.5$, which is our anchor point for reading the trajectory — anything above it means "leaning correct".
3. Layer 4: $p_4=\sigma (2)=\frac{1}{1+e^{-2}}=0.8807\dots$ Why this step? We reuse the same sigmoid because the readout mechanism ($W_U\cdot \text{LN}$) is identical at every layer — only the residual-stream vector $h_l$, hence the logit, has changed. Comparing layers is therefore an apples-to-apples read of accumulated evidence.
4. Layer 6 (final): $p_6=\sigma (4)=\frac{1}{1+e^{-4}}=0.9820\dots$ Why this step? This is the model's actual output layer, so $p_6$ is the real prediction; computing it lets us see how far the earlier layers were from the finished answer.

Trajectory / limiting behaviour: $0.27\to 0.50\to 0.88\to 0.98$ — the correct token is built up gradually across the residual stream, exactly the parent's $h_L=h_0+{\sum }_l\Delta h_l$ picture: each layer adds a little more evidence. In the limit $\ell \to +\infty$, $\sigma \to 1$; as $\ell \to -\infty$, $\sigma \to 0$. (Case F.)

Verify: monotonic increase ✓ (each layer's probability exceeds the previous). $\sigma (0)=0.5$ exactly ✓. And $\sigma (-\ell )=1-\sigma (\ell )$: e.g. $\sigma (-1)=1-\sigma (1)$, which pins layer 0. ✓

Case G — Classify the attention head

What to look at in the figure below: each grid is one pattern matrix. Rows are query positions (the token doing the looking), columns are key positions (the token being looked at); a red cell marks weight $1$ (all the attention), a white cell weight $0$. In the left grid the red cells sit just below the diagonal — every token looks one step back. In the right grid the red cells jump two columns back to the matching letter — each A looks at the earlier A, each B at the earlier B. The shape of the red pattern is the head's fingerprint.

Forecast: one of these is a "previous token head". Which one, and what is the other?

1. Test $A^{(1)}$ for previous-token behaviour: parent's signature is $A_{i,i-1}\approx 1$. Check row 3: attends to position 2 ($=i-1$). Row 4: attends to position 3 ($=i-1$). ✓ (Rows 1–2 hit position 1 since there's nothing earlier.) So $A^{(1)}$ is a previous-token head — the sub-diagonal red band in the left grid. Why this step? Each interpretable head has a fingerprint in the pattern matrix; we match the red pattern against the parent's catalogue.
2. Test $A^{(2)}$ for duplicate-token behaviour: signature is high weight where tokens match. Tokens are A B A B. Row 3 (token A at pos 3) attends to position 1 (the other A). Row 4 (token B at pos 4) attends to position 2 (the other B). ✓ So $A^{(2)}$ is a duplicate-token head — the "two-back to the matching letter" red pattern in the right grid. Why this step? The duplicate signature is what the figure's rightward-then-back red cells reveal; it is the ingredient an induction circuit uses.
3. Conclusion: $A^{(1)}$ = previous-token head, $A^{(2)}$ = duplicate-token head. Together they are the two halves of an induction circuit. (Case G.)

Verify: every row sums to 1 — valid softmax output. For $A^{(1)}$, the "attends to $i-1$" pattern means the weight sits just below the diagonal (a sub-diagonal band), which is exactly what the figure shows. For $A^{(2)}$, weights land on matching tokens two positions back — the hallmark the parent listed. ✓

Case H — Real-world word problem (end to end)

Forecast: which single head should recover more — the previous-token head or the induction head?

1. Gap: $0.90-0.20=0.70$. Why this step? This positive gap is the denominator for every recovery below; corruption genuinely broke the task (from $0.90$ down to $0.20$), so the experiment is valid.
2. Induction head alone: $\frac{0.76-0.20}{0.70}=\frac{0.56}{0.70}=0.80$ → $80\%$. Why this step? The induction head is what copies the completion; expect it to carry most of the signal.
3. Previous-token head alone: $\frac{0.34-0.20}{0.70}=\frac{0.14}{0.70}=0.20$ → $20\%$. Why this step? It only supplies the key the induction head reads; alone it can't finish the job.
4. Both together: $\frac{0.88-0.20}{0.70}=\frac{0.68}{0.70}=0.9714\dots$ → $97.1\%$. Why this step? The circuit is the composition; patching both should approach full recovery.
5. Ranking: induction head ($80\%$) > previous-token head ($20\%$), and together $\approx 97\%$ — the two compose into the circuit the parent described. Forecast answer: the induction head recovers more.

Verify: partial recoveries need not add to the joint recovery (circuits are not linear): $80\%+20\%=100\%$ but the joint is $97\%$ — close, showing near-additivity with mild redundancy, which is physically sensible for a two-head circuit. All fractions lie in $[0,1]$. ✓

Case I — Exam twist: the $1/\sqrt{d_k}$ trap

Forecast: does forgetting the $\sqrt{d_k}$ divide make the attention sharper or softer?

1. Correct (scaled) logit: $\frac{s_{\text{raw}}}{\sqrt{d_k}}=\frac{32}{\sqrt{64}}=\frac{32}{8}=4$. Why this step? Parent note: the factor keeps logit variance $\approx 1$ regardless of $d_k$, so softmax stays out of saturation.
2. Scaled weight: $p=\sigma (4)=\frac{1}{1+e^{-4}}=0.9820\dots$ — confident but not saturated.
3. Unscaled logit: the student keeps $32$. Weight $=\sigma (32)=\frac{1}{1+e^{-32}}\approx 0.99999999999998\dots \approx 1$. Why this step? Without the divide the logit is $8\times$ larger, pushing the softmax into full saturation.
4. The trap: the unscaled softmax is a near-hard argmax ($\approx 1$), giving vanishing gradients and brittle attention; the scaled one ($0.982$) is trainable. Same head, wrong by the missing $\sqrt{64}=8$. (Case I.)

Verify: the ratio of logits is exactly $\sqrt{d_k}=8$ ($32/4=8$) ✓. Scaled $\sigma (4)=0.982$ matches Example 6's layer-6 value (same logit) ✓ — a nice cross-check that our sigmoid is consistent.

Recall

See also: Sparse Autoencoders (the modern tool for undoing superposition, Cases D–E), Attention Mechanisms and Transformer Architecture (Cases G, I), Probing Classifiers and Feature Visualization (reading what a direction means), Model Editing (acting on a located circuit), and the parent Mechanistic interpretability.