Exercises — Mechanistic interpretability

(a) is traditional / post-hoc interpretability — it names what activates a unit but not how the computation runs. (b) is mechanistic interpretability — it describes the algorithm (a chain of composed steps), not just the input→output label. The tell: mechanistic answers always describe a process with steps, traditional answers describe a label.

They add. That is the meaning of the residual ("skip") connection, with ${\text{LN}}_1$ normalizing what Attention reads and ${\text{LN}}_2$ normalizing what the MLP reads: $h_{\text{mid}}=h_l+\text{Attn}({\text{LN}}_1(h_l)),h_{l+1}=h_{\text{mid}}+\text{MLP}({\text{LN}}_2(h_{\text{mid}})).$ Each block writes an increment $\Delta h$ into the shared notepad; nothing is overwritten. This is exactly why we can read partial answers early (the Logit Lens works because the stream is a running sum).

A previous-token head. Reading the matrix: row $i$ (the query position) puts nearly all its weight on column $i-1$ (the key position one step back).

The attention logit is scaled by $1/\sqrt{d_k}$: $\text{logit}=\frac{q\cdot k}{\sqrt{d_k}}=\frac{32}{\sqrt{64}}=\frac{32}{8}=4.$ Why divide? Without scaling, dot products grow like $\sqrt{d_k}$ and push softmax into saturation (all weight on one token, zero gradient). Dividing by $\sqrt{d_k}$ keeps the logit variance near $1$.

Using the parent-note formula: $\text{Effect}(C)=\text{Perf}(\text{model})-\text{Perf}({\text{model}}_{C\leftarrow \text{baseline}})=0.90-0.30=\mathrm{0.60.}$ A large positive effect ($0.60$ out of a possible $0.90$) means $C$ carries most of the load for this behaviour.

Expected number of simultaneously active features is $\approx ms$. The bound $ms<n$ gives $m<\frac{n}{s}=\frac{100}{0.02}=5000.$ So the largest integer is $m=4999$. The network can represent far more features than dimensions ($4999\gg 100$) precisely because features rarely fire together.

Attention adds the sum over heads: $\text{Attn}={\Delta }_1+{\Delta }_2=(2-3,-1+4)=(-1,3).$ Then $h_{\text{mid}}=h_l+\text{Attn}=(1,1)+(-1,3)=(0,4).$ In the second coordinate, head 2 contributes $+4$ vs head 1's $-1$, so head 2 dominates the second coordinate's change. This linearity — heads add independently into the stream — is exactly why we can attribute credit per head.

The Logit Lens logit of a token is $(W_U\text{LN}(h_l)+b_U)$ for that token's row; here $b_U=0$, so it reduces to the dot product of the row with $h$: ${\ell }_{\text{cat}}=(1,1,0)\cdot (1,0,2)=1,$ ${\ell }_{\text{dog}}=(0,0,1)\cdot (1,0,2)=2,$ ${\ell }_{\text{fish}}=(2,0,1)\cdot (1,0,2)=2+0+2=4.$ Largest is ${\ell }_{\text{fish}}=4$, so the Logit Lens predicts "fish" at layer $l$. This is the whole trick: because the stream is a running sum and $W_U$ is linear, we can read a partial prediction from any layer, not just the last.

Index the tokens $1=A,2=B,3=C,4=A,5=B$.

• Previous-token head (P): the query at position 5 attends to position $5-1=4$ (the token "A"). Rule: $A_{i,i-1}\approx 1$.
• Induction head (Q): it looks for the earlier place where the previous token matched. The current previous token is "A" (position 4). The earlier "A" is at position 1, and induction attends to the token after that match — position 2 ("B") — because that is what should come next. So Q at position 5 attends to position 2. Together they implement "A B … A → predict B": the previous-token head lays down the "what came before me" tag, and the induction head reads it to complete the pattern.

(i) Inputs. Clean $x_{\text{clean}}=$ "cat dog mouse cat dog", on which the model correctly predicts "mouse". Corrupted $x_{\text{corrupted}}=$ the same string with an early token swapped so the pattern breaks — e.g. "cat dog mouse fox dog", on which the model fails to predict "mouse". (ii) Patch. Run the model on $x_{\text{corrupted}}$, but overwrite the activation of the single head $H^{\ast }$ with its value from the clean run: $a_{H^{\ast }}\leftarrow a_{H^{\ast }}^{\text{clean}}$. Leave all other activations at their corrupted values. (iii) Criterion. If performance on the corrupted run recovers toward "mouse" only when we patch $H^{\ast }$ (and stays broken when we patch unrelated heads), then $H^{\ast }$ causally mediates the completion. The formal quantity is the recovered fraction $({\text{Perf}}_{\text{patched}}-{\text{Perf}}_{\text{corrupt}})/({\text{Perf}}_{\text{clean}}-{\text{Perf}}_{\text{corrupt}})$; near $1$ means full mediation. Why patch, not just ablate? Ablation would only show that $H^{\ast }$ matters. Patching shows that the specific information $H^{\ast }$ computes on the clean input is what restores the behaviour — that is a claim about what it computes and where it flows.

$\text{recovered}=\frac{{\text{Perf}}_{\text{patched}}-{\text{Perf}}_{\text{corrupt}}}{{\text{Perf}}_{\text{clean}}-{\text{Perf}}_{\text{corrupt}}}=\frac{0.80-0.20}{0.95-0.20}=\frac{0.60}{0.75}=\mathrm{0.8.}$ The head restores $80\%$ of the lost performance — strong (but not total) causal mediation. The missing $20\%$ says other components share the computation; the head is important but not minimally sufficient on its own.

With 3 unit vectors in a plane, the symmetric arrangement that maximizes the minimum angle between them spreads them evenly around the circle: consecutive angle $\theta =\frac{360^{\circ }}{3}=120^{\circ }.$ The interference between any two is their dot product: ${\mathbf{v}}_i\cdot {\mathbf{v}}_j=\cos 120^{\circ }=-\frac{1}{2}.$ Look at the figure: three arrows at $120^{\circ }$ form a "Mercedes star." Note the dot product is negative — the vectors are more than $90^{\circ }$ apart, which is the best you can do when forced to overpack: you cannot make them orthogonal (that needs $n\ge 3$), so the optimizer settles for maximally spread, mildly anti-correlated directions. This is superposition made visible.

The stream is a running sum: $h_L=h_0+{\sum }_{l=1}^L\Delta h_l.$ The Logit Lens reads, from a partial sum $h_l=h_0+{\sum }_{k\le l}\Delta h_k$, the logits ${\text{logits}}_l=W_U\text{LN}(h_l)+b_U,$ i.e. it first applies LayerNorm to the intermediate stream, then the unembedding matrix $W_U$ and its bias $b_U$. Early layers have only added a few $\Delta h_k$ increments, so this readout reflects incomplete computation. A later head or MLP can add a $\Delta h_k$ that reverses the leading logit — e.g. an induction head that only fires in a mid layer supplies the token that outranks an early guess. So a flip is the signature of a component doing real work late in the stack. It is expected because prediction is accumulated, not decided at layer 1. (This is also why the Tuned Lens exists: a learned affine map per layer corrects for the fact that early $h_l$ live in a slightly different basis than $W_U$ expects.)

(a) $m<n/s=64/0.01=6400$, so $m_{\max }=6399$ features. (b) You never reach the ceiling cleanly because the moment two features co-activate their non-orthogonal directions ${\mathbf{v}}_i,{\mathbf{v}}_j$ interfere (${\mathbf{v}}_i\cdot {\mathbf{v}}_j\ne 0$), corrupting the decoded values; the more you pack, the larger and more frequent this interference, until decoding error outweighs the extra capacity. (Sparse autoencoders — see Sparse Autoencoders — are the tool that un-mixes these overlapping directions back into interpretable features.)

• The residual stream is a running ::: sum (components add increments, never overwrite)
• Scale attention logits by ::: $1/\sqrt{d_k}$ to keep softmax out of saturation
• Effect of a component ::: clean performance minus ablated performance
• Standard patching direction ::: clean activations transplanted into the corrupted run
• Superposition packing bound ::: $ms<n$ (expected active features below dimension count)
• Cost of superposition ::: interference / polysemanticity when features co-activate