Mechanistic interpretability: Visual walkthrough

Before any symbol appears, here is the whole story in words:

We will earn every piece of notation. Let's start with what a token even is.

Step 1 — What the model is actually looking at

WHAT. A model reads text as a list of tokens — think of them as beads on a string, one bead per word-piece. We number the positions $1, 2, 3, \dots$ left to right. Position $i$ just means "the $i$ -th bead".

WHY. Every later idea — "attend back", "previous token", "the repeat" — is about positions on this string. If we don't fix what a position is, none of the arrows we draw later have a place to land.

PICTURE. The sentence cat dog mouse cat dog ? laid out as numbered beads. The last bead, position 6, is the one we must fill in. Notice cat dog appears twice — that repeat is the only clue the model has.

Step 2 — Each position carries a vector: the residual stream

WHAT. At every position the model holds not a word but a vector — a list of numbers, an arrow in a high-dimensional space. Call it $h_{i} \in R^{d}$ : an arrow with $d$ coordinates living at position $i$ . Here $d$ is the model dimension — the fixed width of the residual stream that every position and every layer shares. As the model computes, layers add new arrows onto this one.

WHY. Attention heads don't move words around — they move these arrows. To watch an induction head work we must first see the thing it reads and writes: the residual stream, the running vector at each position.

PICTURE. Each bead now sits under a stack of arrows. The bottom arrow $h^{(0)}$ is the raw token embedding plus its position mark; each layer's contribution $Δ h$ is stacked on top. The final arrow is the sum of everything below it.

Where does the very first arrow $h_{i}^{(0)}$ come from? Two ingredients, added together. First, a lookup: the embedding matrix $W_{E} \in R^{V \times d}$ (with $V$ the vocabulary size from Step 6) has one row per possible token; the row for token $t_{i}$ is that token's meaning-arrow, written $W_{E} [t_{i}]$ . Second, a positional encoding $p_{i} \in R^{d}$ : a fixed arrow that depends only on the slot number $i$ , marking where the token sits. We add them:

Step 3 — Attention: one position reading another

WHAT. An attention head lets position $i$ pull information from earlier positions $j$ . It assigns each pair $(i, j)$ a weight $A_{ij}$ between 0 and 1 — "how much does slot $i$ listen to slot $j$ ?" — and these weights across all allowed $j$ add to 1.

WHY. Nothing else in a transformer moves information between positions. If the answer at position 6 depends on what happened at position 3, an attention head is the only path. So the induction algorithm must be a story about which $A_{ij}$ light up.

A crucial rule — you may only look backward. Because the model predicts the next token, position $i$ is forbidden from peeking at any position $j > i$ (that would be reading the future it is trying to guess). This is the causal mask: before the softmax, every score with $j > i$ is set to $- \infty$ , so its weight becomes exactly $0$ . Only $j \leq i$ survive. That is why the attention grid is always lower-triangular — the top-right half is blank.

PICTURE. A grid: rows are the reading position $i$ , columns the read position $j$ . Cells with $j > i$ are blacked out by the mask. A dark cell means "strong attention". We highlight one row — position 6 — and its arrows reaching back.

Where do $q_{i}$ and $k_{j}$ come from? Each head owns two small matrices, $W_{Q}$ and $W_{K}$ , that squeeze the wide $d$ -dimensional residual arrow down to a narrow head dimension $d_{k}$ (typically $d_{k} = d / H$ , where $H$ is the number of heads). Concretely $q_{i} = W_{Q} h_{i}$ and $k_{j} = W_{K} h_{j}$ , both living in $R^{d_{k}}$ . So $d_{k}$ is not new mystery notation — it is simply the length of the query and key arrows this head works with.

Step 4 — The first worker: the previous-token head

WHAT. One special head has a dead-simple pattern: every position attends to the one just behind it. That means $A_{i, i - 1} \approx 1$ and all other weights $\approx 0$ .

WHY. To later say "A was followed by B", some position must carry a memory of what came right before it. The previous-token head does exactly this: it copies information from slot $i - 1$ into slot $i$ . Now position 5 (the second dog) secretly also knows "the token before me was cat".

HOW can $A_{i, i - 1} = 1$ actually happen? Recall from Step 2 that $h_{i}^{(0)} = W_{E} [t_{i}] + p_{i}$ carries the position mark $p_{i}$ . Suppose that mark grows smoothly with $i$ . The head can learn $W_{Q}$ and $W_{K}$ so that the query at $i$ reads out " $p_{i}$ shifted back by one" while the key at $j$ reads out " $p_{j}$ ". Then $q_{i} \cdot k_{j}$ is maximised exactly when $p_{j} \approx p_{i - 1}$ , i.e. when $j = i - 1$ (and the causal mask already guarantees $i - 1 \leq i$ , so this target is always allowed). The dot product peaks on the immediate-left neighbour and the softmax sharpens that peak to $\approx 1$ . So the stripe is achievable purely from the position marks — no token content needed.

PICTURE. The attention grid for this head is a single off-diagonal stripe (still inside the lower triangle). Below it, arrows show information sliding one step forward: each bead absorbs its left neighbour.

Step 5 — The second worker: the induction head

WHAT. The induction head, sitting in a later layer, does the clever move. At the current position (the second cat, position 4), it looks for an earlier position whose previous token matched the current token, and copies that position's token forward as the prediction.

WHY. This is the pattern-completion itself. We are at cat (position 4). Somewhere earlier, cat also appeared (position 1). What came after that earlier cat? dog. So predict dog. The head implements "find where this happened before, read what followed".

HOW do this head's $W_{Q}$ and $W_{K}$ pull it off? This is the heart of the circuit, so let us trace what each matrix reads out of the residual stream — remembering from Step 2's ordering box that this head reads the summed stream, which already contains Step 4's output.

The query is built at the current position $i$ (the second cat). Its residual arrow $h_{i}$ contains, thanks to the embedding $W_{E} [t_{i}]$ , the identity of the current token cat. The induction head's $W_{Q}$ is learned to extract that current-token identity into the query: $q_{i} = W_{Q} h_{i}$ now encodes "I am looking for a place that was preceded by cat".
The key is built at each candidate position $j$ . But here is the trick: position $j$ 's residual arrow $h_{j}$ was modified by the previous-token head in Step 4 — it now carries the identity of $t_{j - 1}$ , the token that came before $j$ . The induction head's $W_{K}$ is learned to extract that previous-token signal into the key: $k_{j} = W_{K} h_{j}$ encodes "the token before me was $t_{j - 1}$ ".
The match $q_{i} \cdot k_{j}$ is therefore large exactly when "the token I'm looking for" (= $t_{i}$ ) aligns with "the token that preceded $j$ " (= $t_{j - 1}$ ) — that is, when $t_{j - 1} = t_{i}$ .

So the two matrices are not magic: $W_{Q}$ reads the current token out of the embedding part of the stream, $W_{K}$ reads the previous token out of the previous-token-head-modified part of the stream, and the dot product fires when they agree. Without Step 4 having written $t_{j - 1}$ into $h_{j}$ , there would be nothing for $W_{K}$ to read — which is exactly why two heads in ordered layers are required.

PICTURE. Two hops drawn in sequence: the current cat sends a query built from its own token identity; that query matches the key at the position right after the earlier cat (a key carrying "cat came before me"); attention lands there; that position's token dog is copied out.

Step 6 — Reading the answer out: the logit lens check

WHAT. Once the induction head has copied dog's information into position 4's residual stream, we can unembed — multiply by the vocabulary matrix $W_{U}$ — and read the predicted token straight out.

WHY. This is the proof that the circuit did its job. Using the logit lens idea from the parent note, we peek at the residual stream after the induction head and see dog already winning — the answer was assembled by the two heads, not by later machinery.

PICTURE. A bar chart of vocabulary logits after the induction head: dog towers over everything else.

Step 7 — Proving it causally: activation patching

WHAT. Claiming "these two heads do it" is a hypothesis. We test it by patching: run a corrupted input where the model fails, then transplant only the induction head's activation from the clean run — if the correct answer returns, that head caused it.

WHY. Correlation isn't enough (a head could light up without mattering). Patching cuts one wire and reconnects it: if reconnecting this wire alone restores the answer, this wire carried the answer. This is the causal test the parent note demands.

Let us pin down the two symbols this test needs. Let $a_{head}$ be the activation of the induction head — the vector that head writes into the residual stream at the current position (its $Δ h$ contribution from Step 2). We write $a_{head}^{clean}$ for that vector on the input the model gets right, and $a_{head}^{corrupt}$ for the input it gets wrong. Let $Perf (\cdot)$ be the model's performance metric: here, the probability it assigns to the correct next token, $p (‘dog’)$ — a number between 0 and 1, read straight off the logits of Step 6.

PICTURE. Two runs side by side. Corrupted run → wrong prediction. Same run but with the induction head's activation swapped in from the clean run → correct prediction jumps back up. The recovered bar is the causal fingerprint.

The one-picture summary

Everything above collapses into one flow: token string → previous-token head writes each bead's predecessor into it → induction head at the current bead searches (backward only, thanks to the causal mask) for the earlier bead with a matching predecessor → copies what followed it → unembed → prediction.

Recall Feynman retelling — say it back in plain words

Imagine you're reading a story and you hit the word "cat" again. A little clerk (Head 1) has been quietly stamping every word with a sticky note saying "the word before me was ___". So the earlier "dog" is wearing a note that says "cat came before me". Now a second clerk (Head 2) standing at the new "cat" shouts: "Who has a note saying 'cat came before me'?" — and he can only shout backward, never at words to his right (that would be reading the future). The earlier "dog" raises its hand. Head 2 grabs that dog's own word — "dog" — and writes it onto our answer slot. When we finally read the slot out loud (multiply by the read-out matrix $W_{U}$ ), "dog" is the loudest voice. The two clerks work because they act in order and never erase each other: Head 1 stamps first, and because the stream only adds, Head 2 can still read the stamp later. And to be sure the clerks really did it, we cut Head 2's wire, watch the answer break, splice the good wire back, watch it heal — that's how we know these two, and only these two, run the show. If several "cat"s came before, the clerk shares his shout among all of them and blends their followers. On the very first "cat", nobody has a matching note yet, so the clerk stays quiet. Take every repeat away and both clerks have nothing to stamp or shout about, so the whole trick quietly switches off.

Recall

What does the previous-token head write into each position? ::: the identity of the token immediately before it How can a head force $A_{i, i - 1} = 1$ ? ::: by learning $W_{Q}, W_{K}$ so the query reads the position mark $p_{i - 1}$ and matches the key with mark $p_{j}$ only at $j = i - 1$ What is $h_{i}^{(0)}$ made of? ::: the token embedding $W_{E} [t_{i}]$ plus the positional encoding $p_{i}$ What does the causal mask do? ::: forces $A_{ij} = 0$ for all $j > i$ , so each position only attends to itself and earlier positions How does the induction head's $W_{Q}$ and $W_{K}$ create the match? ::: $W_{Q}$ extracts the current token $t_{i}$ into the query, $W_{K}$ extracts the Step-4-written predecessor $t_{j - 1}$ into the key; the dot product fires when $t_{j - 1} = t_{i}$ Why can the later induction head read Step 4's output? ::: layers run in order and the residual stream only adds, so the previous-token head's contribution is still present when the induction head reads the summed stream What condition makes an induction head attend to position $j$ ? ::: when $t_{j - 1} = t_{i}$ (its predecessor equals the current token) and $j \leq i$ What happens on the first occurrence of a token? ::: no candidate satisfies the match, so the induction head finds nothing and contributes no completion signal What happens when a token has many earlier occurrences? ::: the softmax splits attention across all matches, blending their followers (recency usually wins) What are the dimensions of $W_{U}$ ? ::: $V \times d$ , mapping a $d$ -dimensional residual arrow to one score per vocabulary word $V$ What is $Perf$ in the patching effect? ::: the probability the model assigns to the correct next token, read from the logits How do we prove the induction head causally matters? ::: patch its clean activation $a_{head}^{clean}$ into a failing corrupted run; if the correct answer returns, it caused the completion

Related vault notes: Attention Mechanisms, Transformer Architecture, Sparse Autoencoders, Probing Classifiers, Model Editing, Feature Visualization, and the parent Mechanistic interpretability.