Foundations — Mechanistic interpretability

Before you can reverse-engineer a transformer, you need to be able to read every symbol the parent note throws at you. This page builds each one from nothing: plain words → a picture → why the topic needs it. We go in dependency order, so nothing appears before it is earned.

0. Numbers on a line, then in a room

Everything starts with a number: a point on a ruler. Zero in the middle, positives to the right, negatives to the left.

Now put two rulers at right angles. A pair of numbers $(3,2)$ becomes an arrow from the origin to a spot in that flat sheet. That arrow is a vector. Add a third ruler pointing out of the page and you can place arrows in a room; with $n$ rulers you get ${\mathbb{R}}^n$.

Read the figure below: the yellow arrow is the vector $(3,2)$; the blue dashed drop shows its first coordinate (how far right), the pink dashed line shows its second coordinate (how far up). That is all a vector is — a bundle of coordinates drawn as one arrow.

Why the topic needs it. Every activation inside a network — what a layer is "thinking" at one moment — is one such arrow. When the parent says "features are directions in activation space", it means arrows in some ${\mathbb{R}}^n$. Hold that picture: thoughts = arrows in a room.

0b. Adding and subtracting arrows

Before any layer can "add its correction to the belt", we must say what adding two arrows even means.

Read the figure below: the yellow arrow $a$ and blue arrow $b$ join tip-to-tail to make the pale sum $a+b$; the pink arrow shows $a-b$, the correction that carries you from $b$ across to $a$.

Why the topic needs it. The residual stream (Section 6) is nothing but repeated vector addition — each layer's update is an arrow added onto the belt — and a layer update $\Delta h_l=h_l-h_{l-1}$ is a vector subtraction. No addition of arrows, no residual stream.

1. Function and the notation $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$

The whole network is one giant machine $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ — $n$ input numbers (a tokenised sentence turned into numbers), $m$ output numbers (scores for each possible next word).

2. Weight matrices $W$, bias vectors, and matrix multiplication

A network's only real move is: mix the numbers of an arrow to make a new arrow, then nudge it. The recipe for the mix is a grid of numbers called a matrix, written $W$.

The parent's cast of matrices, decoded once so they never scare you again:

You will meet these fully in Attention Mechanisms and Transformer Architecture; here only the shape of the idea matters: a matrix is a fixed mixing table the network learned, optionally followed by a bias nudge.

Why the topic needs it. A "circuit" is literally a chosen handful of these mixing tables acting in sequence. Naming the algorithm = describing what the composition of these tables (plus their biases) does.

3. The dot product and "how aligned are two arrows?"

Read the figure below: the yellow arrow $a$ and the blue arrow $b$ lean the same way, so their dot product is large and positive; the pink arrow $c$ stands almost at a right angle to $a$, so $a\cdot c$ is near zero. The lean between arrows is exactly what the number reports.

Why this tool and not another? Attention must decide how much position $i$ should listen to position $j$. "Listening" = "are their query and key arrows aligned?" The dot product is exactly the arithmetic that answers "how aligned?" in one number — no other basic operation gives alignment so cheaply. That is why the attention formula has $Q_i{K}_i^T$ (a table of every query·key dot product) at its heart — we build $Q_i$ and $K_i$ properly in Section 8.

4. Logits, the number $e$, and softmax — turning scores into a "spotlight"

We have a pile of raw alignment scores (logits). We want them turned into weights that are all positive and sum to 1 — a spotlight that shares 100% of its attention across positions.

Why $e^z$ and not just "positive-ify"? Because of facts (1)–(3) above: the exponential is positive (so weights are valid), monotonic (bigger score → bigger weight) and it stretches gaps (a slightly larger logit grabs a much larger share). This makes softmax behave like a soft, differentiable argmax — it can pick a "winner" position without a hard, non-trainable jump.

Read the figure below: the blue bars are the raw scores (logits, rescaled to fit); the yellow bars are what softmax turns them into. Notice the yellow bars are all positive and sum to $1$, and the tallest blue score grabs a disproportionately large yellow share — that is the exponential exaggerating gaps.

Why the topic needs it. Every attention pattern matrix $A$ is a stack of softmaxes — each row is one position's spotlight over all others. Reading a circuit is reading these spotlights.

5. The $1/\sqrt{d_k}$ scaling — keeping the spotlight sane

$d_k$ is just the length of the key/query arrows (how many numbers each has). When you dot two arrows of length $d_k$ made of random-ish numbers, the result grows roughly like $\sqrt{d_k}$.

6. The residual stream and $\sum$ (the running total)

The parent's most important picture: layers do not overwrite each other. Each layer reads the current arrow, computes a small correction, and adds it back (vector addition, Section 0b). The arrow that flows straight through, accumulating additions, is the residual stream.

$h_L=h_0+{\sum }_{l=1}^L\Delta h_l$

Read it aloud: "the final belt = the starting belt + all the corrections layers wrote onto it."

Read the figure below: the yellow line is one coordinate of the belt across layers; each blue arrow is one $\Delta h_l$ (an update $h_l-h_{l-1}$) stacked on top of the previous total. The belt only ever grows by additions — nothing is erased, which is why an early layer's partial answer is still sitting there at the end.

Why the topic needs it. Because additions accumulate linearly, an early layer's answer is already sitting on the belt — you can read it before the end. That is the whole trick behind the Logit Lens (${\text{logits}}_l=W_U\cdot \text{LN}(h_l)$): apply the final word-reader $W_U$ to a mid-stream arrow and peek at what the model already believes.

7. MLP and LayerNorm — the two sublayers that write to the belt

The parent's block updates use two machines by name, $\text{MLP}$ and $\text{LN}$. Meet them.

Why keep $\text{LN}$ explicit? The parent insists on writing $\text{Attn}({\text{LN}}_1(h_l))$ rather than $\text{Attn}(h_l)$. The reason: each sublayer reads a normalised copy of the belt, while the belt itself keeps the raw running sum. Drop the $\text{LN}$ and you misdescribe what the head actually sees — and your circuit story becomes wrong.

$h_{\text{mid}}=h_l+\text{Attn}({\text{LN}}_1(h_l)),h_{l+1}=h_{\text{mid}}+\text{MLP}({\text{LN}}_2(h_{\text{mid}}))$

Note the residual + in each line — that is the vector addition of Sections 0b and 6 in action: read a normalised copy, compute a correction, add it back onto the belt.

8. Heads, $Q_i/K_i/V_i$, concat, and per-head blocks of $W_O$

Attention runs several separate spotlights in parallel; each spotlight is a head.

Now, where do $Q_i$, $K_i$, $V_i$ come from? Each head owns its own slice of the query/key/value mixing tables — write those slices $W_Q^{(i)},W_K^{(i)},W_V^{(i)}$. Feeding the (normalised) belt arrow $x$ through them (matrix mix plus its bias, Section 2) gives that head's three working arrows:

Each head produces its own little result arrow. The results are glued end-to-end — that is what $\text{concat}$ means (lay the $H$ output arrows in a row to make one long arrow) — and passed through the single shared output table $W_O$:

$\text{Attn}(x)=W_O\cdot {\text{concat}}_{i=1}^H\left(\text{softmax}\right(\frac{Q_i{K}_i^T}{\sqrt{d_k}}\left)V_i\right)$

9. Superposition, sparsity $s$, and polysemanticity

Now the payoff concept. We have $n$ dimensions (rulers) but the network wants to store $m$ features with $m>n$.

$\text{activation}={\sum }_{i=1}^m{f}_i{\mathbf{v}}_i$

Here $f_i$ is how strongly feature $i$ fires (mostly $0$), and ${\mathbf{v}}_i$ is its direction arrow. Because the ${\mathbf{v}}_i$ are not orthogonal, one neuron's axis picks up several unrelated ${\mathbf{v}}_i$ — the neuron becomes polysemantic (responds to multiple meanings). Untangling this back into clean features is the job of Sparse Autoencoders.

10. Causal tools: ablation and patching

Two verbs, one idea — poke the network and watch:

These are the experimental backbone of Probing Classifiers and Model Editing: they turn "I have a hunch this head does X" into a testable causal claim.

Prerequisite map

Every arrow here is a "you need this first" edge. The topic sits at the bottom because it consumes all of these.

Equipment checklist

Cover the right side and answer aloud before revealing.

What is $\Delta h_l$ in terms of the