Activation patching: Visual walkthrough — AI-ML deep dive

This page builds the central result of Activation patching — the causal fraction that tells you how much one component inside a network is responsible for a behaviour — starting from a picture of a machine with a broken part, and ending with a single number between 0 and 1.

No formula is used before it is drawn. If you have never seen a neural network's insides, start here at line one.

Step 1 — What is an "activation"? Draw the machine

WHAT. A neural network is a stack of layers. A piece of text goes in the bottom; at the top the model produces a guess for the next word. Between the layers, numbers flow. The bundle of numbers sitting at one specific place (one layer, one attention head, one neuron) at one moment is called an activation.

WHY start here. Every symbol later — $a$ , $C$ , $ℓ$ — is just a name for one of these numbers-in-a-pipe. If you don't see the pipe, the symbols float free.

PICTURE. Think of the network as water pipes. Each junction carries a value. We circle one junction and call it component $C$ at layer $ℓ$ . The number flowing through it right now is its activation $a_{C}^{(ℓ)}$ .

Step 2 — Two runs: the clean world and the corrupted world

WHAT. We run the machine twice on two nearly-identical inputs.

$x_{clean}$ — the input where the model answers correctly.
$x_{corrupt}$ — a tiny edit that makes the model answer wrongly.

WHY two runs and not one? A single run tells you what the model did, never why. We need a contrast: two worlds that differ in exactly the thing we're studying. The difference between them is the entire signal.

PICTURE. Same machine, twice. Left pipe glows correct; right pipe glows corrupted. Every junction now holds two possible numbers: $a_{clean}^{(ℓ)}$ and $a_{corrupt}^{(ℓ)}$ .

Step 3 — Measure "how wrong" with a loss

WHAT. We need a single number for how far an answer is from correct. Call it the loss $L$ . The standard choice is $L = - lo g P (right answer)$ .

WHY the negative log, and why this tool? We want a number that is small when the model is confident and right, and large when it is wrong. Probability $P$ lives in $[0, 1]$ . Taking $- lo g$ of it flips and stretches:

$L = - lo g P (correct token)$

$P = 1$ (certain, correct) $\Rightarrow - lo g 1 = 0$ — zero loss.
$P \to 0$ (model ignores the right answer) $\Rightarrow - lo g (tiny) \to + \infty$ — huge loss.

That is exactly the "how wrong" ruler we wanted: monotone, bottoming out at 0.

PICTURE. A curve diving from $+ \infty$ down to $0$ as probability climbs from 0 to 1. We mark two dots on it: a low dot for the clean run, a high dot for the corrupted run.

Step 4 — The intervention: transplant the clean part

WHAT. Run $x_{corrupt}$ again, but at the single component $C$ overwrite its corrupted value with the clean value we saved in Step 2:

$y_{patched} = f (x_{corrupt} ∣ a_{C}^{(ℓ)} = a_{clean}^{(ℓ)})$

WHY the vertical bar? The bar " $∣$ " reads "given that / forced to". Everything to its right is a surgical override: run the corrupted input normally, but pin component $C$ to the clean number. Everything else in the machine keeps its corrupted values.

PICTURE. The broken machine, with one pipe re-coloured to its clean value — a transplanted organ. Downstream pipes now carry a third, new set of numbers, because the change ripples forward.

Reading each symbol:

$f (\cdot)$ — the whole network as a function: text in, answer out.
$x_{corrupt}$ — the input we actually feed.
$a_{C}^{(ℓ)} = a_{clean}^{(ℓ)}$ — the override: this pipe is forced clean.
$y_{patched}$ — the resulting answer, a hybrid of both worlds.

Step 5 — Read off the restoration

WHAT. Put the patched answer's loss $L (y_{patched})$ back on our ruler from Step 3.

WHY. Its position tells the story:

If it lands near the clean dot → transplanting $C$ fixed the machine → $C$ was doing the work.
If it stays near the corrupted dot → $C$ was irrelevant to this behaviour.

PICTURE. The same loss curve, now with a third dot for the patched run sliding down from the corrupted height toward the clean height. The vertical drop is what we care about.

Step 6 — Turn the drop into a fraction

WHAT. Compare how far we dropped to how far there was to drop.

$Causal fraction = \frac{L ( y _{corrupt} ) - L ( y _{patched} )}{L ( y _{corrupt} ) - L ( y _{clean} )}$

WHY a ratio and not a raw drop? A raw drop of "2.0" means nothing unless you know the gap. Dividing by the full gap normalises: the answer is a fraction of the whole distance, so it compares fairly across different tasks and losses.

Term-by-term:

Numerator $L (y_{corrupt}) - L (y_{patched})$ — distance actually recovered by the transplant.
Denominator $L (y_{corrupt}) - L (y_{clean})$ — the total gap from Step 3.
Ratio — "what share of the damage did this one part repair?"

PICTURE. A vertical bar spanning corrupted-to-clean. The recovered piece is shaded red; the fraction is red-length over full-length.

Step 7 — The edge cases (never leave the reader stranded)

Every fraction has boundaries. Walk all of them so no result surprises you.

Case A — Fraction $= 1$ (full explanation). Patched loss lands exactly on clean. Numerator equals denominator. The component fully accounts for the difference.

Case B — Fraction $= 0$ (irrelevant). Patched loss equals corrupted loss; nothing moved. $C$ plays no causal role here.

Case C — Fraction $< 0$ (it got worse). Possible! The clean value, dropped into a corrupted context, can confuse the model further. A negative fraction is a real, informative outcome — the component interacts destructively.

Case D — Fraction $> 1$ (overshoot). The patch pushed loss below clean. Rare, but happens when the clean activation is unusually helpful in the corrupted context. Not a bug — a hint of interaction.

Case E — Denominator $\approx 0$ (degenerate contrast). If clean and corrupted losses are almost equal, the gap is tiny and the fraction explodes / is undefined ( $\frac{small}{0}$ ). Meaning: your two inputs don't actually differ in behaviour — the experiment has no signal. Fix the inputs, not the metric.

PICTURE. One number line for the fraction with all five regions flagged: the healthy $[0, 1]$ band, the $< 0$ danger zone, the $> 1$ overshoot, and a crossed-out " $0/0$ " marker where the contrast vanishes.

The one-picture summary

Everything above collapses into one diagram: two worlds, one transplant, one ruler, one fraction.

Recall Feynman retelling — say it out loud

I have a machine that answers right on one sentence and wrong on a barely-different one. I want to know which inner part is doing the answering. So I take a snapshot of that part while the machine is working, then I break the machine with the bad sentence, and I glue the good snapshot of that one part back in. I look at my "how-wrong" ruler — a number that's zero when perfectly right and huge when wrong. If gluing the part in slides me most of the way from wrong back to right, that part was the cause: I write the share of the distance I recovered as a fraction. One means "that part explains everything," zero means "that part is a bystander," negative means "that part actually hurt here," and if my two sentences behave the same to begin with, there's no distance to explain and the fraction is meaningless — I need better sentences.

Recall Quick self-test

What does the denominator of the causal fraction represent? ::: The total behavioural gap between the corrupted and clean runs — the full distance available to close. A causal fraction of $0.9$ means what in plain words? ::: Transplanting that one component repaired 90% of the damage caused by corrupting the input. Why divide by the gap instead of reporting the raw loss drop? ::: To normalise, so results are comparable across tasks and loss scales — the answer becomes a share of the whole. When is the causal fraction undefined? ::: When clean and corrupted losses are nearly equal (gap $\approx 0$ ), so the contrast has no signal.

See also: Mechanistic Interpretability · Logit Lens · Ablation Studies · Causal Scrubbing · Causal Mediation Analysis