Activation patching: Foundations — AI-ML deep dive

This page assumes you have seen none of the notation in the Activation patching parent note. We will build every symbol — $x$ , $f$ , $a^{(ℓ)}$ , $L$ , $Δ$ , $P$ , $lo g$ — from the ground up, in the order they depend on each other. By the end you will be able to read the parent note's formulas as plain English.

0. What is a model, really?

Before any symbol, picture the object we study.

A language model reads some text and produces a guess for the next word. Look at the figure: text goes in on the left, flows rightward through a stack of identical layers (drawn as boxes), and a prediction comes out on the right. Between every pair of boxes there are numbers travelling along the wires — those travelling numbers are the whole subject of this page.

1. Activations — the travelling numbers

This is the star of the show, so it gets its own figure.

Now we decorate the letter $a$ so it can point at an exact spot in the machine.

Definition Component (

$C$ ) and its activation $a_{C}^{(ℓ)}$ A component $C$ is the specific part whose numbers we intend to swap — it can be a whole layer, or one attention head inside a layer, or a single neuron.

The picture: circle a small region inside one box. That circled region is $C$ .
Why we need it: the finer the component, the sharper the claim "this exact piece causes the behaviour." The parent note's "head 9.6" means "component = head number 6 inside layer 9."

Two different subscripts, do not mix them up. When we write $a_{C}^{(ℓ)}$ , the letter $C$ down-right says which part of layer $ℓ$ we mean (a location inside the box). This is a different kind of subscript from the run-label $clean / corrupt$ :

$a_{C}^{(ℓ)}$ — subscript $C$ = which component (where inside the machine).
$a_{clean}^{(ℓ)}$ — subscript $clean$ = which run produced the numbers (a name tag, per the mnemonic above). You can stack both: $a_{C, clean}^{(ℓ)}$ = "the numbers held by component $C$ at layer $ℓ$ , recorded during the clean run." A component subscript answers where; a run subscript answers from which input.

Putting the tags together, $a_{clean}^{(ℓ)}$ reads: "the activation, at layer $ℓ$ , that showed up when we ran the clean input." Three facts stacked on one letter: what (an activation $a$ ), where (layer $ℓ$ , up-right; component $C$ if named), from which run (clean, down-right).

2. The output side — logits, softmax, probabilities, $P$ , $lo g$

The model does not shout a single word. Internally its final box hands out one raw number per candidate word — these raw scores are called logits — and a fixed formula turns those into a confidence score for every possible next word. We need to spell out that pipeline, then the tools ( $P$ , $lo g$ ) that read it.

Definition Logits and softmax — how

$f$ actually produces $P$ What does $f (x)$ return? Not a probability directly. The machine's codomain (its output type) is a list of raw real-number scores, one per candidate word — the logits, written $z = f (x)$ where $z = (z_{1}, z_{2}, \dots, z_{V})$ for a vocabulary of $V$ words. A logit can be any real number (negative, zero, large), so it is not yet a probability.

The picture: the tall/short grey sticks before they are squashed — some poke below zero.
To turn logits into probabilities we apply the softmax formula, which exponentiates and normalises so the results are positive and sum to $1$ : $P (word i ∣ context) = \frac{e ^{z_{i}}}{\sum _{j = 1}^{V} e ^{z_{j}}}$ Here $e^{z_{i}}$ makes every score positive; dividing by the total $\sum_{j} e^{z_{j}}$ makes the whole list add to $1$ . This is the missing link $f \to P$ : first $f$ emits logits $z$ , then softmax turns $z$ into the probabilities $P$ .
Why we need it: patching often measures the change in a specific logit $z_{i}$ (a raw score) as well as in $P$ ; you cannot read the parent note's "targeted logit difference" without knowing $z$ exists and where it sits.

Why do we need the next two tools ( $lo g$ and the minus sign)? Because raw probabilities are awkward to compare: the interesting ones are tiny (like $0.002$ ) and our eyes can't rank tiny numbers well. We want a ruler that stretches out small numbers and turns "more confident" into "smaller score."

Definition Logarithm (

$lo g$ , natural log, base $e$ ) — the tool that stretches small numbers $lo g$ is a function that answers: "what power do I raise the base to, to get this number?" Throughout this note $lo g$ means the natural logarithm, base $e \approx 2.718$ (written $ln$ elsewhere). This is the standard choice in ML because it pairs cleanly with the $e^{z}$ inside softmax; if you instead used base $2$ ( $lo g_{2}$ ), the loss numbers would all be scaled by the constant $1/ ln 2 \approx 1.443$ but every comparison and ratio on this page would be unchanged.

Key behaviour for us: as $P$ shrinks toward $0$ , $lo g P$ dives toward minus infinity; when $P = 1$ , $lo g P = 0$ .
The picture: the curve in the figure — flat near $P = 1$ , plunging as $P \to 0$ .
Why this tool and not just $P$ ? We want a quantity that grows large precisely when the model is badly wrong (assigns tiny probability to the right word). $lo g$ turns the cramped interval $(0, 1)$ into a wide runway we can measure on.

Definition The loss

$L$ as a function (input $\mapsto$ surprise) The fancy curly $L$ (a stylised "L" for loss) is our "how wrong is it" score. Precisely, $L$ is a function that takes a run and returns one number: $L (f (x)) = - lo g P (correct word ∣ x)$ Read the chain of maps left to right: an input $x$ goes into $f$ to make logits, softmax turns those into $P$ , we pick out the correct word's probability, take $lo g$ , and negate. So $L$ maps a run to a surprise value.

Notation shorthands the parent uses. $L (corrupt)$ is just short for $L (f (x_{corrupt}))$ ; likewise $L (clean)$ and $L (patched)$ . The word in brackets names which run's output is being scored — same "name tag" idea as before.
The minus sign flips the diving curve upward, so big $L$ = model is confidently wrong, and $L = 0$ = perfectly certain of the right answer.
The picture: the mirror-image curve in the figure — a "surprise meter." A model surprised by the truth scores high.
Why we need it: patching asks "did behaviour improve?" We need a single number that goes down when the model gets more correct. $L$ is that number.

3. Change and gap — the symbol $Δ$

4. How it all fits together

Three runs, one comparison. The figure shows the clean run (correct answer), the corrupt run (wrong answer), and the patched run — corrupt input, but one component's numbers replaced by the clean ones. We then ask: did the answer snap back toward correct?

The causal fraction the parent uses, $Causal fraction = \frac{L ( corrupt ) - L ( patched )}{L ( corrupt ) - L ( clean )},$ is now readable symbol-by-symbol: top = "how much surprise we removed," bottom = "the total surprise there was to remove," ratio = "what fraction of the mistake this one component explains," a number from $0$ (did nothing) to $1$ (explains everything).

Prerequisite map

Each foundation feeds the next: you cannot judge a patch (need $Δ$ , right branch) without a loss $L$ (needs $lo g$ and $P$ , which needs logits + softmax), and you cannot do a patch without knowing what an activation $a_{C}^{(ℓ)}$ is (left branch).

Where to go next

The big-picture field this lives in: Mechanistic Interpretability.
Sibling causal techniques that share this "intervene and measure" logic: Ablation Studies, Causal Mediation Analysis, Causal Scrubbing.
A read-off tool for the output side: Logit Lens.
Same content in Hinglish: 6.3.09 Activation patching (Hinglish).

Equipment checklist

Cover the right side and see if you can answer each before revealing.

What does the single letter

f

stand for?

The whole model treated as one machine;

f (x)

= "run the model on input

x

What type of thing does

f (x)

actually return, before any probability?

A list of raw real-number scores called logits,

z = (z_{1}, \dots, z_{V})

, one per candidate word.

Which formula turns logits

z

into probabilities

P

Softmax:

P (i) = e^{z_{i}} / \sum_{j} e^{z_{j}}

— exponentiate then normalise so they are positive and sum to 1.

What is an activation?

The list of numbers a component holds on the wires while processing one input.

a^{(ℓ)}

, what does the raised

(ℓ)

mean, and what does it NOT mean?

It means "at layer

ℓ

" (a location). It does NOT mean a power/exponent.

What is the difference between the subscript in

a_{C}^{(ℓ)}

and in

a_{clean}^{(ℓ)}

C

says WHICH component (location inside the box);

clean

says WHICH run produced the numbers (a name tag).

The bar

∣

appears in

P (Mary ∣ ctx)

and in

f (x ∣ patch C)

— same meaning?

No. First = "given that" (conditional probability); second = "run

f

under this intervention". Different jobs, same symbol.

Which base does

lo g

use in this note, and does the base change any conclusion?

Natural log, base

e

. Switching to

lo g_{2}

only rescales every loss by

1/ ln 2

; ratios and comparisons are unchanged.

L

— is it a plain number or a function? Spell out the chain it computes.

A function: input

x \to f \to

logits

\to

softmax

\to P \to lo g \to

negate, giving one surprise number.

What does

L (corrupt)

abbreviate?

L (f (x_{corrupt}))

— the surprise of the corrupt run's output.

In words, what is

Δ_{C}

The drop in loss caused by patching component

C

= (surprise before) − (surprise after).

What does a causal fraction of

1

mean? Of

0

1

= this component fully explains the clean-vs-corrupt difference;

0

= it explains none of it.