Activation patching: Question bank — AI-ML deep dive

This is a rapid-fire trap deck for Activation patching. Every item hides a misconception the technique invites. Read the prompt, answer out loud in one sentence, then reveal. If your gut said "yes/no" without a because, you fell in the trap.

Before the traps, four definitions the deck leans on. Read them once so every symbol below is earned.

Definition The model

$f$ and its inputs Write == $f$ == for the model itself — the function that takes a text input and returns the network's output (the probabilities it assigns to the next token). So $f (x)$ just means "run the model on input $x$ and read the answer." Two special inputs recur throughout: the ==clean input $x_{clean}$ ==, a prompt on which the model behaves correctly (e.g. "When Mary and John went to the store, John gave a drink to ___" → predicts "Mary"), and the ==corrupted input $x_{corrupt}$ ==, a minimally changed prompt on which it behaves wrongly (e.g. swap "John" for "Susan" so it predicts "Susan"). We write == $y_{clean} = f (x_{clean})$ and $y_{corrupt} = f (x_{corrupt})$ for the two outputs, and $y_{patched}$ == for the output of the corrupted run after we splice in a clean activation. So $y$ is always shorthand for "whatever $f$ produced," and $L (y)$ scores that output.

Definition Component, and how patching adapts to its type

A component is any addressable piece of the network you can read and overwrite: a single neuron (one number in one layer), an attention head (its output vector at a token position), or a whole layer. Here a layer's activation just means the running summary vector the network carries from token to token — the state each layer reads, updates, and passes on (transformer folk call this the residual stream, but you only need "the running state at that layer"). Patching works the same way for all three — save the activation, then overwrite it — but the shape of what you swap differs: a neuron swap moves one scalar, a head swap moves a vector, a layer swap moves the entire running state. Bigger components carry more information, so they tend to show larger effects and less specificity.

Definition The patch effect

$Δ_{C}$ Pick a scalar ==metric $L$ == that measures distance from the clean answer (e.g. the loss $- lo g P (correct token)$ , or a targeted logit difference). For a component $C$ , the patch effect is how much the metric improved when we injected $C$ 's clean activation into the corrupted run: $Δ_{C} = L (f (x_{corrupt})) - L (f (x_{corrupt} ∣ patch C))$ Why this difference captures causal effect: both terms run the same corrupted input, so everything is held fixed except that in the second term we forced $C$ to carry its clean value. Any change in the metric can therefore only be blamed on $C$ — that is precisely what "causal effect of $C$ " means. So $Δ_{C} > 0$ means patching reduced the metric (moved toward clean); $Δ_{C} < 0$ means it made things worse. $Δ_{A \cup B}$ is the same quantity when you patch $A$ and $B$ together.

Definition Causal fraction and the clean-to-corrupt gap

The ==clean-to-corrupt gap $G$ == is the total distance the corrupted run sits from clean under the chosen metric: $G = L (f (x_{corrupt})) - L (f (x_{clean}))$ Why this is "all the distance there is to close": $L (y_{corrupt})$ is how bad we start and $L (y_{clean})$ is the best we could hope for, so their difference is the entire budget any patch could possibly recover. The causal fraction normalizes a patch effect by that budget: $Causal fraction = \frac{Δ _{C}}{G} = \frac{L ( y _{corrupt} ) - L ( y _{patched} )}{L ( y _{corrupt} ) - L ( y _{clean} )}$ Why it usually lands in $[0, 1]$ : the denominator is all the closable distance; a patch that closes none gives $0$ , one that fully restores clean behavior ( $L (y_{patched}) = L (y_{clean})$ ) gives exactly $1$ . Values below $0$ (patch made it worse) or above $1$ (patch overshot the clean baseline) are possible and are read as warning signs, not extra importance.

The picture below fixes the whole vocabulary — the three runs, the two baselines, the gap $G$ , and where the patch effect $Δ_{C}$ lives.

True or false — justify

True or false: A component that activates strongly on the relevant token is therefore causally responsible for the behavior.

False — strong activation is correlation; it may be epiphenomenal (driven by computation elsewhere) or redundant, and only patching reveals whether removing it actually breaks behavior.

True or false: Attention weights alone can tell us which head is doing the work.

False — attention shows where a head looks, not whether the model uses that information; only an intervention like patching tests the causal link.

True or false: If patching a component restores clean behavior, that component is causally involved in producing that behavior.

True — restoration under intervention is exactly the causal signal, since we changed only that component and the output moved toward clean.

True or false: A causal fraction of 1.0 means the patched component is the only thing that matters.

False in general — it means that component's clean activation is sufficient to close the gap

G

here, but redundant paths could also each score high, so sufficiency is not exclusivity.

True or false: Patching always moves the corrupted output toward the clean output.

False — if the component is irrelevant, the metric barely changes (

Δ_{C} \approx 0

); and a badly chosen patch site can even distort behavior in neither direction cleanly.

True or false: Cross-entropy loss is a safe default metric for any patching experiment.

False — it averages over all tokens, so it can dilute or hide the specific capability you care about; the metric

L

must target the exact behavior (e.g. logit difference between the two candidate tokens).

True or false: Activation patching and ablation are the same intervention.

False — ablation destroys an activation (zeroing or mean-replacing it), while patching replaces it with a meaningful counterfactual value from another run, so patching tests restoration, not just removal.

True or false: Patching is a form of causal mediation analysis applied to internal activations.

True — it treats the activation as a mediator and measures how routing a counterfactual value through it changes the outcome, which is exactly mediation.

Spot the error

"I patched all 144 attention heads at once and got a huge effect, so the model's IOI (Indirect Object Identification — completing 'John gave a drink to ___' with the right name) ability lives in the attention heads." — what's wrong?

Patching everything at once cannot decompose which heads matter; redundant heads inflate the joint effect

Δ_{A \cup B}

, so you must patch individually first, then test groups for synergy.

"Neuron 472 has 90% selectivity for pronouns, so it resolves pronouns — no need to patch." — what's wrong?

Selectivity is a correlational property; without patching you cannot rule out that pronoun resolution happens elsewhere and neuron 472 is a downstream side-effect (epiphenomenal).

"I'm studying toxicity, so I patched a layer and measured drop in overall accuracy." — what's wrong?

Overall accuracy is a generic metric; it can stay flat while the toxic-token probability shifts, so you need a targeted metric

L

like the logit difference between toxic and safe completions.

"My causal fraction came out to 1.4, which means the component is super important." — what's wrong?

A fraction above 1 means

Δ_{C}

overshot the gap

G

(the patch beat the clean baseline); this usually signals a metric or normalization problem, or an unstable corrupted baseline, not extra importance.

"Both head A and head B individually gave

Δ = 0.05

, so together they must give about

0.10

." — what's wrong?

Effects are not additive when components interact;

Synergy (A, B) = Δ_{A \cup B} - (Δ_{A} + Δ_{B})

can be strongly positive (collaboration) or negative (redundancy), so you must patch the pair directly.

"I got zero effect patching this head, so it is useless to the model." — what's wrong?

Δ_{C} \approx 0

on this clean/corrupt pair means it is not causal for this specific behavior; it may be load-bearing for a completely different capability the current metric never probes.

"To find where a computation lives, I patch the corrupted run into the clean run." — what's wrong?

The standard direction inserts the clean activation into the corrupted run to test restoration; swapping the direction (corrupt into clean, "noising") answers a different question — how fragile clean behavior is, not what restores corrupted behavior.

Why questions

Why do we need two inputs (clean and corrupted) instead of just one?

The corrupted input creates the controlled gap

G

, and only by injecting the clean activation into that gap can we isolate one component's causal contribution against a matched baseline.

Why must clean and corrupted inputs be as similar as possible (same length, same structure)?

If they differ in many ways, the activation you swap carries confounds beyond the feature you study, so restoration could come from irrelevant differences rather than the mechanism you're testing.

Why does patching layer 20 alone barely help in the greater-than example while layer 15 helps a lot?

Layer 15 does the numerical reasoning and layer 20 only formats numbers into tokens; restoring formatting is useless while the underlying "greater-than" computation is still corrupted upstream.

Why is the causal-fraction denominator the full clean-to-corrupt gap

G

rather than just the corrupt loss?

G

is all the closable distance, so dividing by it normalizes

Δ_{C}

into

[0, 1]

, where "1" means the component closed all the closable distance, making effects comparable across tasks of different difficulty.

Why can a single patch experiment fail to detect a genuinely important circuit?

If the circuit is spread over redundant paths, patching any one path leaves the others intact and the behavior barely moves, hiding importance that only joint patching (or path patching) reveals.

Why is patching considered stronger evidence than the logit lens alone?

The logit lens reads off what a layer represents but is still observational; patching intervenes, so it tests whether that representation is actually used downstream.

Why complement patching with Causal Scrubbing?

Patching confirms a component matters, but causal scrubbing stress-tests a full hypothesized circuit by resampling activations consistent with your claim, catching hypotheses that pass single patches yet are still wrong.

Edge cases

What happens when clean and corrupted inputs are identical?

The gap

G = 0

and there is no distinguishing activation to swap, so the causal fraction is undefined (division by zero) and the experiment carries no information.

What does a negative

Δ_{C}

(negative causal effect) mean?

The patched activation made the metric worse than the corrupted baseline, meaning the clean value from this component actively hurts in the corrupted context — often a sign of interaction with other unpatched components.

Two heads are perfectly redundant. What do individual patches show, and how do you detect them?

Each individually shows a small

Δ

(the other head compensates), and you detect redundancy by patching both together and seeing

Synergy (A, B) < 0

, i.e. the pair's effect is less than the sum.

What if the corrupted run already produces the correct answer by luck?

Then there is no error to restore, the gap

G

is near zero, and patching results are noisy and meaningless — you must choose a corruption that reliably breaks the target behavior.

What does a causal fraction near 0 for every component you tested imply?

Either the true mechanism is a component/location you haven't patched, or the behavior is distributed so widely that no single site is decisive — a cue to widen the search or patch groups.

Recall One-line summaries to self-test

Correlation vs causation ::: Activation strength and attention weights are correlational; only intervention (patching) is causal. Restoration test ::: Insert the clean activation into the corrupted run and see if the metric returns toward clean. Patch effect ::: $Δ_{C}$ is the drop in metric $L$ from patching $C$ ; causal fraction is $Δ_{C} / G$ . Individual before joint ::: Patch components alone first, then groups, to separate real effect from redundancy/synergy. Metric matters ::: The metric $L$ must measure the exact capability, not generic loss.