Activation patching

Activation patching is a causal intervention technique in mechanistic interpretability that isolates the specific role of a component (neuron, attention head, layer) by replacing its activations from one input with those from another, then measuring the downstream effect on model behavior.

Core Intuition

Why Not Just Look at Attention Weights?

Attention weights show correlation, not causation. A head might attend to pronouns but not actually use that information. Patching tests: "If I remove this component's ability to see the clean input, does the model's behavior change?"

The Mechanism

Formula Causal Effect Quantification

The causal effect of component $C$ is: $\Delta_C = \mathcal{L}(f(x_{\text{corrupt}})) - \mathcal{L}(f(x_{\text{corrupt}} | \text{patch } C))$ Where $\mathcal{L}$ is a loss measuring distance from the clean answer. Larger positive $\Delta_C$ = patching reduced the loss more = more causally important.

Derivation from first principles: What are we actually asking? "How much does component $C$ causally contribute to the difference between clean and corrupted behavior?" Step 1: Define the counterfactual Let $f(x | a^{(\ell)}_C = v)$ denote running the model with component $C$ 's activation forced to value $v$ .

Step 2: Baseline behaviors

Clean output: $y_{\text{clean}} = f(x_{\text{clean}})$
Corrupted output: $y_{\text{corrupt}} = f(x_{\text{corrupt}})$
Behavioral gap: $\mathcal{L}(y_{\text{corrupt}}) - \mathcal{L}(y_{\text{clean}})$

Step 3: The intervention We ask: "What if the corrupted run had access to $C$ 's clean activation?" $y_{\text{patched}} = f(x_{\text{corrupt}} | a^{(\ell)}_C = a^{(\ell)}_{\text{clean}})$ Step 4: Measure restoration If $C$ is causally important: $\mathcal{L}(y_{\text{patched}}) \to \mathcal{L}(y_{\text{clean}})$

The effect size is how much of the gap we closed: $\text{Causal fraction} = \frac{\mathcal{L}(y_{\text{corrupt}}) - \mathcal{L}(y_{\text{patched}})}{\mathcal{L}(y_{\text{corrupt}}) - \mathcal{L}(y_{\text{clean}})}$

Why this formula?

Numerator: How much patching improved things
Denominator: Total behavioral gap to close
Result: Fraction ∈ [0,1] where 1 = "component fully explains the difference"

Worked Examples

Worked example Example 1: Indirect Object Identification (IOI)

Task: Complete "When Mary and John went to the store, John gave a drink to ___" → should predict "Mary"

Setup:

Clean: "When Mary and John..." (predicts "Mary" correctly)
Corrupted: "When Mary and Susan..." (predicts "Susan" incorrectly) Question: Which attention head detects that "Mary" is the indirect object?

Procedure:

Run clean input → save activations at head 9.6 (layer 9, head 6)
Run corrupted input → model outputs "Susan" (wrong)
Patch: Re-run corrupted, but swap head 9.6's output with the clean activation
Result: Model now outputs "Mary" (correct!)

Why this step? Head 9.6 must be encoding "Mary is the indirect object" because restoring its clean activation restores correct behavior. Other heads we patched didn't recover the answer.

Calculation:

$\mathcal{L}(\text{corrupt}) = -\log P(\text{Mary}|\text{Susan context}) = 5.2$
$\mathcal{L}(\text{patched}) = -\log P(\text{Mary}|\text{Susan context, patched}) = 0.8$
$\mathcal{L}(\text{clean}) = 0.3$
Causal fraction: $(5.2 - 0.8)/(5.2 - 0.3) = 0.90$ → head explains 90% of the error

Worked example Example 2: Multilayer Patching for Greater-Than

Task: "The war lasted from 1712 to 17__" — the model should predict a two-digit year greater than 12 (e.g. "15", "20", ..., "99"), never a year ≤ 12.

Hypothesis: Layer 15 computes the numerical "greater-than" relationship, layer 20 converts the internal number to output tokens.

Setup:

Clean: "...from 1712 to 17" → correctly puts probability mass on years > 12 (e.g. "15", "83")
Corrupted: "...from 1799 to 17" → the "99" anchor is fed in, so the model wrongly puts mass on years ≤ 99 but around 99, breaking the greater-than logic relative to the clean anchor "12"

We patch the clean anchor's activations (which encode "the lower bound is 12") into the corrupted run and measure how much the "greater-than-12" behavior is restored. The metric is the probability mass placed on valid years (> 12).

Patching experiment (higher effect = more greater-than behavior restored):

Component Patched	Mass on years > 12	Causal Effect
None (corrupted)	0.20	0.00 (baseline)
Layer 15 only	0.75	0.65
Layer 20 only	0.24	0.05
Both L15 + L20	0.93	0.92

Why this pattern?

Layer 15: Restores the "output must be > lower-bound" computation → most of the greater-than behavior returns.
Layer 20: Only formats numbers into tokens; patching it alone barely helps because the reasoning is still corrupted.
Combined: Full causal chain (compute relationship → format token) is restored.

This proves: The computation is hierarchical—L15 does the numerical reasoning, L20 does the token formatting.

Common Mistakes

Variants & Extensions

1. Activation Addition (Opposite Direction)

Instead of replacing with clean activations, add a steering vector: $a^{(\ell)}_{\text{patched}} = a^{(\ell)}_{\text{original}} + \alpha \cdot v_{\text{steer}}$

Used for: Enhancing capabilities (e.g., "add more French-ness" to translation)

2. Path Patching

Don't patch the entire layer—patch only the path from component A → component B.

Why? Isolates direct vs. indirect effects. Maybe head 3.2 doesn't directly solve the task, but it feeds head 5.1 which does.

Technique: Replace residual stream contribution from A→B specifically, leave other paths intact.

3. Iterative Patching

Patch components in sequence, always keeping the best patches from previous steps. Builds a minimal circuit that explains behavior.

Connections

Mechanistic Interpretability: Activation patching is the primary tool for circuit discovery
Causal Scrubbing: Formalization that generalizes patching with ground-truth circuits
Ablation Studies: Removing components is "patching with zeros"—less informative than counterfactual patching
Logit Lens: Complements patching by showing what representations exist; patching shows why they matter
Causal Mediation Analysis: Statistical framework patching implements for neural networks

Active Recall

Recall Explain to a 12-year-old

Imagine your brain is solving a math problem, but one part gives you the wrong number. You want to figure out: "Which part of my brain messed up?"

Here's what you do: You solve the problem twice. First time, you get it wrong. Second time, someone whispers the correct number for just one step in your brain. If suddenly you get the whole problem right, you know: "Aha! That step was the one that mattered!"

That's activation patching. We take a robot brain (neural network) doing something wrong, and we secretly give it the "correct" information for one tiny part. If the robot suddenly works, we know that part was super important. It's like being a detective, but for robot thoughts!

#flashcards/ai-ml

What is activation patching? :: A causal intervention technique where you replace a component's activations from a corrupted run with those from a clean run, then measure if behavior is restored—proving that component was causally responsible for the correct behavior.

Why is patching better than just measuring activation magnitude? :: Activation magnitude shows correlation (what fires together), but patching shows causation (what's actually necessary). A neuron can activate strongly but be redundant or epiphenomenal.

Write the formula for causal effect and explain each term :: $\Delta_C = \mathcal{L}(f(x_{\text{corrupt}})) - \mathcal{L}(f(x_{\text{corrupt}} | \text{patch } C))$ . It measures loss reduction when patching component C. Positive Δ means patching helped (lowered loss), larger values mean stronger causal role.

What's the difference between patching and ablation?

Ablation removes a component (sets to zero/mean), showing what happens without it. Patching replaces with a counterfactual activation, showing what happens with different information. Patching gives cleaner causal attribution.

You patch head 4.2 and behavior doesn't change. What does this mean?

Head 4.2 is not causally important for this specific behavior, even if it activates. It could be redundant, epiphenomenal, or working on a different task.

What is path patching?

Patching only the specific information flow from component A to component B, leaving other paths intact. Isolates direct causal effects from indirect ones.

How do you detect if two components are redundant vs. synergistic?

Calculate synergy = Δ(A∪B) - (ΔA + ΔB). Negative synergy means redundant (overlap), positive means synergistic (they cooperate), zero means independent.

Why must your metric match your research question?

Generic metrics (like overall cross-entropy) average over all behaviors. If you're studying toxicity detection, you need to measure the specific toxic token logit, not average loss across all tokens.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Activation patching ek detective jaisa technique hai jisme hum neural network ke specific parts ko test karte hain. Socho ki ek robot galat answer de raha hai—hum janna chahte hain ki "kaun sa part responsible hai is mistake ke liye?"

Basic idea simple hai: Hum dobaar model chalate hain. Pehli baar, sahi input dete hain aur model sahi output deta hai. Dusri baar, galat input dete hain aur model galat ho jata hai. Ab detective mode start: Hum galat run mein se ek specific layer ya attention head ke activations ko temporarily swap kar dete hain sahi run ke activations se. Agar suddenly model sahi answer dene lag jaye, matlab us component ne hi solve kiya tha original problem ko! Ye correlation nahi, causation hai—hum prove kar rahe hain ki ye part actually important hai, sirf coincidence nahi.

Ye technique bahut powerful hai kyunki traditional methods sirf dekhte the ki "kis neuron mein zyada activation hai" (high magnitude = important lagta hai). Lekin patching se pata chalta hai ki high activation ka matlab zaroori nahi ki wo component actually kuch kaam kar raha hai—shayad wo redundant hai ya sirf side-effect hai. Isliye interpretability research mein patching ab gold standard ban gaya hai circuits discover karne ke liye. Jab tum transformer models mein specific behaviors (jaise pronoun resolution ya factual recall) ko understand karna chahte ho, activation patching se tum exact mechanism ko isolate kar sakte ho—kaun sa head kis information ko process karta hai, kis layer pe computation hota hai, sab kuch systematically test ho sakta hai.

Core Intuition

Why Not Just Look at Attention Weights?

The Mechanism

Worked Examples

Common Mistakes

Variants & Extensions

1. Activation Addition (Opposite Direction)

2. Path Patching

3. Iterative Patching

Connections

Active Recall

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Interpretability & Explainability

Connections