Activation patching — Hinglish notes

6.3.9 · AI-ML › Interpretability & Explainability

Activation patching ek causal intervention technique hai mechanistic interpretability mein jo kisi component (neuron, attention head, layer) ke specific role ko isolate karta hai — ek input ki activations ko doosre input ki activations se replace karke, phir model behavior par downstream effect measure karke.

Core Intuition

Sirf Attention Weights Kyun Nahi Dekhte?

Attention weights correlation dikhate hain, causation nahi. Ek head pronouns par attend kar sakta hai lekin us information ka actual use nahi karta. Patching test karta hai: "Agar mein is component ki clean input dekhne ki ability hata dun, to kya model ka behavior badlega?"

The Mechanism

Formula Causal Effect Quantification

Component $C$ ka causal effect hai: $Δ_{C} = L (f (x_{corrupt})) - L (f (x_{corrupt} ∣ patch C))$ Jahan $L$ ek loss hai jo clean answer se distance measure karta hai. Larger positive $Δ_{C}$ = patching ne loss zyada reduce kiya = zyada causally important.

First principles se derivation: Hum actually kya pooch rahe hain? "Component $C$ clean aur corrupted behavior ke beech ke difference mein kitna causally contribute karta hai?" Step 1: Counterfactual define karo $f (x ∣ a_{C}^{(ℓ)} = v)$ se matlab hai model ko component $C$ ki activation value $v$ par force karke run karna.

Step 2: Baseline behaviors

Clean output: $y_{clean} = f (x_{clean})$
Corrupted output: $y_{corrupt} = f (x_{corrupt})$
Behavioral gap: $L (y_{corrupt}) - L (y_{clean})$

Step 3: Intervention Hum poochte hain: "Kya hota agar corrupted run ko $C$ ki clean activation milti?" $y_{patched} = f (x_{corrupt} ∣ a_{C}^{(ℓ)} = a_{clean}^{(ℓ)})$ Step 4: Restoration measure karo Agar $C$ causally important hai: $L (y_{patched}) \to L (y_{clean})$

Effect size hai ki humne gap kitna close kiya: $Causal fraction = \frac{L ( y _{corrupt} ) - L ( y _{patched} )}{L ( y _{corrupt} ) - L ( y _{clean} )}$

Yeh formula kyun?

Numerator: Patching ne cheezein kitni improve kiin
Denominator: Close karne ke liye total behavioral gap
Result: Fraction ∈ [0,1] jahan 1 = "component poora difference explain karta hai"

Worked Examples

Worked example Example 1: Indirect Object Identification (IOI)

Task: "When Mary and John went to the store, John gave a drink to ___" complete karo → "Mary" predict karna chahiye

Setup:

Clean: "When Mary and John..." ("Mary" correctly predict karta hai)
Corrupted: "When Mary and Susan..." ("Susan" incorrectly predict karta hai) Question: Kaun sa attention head detect karta hai ki "Mary" indirect object hai?

Procedure:

Clean input run karo → head 9.6 (layer 9, head 6) par activations save karo
Corrupted input run karo → model "Susan" output karta hai (galat)
Patch: Corrupted re-run karo, lekin head 9.6 ka output clean activation se swap karo
Result: Model ab "Mary" output karta hai (sahi!)

Yeh step kyun? Head 9.6 "Mary indirect object hai" encode kar raha hoga kyunki uski clean activation restore karne se sahi behavior restore ho jaata hai. Doosre heads jo humne patch kiye unhone answer recover nahi kiya.

Calculation:

$L (corrupt) = - lo g P (Mary ∣ Susan context) = 5.2$
$L (patched) = - lo g P (Mary ∣ Susan context, patched) = 0.8$
$L (clean) = 0.3$
Causal fraction: $(5.2 - 0.8) / (5.2 - 0.3) = 0.90$ → head 90% error explain karta hai

Worked example Example 2: Greater-Than ke liye Multilayer Patching

Task: "The war lasted from 1712 to 17__" — model ko 12 se bada do-digit year predict karna chahiye (jaise "15", "20", ..., "99"), kabhi bhi ≤ 12 wala year nahi.

Hypothesis: Layer 15 numerical "greater-than" relationship compute karta hai, layer 20 internal number ko output tokens mein convert karta hai.

Setup:

Clean: "...from 1712 to 17" → sahi se years > 12 par probability mass rakhta hai (jaise "15", "83")
Corrupted: "...from 1799 to 17" → "99" anchor feed hota hai, isliye model galat tarike se mass years ≤ 99 lekin around 99 par rakhta hai, jo clean anchor "12" ke relative greater-than logic ko tod deta hai

Hum clean anchor ki activations (jo "lower bound 12 hai" encode karti hain) ko corrupted run mein patch karte hain aur measure karte hain ki "greater-than-12" behavior kitna restore hota hai. Metric hai valid years (> 12) par rakha gaya probability mass.

Patching experiment (higher effect = zyada greater-than behavior restore):

Component Patched	Mass on years > 12	Causal Effect
None (corrupted)	0.20	0.00 (baseline)
Layer 15 only	0.75	0.65
Layer 20 only	0.24	0.05
Both L15 + L20	0.93	0.92

Yeh pattern kyun?

Layer 15: "Output > lower-bound hona chahiye" computation restore karta hai → greater-than behavior ka zyada hissa wapas aata hai.
Layer 20: Sirf numbers ko tokens mein format karta hai; ise akele patch karne se zyada help nahi milti kyunki reasoning abhi bhi corrupted hai.
Combined: Poora causal chain (relationship compute karo → token format karo) restore ho jaata hai.

Yeh prove karta hai: Computation hierarchical hai — L15 numerical reasoning karta hai, L20 token formatting karta hai.

Common Mistakes

Variants & Extensions

1. Activation Addition (Opposite Direction)

Clean activations se replace karne ki bajaye, ek steering vector add karo: $a_{patched}^{(ℓ)} = a_{original}^{(ℓ)} + α \cdot v_{steer}$

Use hota hai: Capabilities enhance karne ke liye (jaise, translation mein "zyada French-ness add karo")

2. Path Patching

Poori layer patch mat karo — sirf component A → component B tak ki path patch karo.

Kyun? Direct vs. indirect effects isolate karta hai. Ho sakta hai head 3.2 directly task solve nahi karta, lekin head 5.1 ko feed karta hai jo karta hai.

Technique: Specifically A→B se residual stream contribution replace karo, baaki paths intact rakhho.

3. Iterative Patching

Components ko sequence mein patch karo, hamesha previous steps ke best patches rakhte hue. Ek minimal circuit build karta hai jo behavior explain kare.

Connections

Mechanistic Interpretability: Activation patching circuit discovery ke liye primary tool hai
Causal Scrubbing: Formalization jo patching ko ground-truth circuits ke saath generalize karta hai
Ablation Studies: Components remove karna "zeros se patch karna" hai — counterfactual patching se kam informative
Logit Lens: Patching ka complement jo dikhata hai ki kya representations exist karte hain; patching dikhata hai kyun woh matter karte hain
Causal Mediation Analysis: Statistical framework jise patching neural networks ke liye implement karta hai

Active Recall

Recall Ek 12-saal ke bachche ko explain karo

Socho tumhara dimaag ek math problem solve kar raha hai, lekin ek part galat number deta hai. Tum figure out karna chahte ho: "Mere dimaag ka kaun sa hissa gadbad kar raha tha?"

Yeh karo: Problem do baar solve karo. Pehli baar galat ho jaata hai. Doosri baar, koi tumhare dimaag mein sirf ek step ke liye sahi number whisper karta hai. Agar achanak poori problem sahi ho jaaye, to tum jaante ho: "Aha! Woh step wahi tha jo matter karta tha!"

Yahi hai activation patching. Hum ek robot brain (neural network) ko kuch galat karte hue lete hain, aur secretly use ek tiny hisse ke liye "correct" information dete hain. Agar robot achanak kaam karne lage, to hum jaante hain woh hissa super important tha. Yeh detective banne jaisa hai, lekin robot thoughts ke liye!

#flashcards/ai-ml

Activation patching kya hai? :: Ek causal intervention technique jahan tum ek component ki activations ko corrupted run se clean run ki activations se replace karte ho, phir measure karte ho ki behavior restore hota hai ya nahi — yeh prove karte hue ki woh component sahi behavior ke liye causally responsible tha.

Activation magnitude measure karne se patching behtar kyun hai? :: Activation magnitude correlation dikhati hai (kya saath fire karta hai), lekin patching causation dikhata hai (kya actually necessary hai). Ek neuron strongly activate kar sakta hai lekin redundant ya epiphenomenal ho sakta hai.

Causal effect ka formula likho aur har term explain karo :: $Δ_{C} = L (f (x_{corrupt})) - L (f (x_{corrupt} ∣ patch C))$ . Yeh component C patch karne par loss reduction measure karta hai. Positive Δ matlab patching ne help ki (loss kam hua), bade values matlab stronger causal role.

Patching aur ablation mein kya difference hai?

Ablation ek component remove karta hai (zero/mean par set karta hai), dikhata hai kya hota hai uske bina. Patching ek counterfactual activation se replace karta hai, dikhata hai kya hota hai different information ke saath. Patching cleaner causal attribution deta hai.

Tum head 4.2 patch karte ho aur behavior nahi badlta. Iska kya matlab hai?

Head 4.2 is specific behavior ke liye causally important nahi hai, chahe woh activate bhi kare. Yeh redundant, epiphenomenal, ya kisi doosre task par kaam kar raha ho sakta hai.

Path patching kya hai?

Sirf component A se component B tak ke specific information flow ko patch karna, baaki paths intact rakhte hue. Direct causal effects ko indirect ones se isolate karta hai.

Tum kaise detect karte ho ki do components redundant hain ya synergistic?

Synergy calculate karo = Δ(A∪B) - (ΔA + ΔB). Negative synergy matlab redundant (overlap), positive matlab synergistic (woh cooperate karte hain), zero matlab independent.

Tumhara metric tumhare research question se match kyun karna chahiye?

Generic metrics (jaise overall cross-entropy) saare behaviors par average karte hain. Agar tum toxicity detection study kar rahe ho, to tumhein specific toxic token logit measure karna hoga, na ki saare tokens par average loss.