Mechanistic interpretability

6.3.6 · AI-ML › Interpretability & Explainability

Mechanistic Interpretability Kya Hai?

Yeh kyun zaroori hai:

Safety: Failure modes samajhne ke liye algorithm jaanna zaroori hai, sirf activations nahi
Scientific understanding: Neural networks novel algorithms implement kar sakte hain jinse hum seekh sakte hain
Debugging: Dhundho kahan computation galat ho rahi hai, na ki sirf kahan output galat hai
Alignment: Ensure karo ki models us tarah reason karein jaise hum sochte hain woh karte hain

Ek transformer layer computation se shuru karo. Real transformers LayerNorm apply karte hain (ya to pre-norm ya post-norm). Common pre-norm form likhte hain: $h_{mid} = h_{l} + Attn (LN_{1} (h_{l}))$ $h_{l + 1} = h_{mid} + MLP (LN_{2} (h_{mid}))$

LayerNorm explicitly kyun include karein? Normalization ki placement change karti hai ki nonlinearities kahan act karte hain, aur yeh residual-stream picture ke liye essential hai — yeh normalize karta hai ki har sublayer residual stream se kya padhta hai jabki stream khud raw sums accumulate karta rehta hai. Ise drop karna (jaise bare $h_{l + 1} = h_{l} + MLP (h_{l} + Attn (h_{l}))$ ) actual computation ko misrepresent karta hai.

Ab attention sublayer ko decompose karo. $H$ heads mein se har ek value-weighted sum compute karta hai, phir saare head outputs concatenate hote hain aur ek single output matrix $W_{O}$ apply hoti hai: $Attn (x) = W_{O} \cdot concat_{i = 1}^{H} (softmax (\frac{Q _{i} K _{i}^{T}}{d _{k}}) V_{i})$

Equivalently, $W_{O}$ ko per-head blocks $W_{O} = [W_{O}^{(1)} W_{O}^{(2)} \dots W_{O}^{(H)}]$ mein split karke jo har head ke slice par act karte hain, hum additive-over-heads form likh sakte hain: $Attn (x) = \sum_{i = 1}^{H} W_{O}^{(i)} \cdot softmax (\frac{Q _{i} K _{i}^{T}}{d _{k}}) V_{i}$

Yahan $W_{O}^{(i)}$ independent projection matrices nahi hain — yeh single shared $W_{O}$ ke column-blocks hain. $1/ d_{k}$ factor kyun? Yeh dot-product logits ko rescale karta hai taaki unka variance $\approx 1$ rahe regardless of head dimension $d_{k}$ , softmax ko saturation se baahir rakhta hai. Yeh decomposition kyun? Kyunki empirically, individual heads aksar distinct operations perform karte hain (jaise "previous token head", "induction head").

Ek circuit in heads + MLP neurons ka subset hota hai jo:

Output par strong causal effect rakhta ho
Ek coherent algorithm implement karta ho
Minimally sufficient ho (components hatane se functionality toot jaaye)

2. Activation Patching & Causal Analysis

Sirf ablation ki bajaye patching kyun?

Ablation batata hai ki component matter karta hai ya nahi
Patching batata hai ki yeh kaunsi information compute karta hai aur kahan woh information flow karti hai

Patching method ki derivation:

Do inputs $x_{clean}$ aur $x_{corrupted}$ consider karo jahan model clean par succeed karta hai lekin corrupted par fail karta hai.

Maan lo $a_{i}^{clean}$ component $i$ ki activation hai clean input par. Patching experiment: $model (x_{corrupted}) but with a_{j} \leftarrow a_{j}^{clean}$

Agar performance recover ho jaaye, component $j$ correct computation ko causally mediate karta hai.

Yeh kyun kaam karta hai: Agar $j$ success ke liye zaroori key information compute karta hai, toh us computation ko transplant karna functionality restore karta hai.

3. Superposition & Polysemanticity

Information theory se derivation:

Available capacity: $n$ dimensions Required features: $m$ features, har ek probability $s$ ke saath active

Agar features orthogonal hote, toh humein $m$ dimensions chahiye hote. Lekin sparsity ke saath:

Expected simultaneous features: $\approx m s$
Agar $m s < n$ , hum superposition ke zariye $m > n$ features pack kar sakte hain

Cost: Jab multiple features ek saath activate hote hain toh features ke beech interference.

Mathematical model (simplified toy example):

Feature vectors $v_{i} \in R^{n}$ optimize karo interference minimize karne ke liye: $min_{{v_{i}}} E_{f \sim Sparse (s)} [∥ \sum_{i = 1}^{m} f_{i} v_{i} - \sum_{i = 1}^{m} f_{i} v_{i}^{*} ∥^{2}]$

jahan $v_{i}^{*}$ orthonormal basis vectors hain ("ideal" non-interfering representation).

Consequence: Individual neurons polysemantic hote hain — yeh multiple unrelated features par respond karte hain.

Key Techniques

Logit Lens & Tuned Lens

Yeh kyun kaam karta hai:

Transformers mein, residual stream $h_{l}$ information accumulate karta hai: $h_{L} = h_{0} + \sum_{l = 1}^{L} Δ h_{l}$

Agar information linearly represented hai, toh early layers mein already partial "answers" hone chahiye jo unembedding matrix $W_{U}$ padh sake.

Derivation:

Final prediction hai: $p (w) = softmax (W_{U} \cdot LN (h_{L}))$

Intermediate layer ke liye: $p_{l} (w) = softmax (W_{U} \cdot LN (h_{l}))$

$p_{l}$ ko $p_{L}$ se compare karna batata hai ki predictions layers across kaise evolve hoti hain.

Attention Pattern Analysis

Common interpretable patterns:

Previous token head: $A_{i, i - 1} \approx 1$
Induction head: $A_{i, j}$ high hota hai jab position $j - 1$ par token position $i - 1$ ke token se match karta hai (pattern completion)
Duplicate token head: $A_{i, j}$ high hota hai jab positions $i$ aur $j$ par tokens identical hain

Patterns analyze kyun karein? Attention positions ke beech information move karne ka main mechanism hai. Pattern routing algorithm reveal karta hai.

Worked example Worked Example: Induction Circuit Dhundna

Task: Samajhna ki model repeated sequences jaise "A B ... A B" kaise complete karta hai

Step 1: Hypothesis Model do heads use karta hai: ek "previous token" detect karne ke liye, ek pattern complete karne ke liye.

Step 2: Activation patching se test karo

Input: "cat dog mouse cat dog [?]"
Clean run: Model "mouse" predict karta hai
Position "cat" corrupt karo: Model fail karta hai
Specific attention head ki activations patch karo: Performance recover hoti hai Yeh step kyun? Confirm karta hai ki head causally completion ko mediate karta hai.

Step 3: Attention patterns analyze karo Head1 (layer 1): Position 5, position 4 ko attend karta hai (previous token) Head 2 (layer 2): Position 5, position 2 ko attend karta hai (current token ki previous occurrence)

Yeh step kyun? Algorithm dikhata hai: Head 2 previous "cat" dhundta hai, uske baad jo aaya tha use copy karta hai.

Step 4: Composition verify karo Head 1 output → Head 2 query. Is path ko patch karo: circuit toot jaata hai.

Result: Do-head induction circuit identify kiya jo pattern implement karta hai: "Current token ki previous occurrence dhundo, jo uske baad aaya tha use copy karo."

Worked example Worked Example: Polysemantic Neuron Decomposition

Setup: Vision model mein MLP neuron "car wheels" aur "letter O wale text" dono ke liye activate hota hai.

Step 1: Activation patterns measure karo Neuron fire karta hai: wheels (0.8), letter O (0.7), circles (0.6), ovals (0.5) Yeh step kyun? Sabhi contexts identify karo jahan neuron active hai.

Step 2: Sparse dictionary learning Overcomplete basis seekho: $h = \sum_{i = 1}^{m} α_{i} d_{i}$ jahan $m > n$

Minimize karne ke liye train karo: $∥ h - \sum_{i} α_{i} d_{i} ∥^{2} + λ ∥ α ∥_{1}$

L1 penalty kyun? Sparsity enforce karta hai — har input thode dictionary features use karta hai.

Step 3: Learned features analyze karo Dictionary feature $d_{23}$ : sirf wheels (text par koi false positives nahi) Dictionary feature $d_{67}$ : sirf round text characters

Result: Single polysemantic neuron do interpretable features ka superposition tha.

Yeh kyun matter karta hai: Neuron level par dekh ke computation samajh nahi sakte — feature level tak decompose karna padega.

Common mistake Common Mistake: Correlation ko Mechanism Samajh Lena

Galat approach: "Yeh neuron cats ki images par activate hota hai, toh yeh cats detect karta hai."

Yeh sahi kyun lagta hai: Neuron activation aur cat images ke beech high correlation.

Yeh galat kyun hai:

Correlation yeh imply nahi karta ki neuron cat-ness compute karta hai
Spurious correlation ho sakta hai (cats aksar outdoors hoti hain → neuron actually grass detect karta hai)
Yeh explain nahi karta ki neuron final decision mein KAISE contribute karta hai

Fix: Neuron ki computational role verify karne ke liye causal methods (activation patching, ablation) use karo:

Kya ise ablate karne se cat classification hurt hoti hai?
Kya iske activation ko patch karne se non-cat images mein "cat-ness" transfer hoti hai?
Yeh kaunsi information compute karta hai jise downstream layers use karti hain?

Galti ka steel-man: Activation patterns hypothesis generation ke liye useful starting point hain. Error wahan rukna hai causal claims test karne ke bajaye.

Common mistake Common Mistake: Interpretable Components Assume Karna

Galat approach: "Har attention head ek interpretable kaam karta hoga."

Yeh sahi kyun lagta hai: Bahut saare heads ke clear roles HOTE hain (previous token head, induction head).

Yeh galat kyun hai:

Superposition heads par bhi apply hota hai — ek head multiple features implement kar sakta hai
Heads multiple circuits ka part ho sakte hain
Kuch computations ke liye multiple heads milke kaam karte hain (individually interpret nahi kar sakte)

Fix:

Circuit-level analysis se shuru karo (end-to-end algorithm)
Components tak tab decompose karo jab unke distinct, separable functions hon
Actual features dhundne ke liye attention head outputs par dictionary learning use karo Example: Head kuch contexts mein "previous token" ko attend kar sakta hai aur doosron mein "sentence mein first name" ko — yeh polysemantic hai.

Doosre Interpretability Methods Se Connections

vs. Feature visualization:

Feature viz dikhata hai ki component KYA respond karta hai
Mechanistic interpretability dikhati hai ki yeh KAISE compute karta hai aur yeh output ke liye KYU matter karta hai

vs. Saliency methods:

Saliency ek single decision ke liye importance dikhata hai
Mechanistic interpretability general algorithm dhundti hai

vs. Probing:

Probing: "Kya information X present hai?"
Mechanistic: "X kaise compute aur use hoti hai?"

Applications & Implications

AI Safety: Deployment se pehle deceptive reasoning circuits detect karo
Capability analysis: Samjho ki model kya seekh sakta hai uski limits kya hain
Scientific discovery: Networks novel algorithms implement kar sakte hain jo humans ne design nahi kiye
Model editing: Specific circuits target karke behavior surgically modify karo

Recall Ek 12-saal ke bacche ko explain karo

Socho tumhe ek super advanced alien computer mila hai jo chess bahut accha khelti hai, lekin tum samajh nahi rahe ki andar kaise kaam karta hai. Tumhare paas ise figure out karne ke do tarike hain: Purana tarika (regular interpretability): Dekho ise khelते aur notice karo "oh, jab board par queen hoti hai, yeh light blink karti hai." Tum sirf bahar se patterns notice kar rahe ho.

Naya tarika (mechanistic interpretability): Tum carefully ise kholo aur wires trace karo. Tumhein pata chalta hai ki parts ka ek group milkar "aisi pieces dhundta hai jo capture ho sakti hain", doosra group "evaluate karta hai ki king safe hai ya nahi", aur yeh groups ek specific tarike se connected hain decisions lene ke liye. Tumne woh actual step-by-step recipe figure out kar li jo computer use karta hai!

Mechanistic interpretability aise detective ki tarah hai jo sirf yeh nahi dekhta ki AI kya karta hai, balki actual "recipe" ya "algorithm" figure out karta hai jo woh follow kar raha hai, step by step, wire by wire. Yeh bahut zaroori hai kyunki agar hum AI par important decisions ke liye trust karne wale hain, toh hume sirf YEH nahi samajhna ki woh KYA decide karta hai, balki yeh bhi ki WOH KAISE SOCHTA HAI!

Connections

Feature Visualization — complementary: WHAT dikhata hai, mechanistic HOW dikhata hai
Attention Mechanisms — attention patterns study ka primary object hain
Sparse Autoencoders — polysemantic neurons decompose karne ka tool
Transformer Architecture — residual stream structure mechanistic analysis enable karta hai
Probing Classifiers — related lekin different: presence vs. computation
Model Editing — mechanistic understanding ke applications

#flashcards/ai-ml

Mechanistic interpretability kya hai aur yeh post-hoc interpretability se kaise differ karti hai? :: Mechanistic interpretability neural networks ke learned algorithms ko reverse-engineer karti hai individual components aur circuits analyze karke. Post-hoc interpretability decisions ke baad unhe explain karti hai; mechanistic interpretability woh computational mechanism explain karti hai jo decisions produce karta hai.

Circuits Hypothesis kya hai?

Yeh hypothesis ki neural networks circuits se bane hote hain — sparse subgraphs jo specific, interpretable algorithms implement karte hain. Har circuit ek human-understandable computation perform karta hai jaise edge detection ya pattern completion.

Attention score mein 1/√d_k scaling factor kyun hota hai?

Yeh query-key dot products ko rescale karta hai taaki unka variance approximately 1 rahe regardless of head dimension d_k, softmax ko uski saturated regime se baahir rakhta hai aur useful gradients preserve karta hai.

Multi-head attention mein, kya har head ke liye alag output matrix W_O hoti hai?

Nahi. H head outputs concatenate hote hain aur ek single shared output matrix W_O apply hoti hai. Per-head blocks W_O^(i) sirf us ek W_O ke column-slices hain, jo tumhein attention ko heads par additive sum ke roop mein likhne deta hai.

Activation patching kya hai aur yeh ablation se zyada powerful kyun hai?

Activation patching ek run ki activations ko doosre run ki activations se replace karta hai yeh test karne ke liye ki behavior transfer hota hai ya nahi. Yeh ablation se zyada powerful hai kyunki ablation sirf dikhata hai ki component matter karta hai ya nahi, jabki patching dikhata hai ki yeh KAUNSI information compute karta hai aur WOH information KAHAN flow karti hai.

Superposition kya hai aur yeh polysemanticity kyun cause karta hai?

Superposition tab hota hai jab networks apne dimensions se zyada features represent karte hain features ko non-orthogonal directions ke roop mein encode karke. Yeh polysemanticity cause karta hai (neurons multiple unrelated features par respond karte hain) kyunki features ek doosre ke saath interfere karte hain jab woh orthogonal nahi hote.

Logit lens technique kya hai?

Ek technique jo intermediate layer activations ko directly vocabulary logits mein project karti hai yeh reveal karne ke liye ki model har layer par kya "sochta" hai. Yeh kaam karta hai kyunki residual stream information linearly accumulate karta hai, toh early layers mein partial "answers" hote hain jo unembedding matrix padh sakti hai.

Induction head kya hai aur yeh kaunsa algorithm implement karta hai?

Ek induction head ek attention pattern hai jo repeated sequences complete karta hai. Yeh implement karta hai: "Current token ki previous occurrence dhundo, phir jo uske baad aaya tha use copy karo." Example: "A B ... A B" mein, yeh predict karta hai ki agla token wahi hona chahiye jo pehle "A B" ke baad aaya tha.

Neuron activation aur ek concept ke beech correlation yeh claim karne ke liye insufficient kyun hai ki neuron woh concept "detect" karta hai?

Correlation yeh prove nahi karta ki neuron woh concept COMPUTE karta hai — yeh spurious correlation ho sakta hai, aur yeh explain nahi karta ki neuron decisions mein KAISE contribute karta hai. Computational role verify karne ke liye causal methods (patching, ablation) use karne chahiye.

Dictionary learning polysemantic neurons decompose karne mein kaise help karta hai? :: Dictionary learning L1 penalty ke saath sparsity ke liye ek overcomplete basis (dimensions se zyada features) dhundta hai. Yeh ek single polysemantic neuron ko multiple interpretable features mein decompose karta hai jo superposition mein the, alag-alag computations ko separate karta hai.

Neural network mein circuit identify karne ke liye teen criteria kya hain?

1) Output par strong causal effect (ablation/patching ke zariye), 2) Ek coherent algorithm implement karta hai jo human-understandable ho, 3) Minimally sufficient ho (koi bhi component hatane se functionality toot jaaye).