Attention visualization and limitations

6.3.4 · AI-ML › Interpretability & Explainability

Overview

Attention visualization wo process hai jisme transformer models ke attention weights ko inspect karte hain taaki yeh samjha ja sake ki prediction karte waqt model kaun se input tokens par focus karta hai. Yeh intuitive lagta hai—"chalo dekhte hain model kya dekhta hai"—lekin explanation tool ke roop mein attention visualization ki kaafi significant limitations hain jo har practitioner ko samajhni chahiye.

Ise aise samjho jaise kisi ko padhte hue dekh rahe ho: woh jahan zyada der tak aankhein tikaayen, shayad woh unhe important lagta hoga. Lekin sirf isliye ki unhone ek word dekha, iska matlab yeh nahi ki usi wajah se unka conclusion aya.

Core Concepts

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

Attention weights woh matrix hai $A=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)$ jahan $A_{ij}$ represent karta hai ki position $i$ position $j$ ko kitna attend karta hai.

KYU softmax? Yeh scores ko ek probability distribution mein normalize karta hai: saare weights sum karke 1 hote hain, jisse inhe "importance scores" ki tarah interpret kiya ja sakta hai.

KYA matlab hai $A_{ij}$ ka? Yeh token $j$ ki information ka woh fraction hai jo is layer par token $i$ ki representation mein flow karta hai.

KAISE extract karein? Apne model mein attention compute karne ke baad, $V$ se multiply karne se pehle softmax output ko simply save kar lo.

Attention ko First Principles se Derive Karna

Chaliye samjhte hain ki attention yeh form kyun leta hai:

1. Goal: Multiple input tokens se information combine karna, learned importance weights ke saath
2. Similarity scoring: Query vector $q_i$ aur key vector $k_j$ ke liye, dot product ke zariye similarity compute karo: $q_i\cdot k_j$
   • Dot product kyun? Yeh alignment measure karta hai: high hota hai jab vectors ek hi direction mein point karte hain
   • High dimensions mein, raw dot products $d_k$ ke saath badhte hain, isliye hum scale karte hain: $\frac{q_i\cdot k_j}{\sqrt{d_k}}$
3. Normalization: Scores ko softmax use karke probabilities mein convert karo: ${\alpha }_{ij}=\frac{\exp (q_i\cdot k_j/\sqrt{d_k})}{{\sum }_{j^{\prime }}\exp (q_i\cdot k_{j^{\prime }}/\sqrt{d_k})}$
   • Softmax kyun? Ensure karta hai ki weights positive hain aur sum karke 1 hote hain (valid probability distribution)
4. Weighted combination: Output hai ${\sum }_j{\alpha }_{ij}{v}_j$—har value vector apne attention score se weighted

Yeh humein upar wala formula deta hai. Koi magic nahi, bas learned similarity → normalized weights → weighted average.

$\text{MultiHead}(Q,K,V)=\text{Concat}({\text{head}}_1,..,{\text{head}}_H)W^O$

jahan ${\text{head}}_h=\text{Attention}(Q{W}_h^Q,K{W}_h^K,V{W}_h^V)$

Har head ka apna attention matrix $A^{(h)}$ hota hai. Visualization typically yeh dikhata hai:

• Per-head heatmaps: har head $h$ ke liye $A_{ij}^{(h)}$
• Averaged attention: ${\bar{A}}_{ij}=\frac{1}{H}{\sum }_h{A}_{ij}^{(h)}$
• Max attention: ${\max }_h{A}_{ij}^{(h)}$ (kaun sa head sabse zyada attend karta hai)

Visualization Methods

Input: "The cat sat on the mat" → Agla word predict karo

Visualization: 7×7 heatmap jahan cell $(i,j)$ dikhata hai ki token $i$ token $j$ ko kitna attend karta hai.

Interpretation attempt: Agar "mat" strongly "on" aur "the" ko attend karta hai, toh hum soch sakte hain ki model prepositional context use kar raha hai.

Code structure:

import matplotlib.pyplot as plt
import numpy as np
 
# attention_weights shape: [seq_len, seq_len]
attention = model.get_attention_weights(input_ids, layer=6, head=3)
plt.imshow(attention, cmap='viridis')
plt.xlabel('Key position')
plt.ylabel('Query position')

Yeh step kyun? Heatmap raw $A$ matrix dikhata hai, jisse patterns visually obvious ho jaate hain.

Task: BERT ke saath Sentiment classification

Approach: Layer 8 ke saare 12 heads ko side-by-side visualize karo

Observation:

• Head 2: Punctuation ko attend karta hai (exclamation marks, question marks)
• Head 7: Negation words ko attend karta hai ("not", "never")
• Head 10: Dispersed attention (uninformative)

Interpretation: Alag-alag heads alag-alag linguistic features mein specialize karte hain.

Critical question: Lekin kya yeh patterns prediction ko cause karte hain? (Spoiler: Zaroori nahi)

Yeh step kyun? Heads ki comparison specialization reveal karti hai, lekin causality ko hume alag se test karna hoga.

Problem: Single-layer attention poore model mein information flow nahi dikhata.

Solution: Attention rollout layers mein cumulative attention compute karta hai:

$A_{\text{rollout}}^{(l)}=A^{(l)}\cdot A^{(l-1)}\cdot ...\cdot A^{(1)}$

Derivation: Information layer se layer mein flow karti hai. Agar layer 1 token $j$ se $k$ tak information move karta hai, aur layer 2 $k$ se $i$ tak, toh effectively $j$, $k$ ke through $i$ ko influence karta hai.

Matrix multiplication $A^{(2)}\cdot A^{(1)}$ is transitive flow ko capture karta hai: product mein entry $(i,j)$ saare intermediate tokens $k$ par sum karta hai.

Yeh step kyun? Deep information flow capture karta hai, sirf local attention nahi.

Limitation: Assume karta hai ki attention hi information ka ek maatra pathway hai (aisa nahi hai—residual connections aur MLPs bhi matter karte hain).

Fundamental Limitations

Galat intuition: Agar kisi token ko high attention weight milta hai, toh woh prediction ke liye important hona chahiye.

Kyun sahi lagta hai: Attention ka literally matlab hai "dhyan dena"—zyada attention ka matlab zyada influence hona chahiye, hai na?

Haqeeqat: Attention weights directly causal importance reveal nahi karte. Jain & Wallace (2019) aur Serano & Smith (2019) ki research ne dikhaya:

1. Adversarial examples: Tum bilkul alag attention distributions dhundh sakte ho jo same output produce karte hain
2. Attention is not explanation: Token par high attention ka matlab yeh nahi ki use ablate karne se prediction change hogi
3. Value vectors matter karte hain: High attention weight ${\alpha }_{ij}$ ke bawajood, agar value vector $v_j$ downstream layers ke null space mein hai, toh woh kuch contribute nahi karta

Mathematical fix: ${\text{Contribution}}_{ij}={\alpha }_{ij}\cdot ||v_j||\cdot {\text{downstream}}_{\text{sensitivity}}$

True importance ke liye gradient ya ablation impact measure karna padta hai, sirf attention weights nahi.

Yeh kyun hota hai: Attention ek deep network mein sirf ek operation hai. Value vectors LayerNorm, residual additions, aur MLPs se guzarte hain jo unke contribution ko amplify ya nullify kar sakte hain.

Galat approach: Saare attention heads ko average karo aur result interpret karo.

Kyun sahi lagta hai: Averaging noise smooth out karta hai aur ek "consensus" view deta hai.

Haqeeqat: Heads ho sakte hain:

• Redundant: Multiple heads same pattern seekh rahe hain (wasteful lekin harmless)
• Antagonistic: Kuch heads confounders ko attend karte hain, kuch true signal ko—averaging inhe mix kar deta hai
• Specialized: Ek head task ki saari information carry karta hai; averaging use dilute kar deta hai

Fix: Individually heads ko gradient-based attribution ke saath analyze karo taaki pata chale kaunse heads actually classification layer use karta hai.

Attention flow: Attention weights ke through forward-pass information routing

Gradient flow: Backward-pass credit assignment jo dikhata hai kaun se inputs loss ko affect karte hain

Attention correlation dikhata hai, gradients causation dikhate hain.

Token $x_j$ aur output $y$ ke liye:

• Attention: "Model ne $x_j$ ko kitna dekha?"
• Gradient $\frac{\partial \mathcal{L}}{\partial x_j}$: "$x_j$ ko change karne se loss kitna change hota hai?"

Yeh disagree kar sakte hain! Ek token ko high attention mil sakta hai lekin near-zero gradient ho sakta hai (examine kiya gaya lekin use nahi kiya), ya low attention ho lekin high gradient ho (doosre tokens ke through indirect influence).

Practical Visualization Guidelines

$L$ layers ke liye attention matrices $A^{(1)},..,A^{(L)}$ ke saath:

${\tilde{A}}^{(l)}=A^{(l)}+I$ $A_{\text{rollout}}={\tilde{A}}^{(L)}\cdot {\tilde{A}}^{(L-1)}\cdot ...\cdot {\tilde{A}}^{(1)}$

Identity $I$ kyun add karein? Residual connections ka matlab hai ki har layer attention padhta bhi hai aur raw input pass through bhi karta hai. $I$ add karna residual stream ko model karta hai.

Normalization: Probability interpretation maintain karne ke liye har product ke baad row-normalize karo: ${\tilde{A}}_{\text{norm}}=\text{diag}((\tilde{A}\cdot 1{)}^{-1})\cdot \tilde{A}$

Yeh step kyun? Normalization ke bina, values multiplication ke through geometrically explode ho jaati hain.

Kab use karein: Jab aapko input se output tak end-to-end attribution ki parwah ho, sirf local patterns ki nahi.

Gradient information incorporate karne ke liye:

${\text{Attribution}}_{ij}=A_{ij}\cdot \frac{\partial y}{\partial A_{ij}}$

Derivation: Chain rule se, attention weight $A_{ij}$ ka output $y$ par influence hai: $\frac{\partial y}{\partial x_j}={\sum }_i\frac{\partial y}{\partial A_{ij}}\cdot \frac{\partial A_{ij}}{\partial x_j}$

Term $\frac{\partial y}{\partial A_{ij}}$ output ka attention weight w.r.t. gradient hai—yeh us attention connection ke liye prediction ki sensitivity measure karta hai.

$A_{ij}$ (forward importance) ko is gradient (backward sensitivity) se multiply karna akele attention se behtar attribution score deta hai.

Yeh step kyun? "Model ne kya dekha" (attention) aur "output ke liye kya matter kiya" (gradient) dono ko combine karta hai.

When to Trust Attention Visualization

Trust karo: ✓ Exploratory analysis: "Model kaun se patterns seekhta hai?" ✓ Debugging: "Kya model padding tokens ko attend kar raha hai?" (bug indicator) ✓ Linguistic analysis: "Kya heads syntactic structures capture karte hain?"

Trust mat karo: ✗ Causal explanations: "Is token ne prediction cause ki" ✗ Model debugging: "Model fail hua kyunki usne X ko attend nahi kiya" ✗ Fairness audits: "Model discriminate karta hai kyunki attention pattern Y hai"

Gold standard: Attention visualization ko combine karo:

• Ablation studies ke saath: Tokens remove karo aur output change measure karo
• Gradient-based attribution ke saath: Integrated Gradients, Attention×Gradient
• Probing ke saath: Representations par classifiers train karo taaki test ho sake kya information encoded hai

Ab yahan trick hai: sirf isliye ki tumne kuch dekha, iska matlab yeh nahi ki usne answer karne mein madad ki! Shayad tumne galat section padha, ya tumhe answer pehle se yaad tha. Tumhane kahan dekha yeh track karna hamein tumhari strategy ke baare mein kuch batata hai, lekin yeh prove nahi karta ki kyun tumne question sahi ya galat kiya.

Yahi limitation hai: attention maps dikhate hain AI ne kya "dekha," lekin kya actually uske decision ka cause bana yeh nahi. Woh pata karne ke liye, humein experiments karne padte hain—jaise notes ke hisse cover karke dekhna ki answer change hota hai ya nahi (yahi ablation testing hai).

Connections

• Transformer Architecture - Jahan se attention aata hai
• Integrated Gradients - Based attribution alternative
• Attention Mechanisms - Mathematical foundation
• SHAP for Deep Learning - Model-agnostic explanation method
• Layer-wise Relevance Propagation - Ek aur gradient-based approach
• Probing Classifiers - Test karna ki representations kya encode karti hain
• BertViz Tool - Practical visualization library
• Adversarial Attention - Attention distribution robustness

#flashcards/ai-ml

Transformer mein attention weights kya hote hain? :: Matrix $A=\text{softmax}(Q{K}^T/\sqrt{d_k})$ jahan $A_{ij}$ represent karta hai ki position $i$ position $j$ ko kitna attend karta hai—token $j$ ki information ka woh fraction jo token $i$ ki representation mein flow karta hai.

Concept Map