Multi-head attention — Hinglish notes

4.1.5 · AI-ML › Transformer Architecture

What Problem Does It Solve?

Single-head attention ki representational capacity limited hoti hai. $Q, K, V$ projections ka ek single set sirf ek type ka relationship pattern seekh sakta hai. Lekin language (aur zyaattar sequential data) mein ek saath kaafi saare structure types hote hain:

Syntactic dependencies (subject verb se agree karta hai)
Semantic relationships (synonym, antonyms)
Positional patterns (nearby words vs. distant context)
Co-reference (pronouns jo pehle ke nouns se link hote hain)

Multi-head attention (MHA) multiple representation subspaces parallel mein seekhta hai, jisme har head alag patterns mein specialize karta hai.

Definition and Architecture

Derivation From First Principles

Step 1: Why Reduce Dimension Per Head?

Goal: $h$ attention heads ko parallel mein run karo bina computational cost badhaaye.

Agar hamare paas $d_{model} = 512$ hai aur hum $h = 8$ heads chahte hain:

Naive approach: Har head full 512-dim use kare → total cost $8 \times cost (512)$
Smart approach: Har head $d_{k} = 512/8 = 64$ use kare → total cost $\approx cost (512)$

Yeh kaam kyun karta hai: Attention cost $O (n^{2} d)$ hai jahan $n$ = sequence length, $d$ = dimension.

8 heads of 64-dim each: $8 \times O (n^{2} \times 64) = O (n^{2} \times 512)$
Ek 512-dim head jitna hi total FLOPs, lekin hume milte hain 8 alag views!

Step 2: Projecting Into Head Subspaces

Head $i$ ke liye, hum shared $Q, K, V$ (shape $[n, d_{model}]$ ) ko $[n, d_{k}]$ mein project karte hain:

$Q_{i} = Q W_{i}^{Q}, K_{i} = K W_{i}^{K}, V_{i} = V W_{i}^{V}$

jahan $W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}$ learned hain, har head ke liye alag.

Alag projections kyun?

Agar saare heads same $W^{Q}, W^{K}, W^{V}$ use karein, toh woh identical attention patterns compute karenge—koi faayda nahi!
Alag projections har head ko same input se alag features extract karna seekhne dete hain.

Intuition: Har $W_{i}^{Q}$ ko ek alag sawaal poochne ki tarah socho:

Head 1 capture kar sakta hai "Kaun se words grammatically related hain?"
Head 2 capture kar sakta hai "Kaun se words semantically similar hain?"
Head 3 capture kar sakta hai "Kaun se words position mein nearby hain?"

Step 3: Compute Scaled Dot-Product Attention Per Head

Har head independently compute karta hai:

$head_{i} = softmax (\frac{Q _{i} K _{i}^{T}}{d _{k}}) V_{i}$

Yeh har head ke liye $[n, d_{v}]$ shape ka output deta hai.

Same scaled dot-product formula kyun?

Kaam karta hai! Jaadu alag-alag learned projections $W_{i}^{Q, K, V}$ mein hai, attention mechanism mein khud nahi.
Same mechanism use karne se har head same tarike se interpretable hota hai.

Step 4: Concatenate Heads

$h$ head outputs ko side-by-side stack karo:

$Concat (head_{1}, \dots, head_{h}) = [head_{1} ∣ head_{2} ∣ \dots ∣ head_{h}]$

Yeh shape $[n, h \cdot d_{v}] = [n, d_{model}]$ produce karta hai (kyunki $h \cdot d_{v} = d_{model}$ ).

Sum/average ki jagah concatenate kyun?

Hum har head ki saari information preserve karna chahte hain.
Sum karne se woh merge ho jaate, aur har head ne jo distinct patterns seekhe hain woh kho jaate.
Concatenation unhe alag rakhta hai, lekin unhe blend karne ke liye ek aur step chahiye...

Step 5: Final Linear Projection

$MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}$

jahan $W^{O} \in R^{d_{model} \times d_{model}}$ learned hai.

Yeh final projection kyun?

Concatenated heads abhi bhi alag "channels" mein hain— $W^{O}$ model ko heads ke across information mix karne deta hai.
Yeh seekhta hai ki alag types ki attention (syntactic + semantic + positional) ko kaise combine karein.
Output ko full $d_{model}$ dimension par waapas laata hai, next layer ke liye ready.

Complete Forward Pass Example

Worked example 8-Head Attention with

$d_{model} = 512$

Setup:

Sequence length $n = 10$ tokens
Model dimension $d_{model} = 512$
Number of heads $h = 8$
Per-head dimension $d_{k} = d_{v} = 512/8 = 64$

Input: $Q, K, V \in R^{10 \times 512}$ (previous layer ya embeddings se)

Step 1 — Project to Heads: Har head $i = 1, \dots, 8$ ke liye:

$Q_{i} = Q W_{i}^{Q}$ jahan $W_{i}^{Q} \in R^{512 \times 64}$ → $Q_{i} \in R^{10 \times 64}$
$K_{i} = K W_{i}^{K}$ → $K_{i} \in R^{10 \times 64}$
$V_{i} = V W_{i}^{V}$ → $V_{i} \in R^{10 \times 64}$ Yeh step kyun? Har head ko data ka apna "view" milta hai ek lower-dimensional subspace mein.

Step 2 — Attention Per Head: $head_{i} = softmax (\frac{Q _{i} K _{i}^{T}}{64}) V_{i}$

Scores matrix: $Q_{i} K_{i}^{T} \in R^{10 \times 10}$ (har token kitna har doosre token par attend karta hai)
$64 = 8$ se scale karo gradients stabilize karne ke liye
Har row par Softmax (attention weights ka sum 1 hota hai)
$V_{i}$ se multiply karo → $head_{i} \in R^{10 \times 64}$

Yeh step kyun? Is head ke subspace ke liye context-aware representation compute karo.

Step 3 — Concatenate: $Concat (head_{1}, \dots, head_{8}) \in R^{10 \times 512}$

8 outputs ko horizontally stack karo: $[64 + 64 + \dots + 64] = 512$ dims.

Yeh step kyun? Har head ki distinct information preserve karo.

Step 4 — Output Projection: $Output = Concat (\dots) W^{O}$ jahan $W^{O} \in R^{512 \times 512}$ → $Output \in R^{10 \times 512}$

Yeh step kyun? Heads ko interact karne do aur apni specialized knowledge blend karne do.

Worked example Concrete Numbers Walkthrough

Chalo head 1 ke output mein token position 3 ki ek entry compute karte hain.

Input: Token 3 ka query vector $q_{3, 1} \in R^{64}$ ( $W_{1}^{Q}$ se projection ke baad)

Step 1 — Compute Scores: Har position $j = 1, \dots, 10$ ke liye: $score_{3, j} = \frac{q _{3, 1}^{T} k _{j, 1}}{64} = \frac{q _{3, 1}^{T} k _{j, 1}}{8}$

Example values:

$score_{3, 1} = 32/8 = 4.0$
$score_{3, 2} = 16/8 = 2.0$
$score_{3, 3} = 40/8 = 5.0$ (apne aap par attend karna)
... (7 aur)

8 se divide kyun? Dot products ko magnitude mein explode hone se rokta hai.

Step 2 — Softmax: $weight_{3, j} = \frac{e ^{score_{3, j}}}{\sum _{j^{'}} e ^{score_{3, j^{'}}}}$

Example (simplified):

$weight_{3, 1} = 0.12$
$weight_{3, 2} = 0.03$
$weight_{3, 3} = 0.45$ (apne aap par highest attention)
... (sum to 1.0)

Softmax kyun? Scores ko probabilities mein convert karta hai (har token par kitna dhyan dena hai).

Step 3 — Weighted Sum of Values: $head_{1} [3] = \sum_{j = 1}^{10} weight_{3, j} \cdot v_{j, 1}$ Yeh ek 64-dimensional vector produce karta hai jo saari positions ki information mix karta hai, attention se weighted.

Yeh step kyun? Output ek context-aware representation hai: token 3 ne doosri positions se relevant info "gather" ki hai, based on jo head 1 ko important lagta hai.

Common Mistakes

Common mistake Mistake 3: "Concatenation sirf ek detail hai—heads ko sum karna bhi kaam karega"

Kyun sahi lagta hai: Dono multiple heads ki information combine karte hain, toh kya fark hai?

Kyun galat hai:

Sum karne se information kho jaati hai: Agar head 1 position 5 par strongly attend kare aur head 2 position 8 par strongly attend kare, toh unke outputs ko summing karna in do distinct patterns ko mix kar deta hai.
Concatenation distinctness preserve karta hai: Final $W^{O}$ projection seekh sakta hai ki heads ko zaroorat ke hisaab se kaise weight/combine karein, lekin har head ki raw information abhi bhi accessible hai.
Ablation studies dikhati hain ki concatenation + $W^{O}$ summing se better perform karta hai.

Fix: Concatenation + linear projection model ko flexibility deta hai ki head outputs ko zaroorat ke hisaab se preserve bhi kare aur mix bhi kare.

Parameter Count

Chalo multi-head attention ke parameters count karte hain:

Per head:

$W_{i}^{Q}$ : $d_{model} \times d_{k}$
$W_{i}^{K}$ : $d_{model} \times d_{k}$
$W_{i}^{V}$ : $d_{model} \times d_{v}$
Total per head: $3 d_{model} \cdot d_{k}$

All heads: $h \cdot 3 d_{model} \cdot d_{k} = 3 d_{model} \cdot (h \cdot d_{k}) = 3 d_{model}^{2}$ (kyunki $h \cdot d_{k} = d_{model}$ )

Output projection: $W^{O}$ ke paas $d_{model} \times d_{model} = d_{model}^{2}$ parameters hain

Total: $3 d_{model}^{2} + d_{model}^{2} ==$ 4 d_{\text{model}}^2==$ parameters

Computational Complexity

Per head:

$Q_{i}, K_{i}, V_{i}$ compute karna: har ek $O (n \cdot d_{model} \cdot d_{k})$ , total $3 n \cdot d_{model} \cdot d_{k}$
Attention scores $Q_{i} K_{i}^{T}$ : $O (n^{2} d_{k})$
Softmax: $O (n^{2})$
$V_{i}$ ke saath weighted sum: $O (n^{2} d_{v})$

All heads: Per-head cost ka $h$ guna

Dominant term: Attention scores $O (h \cdot n^{2} d_{k}) = O (n^{2} d_{model})$ (kyunki $h \cdot d_{k} = d_{model}$ )

Output projection: $O (n \cdot d_{model}^{2})$

Lambi sequences ke liye jahan $n ≫ d_{model}$ , $O (n^{2} d_{model})$ attention term dominate karta hai. Isliye Transformers bahut lambi sequences (length 10,000+) ke saath struggle karte hain—sequence length mein quadratic!

Recall Ek 12-Saal-Ke Bachche Ko Explain Karo

Socho tum apne dost ki lambi kahani samajhne ki koshish kar rahe ho. Tum ek saath sab par focus nahi kar sakte, toh tumhara brain kuch smart karta hai: woh ek saath alag-alag tareekon se alag-alag parts par dhyan deta hai.

Tumhare brain ka ek hissa sunata hai ki kisne kya kiya (jaise "Alice ne ball throw ki").
Doosra hissa emotions sunata hai (jaise "woh excited thi").
Teesra hissa track karta hai ki cheezein kab hui (jaise "pehle... phir... aakhir mein..."). Multi-head attention waise hi kaam karta hai! Computer ke paas multiple "attention heads" hote hain (jaise tumhare brain ke alag-alag hisse), aur har ek text mein alag patterns par focus karta hai:
Head 1 un words ko connect kar sakta hai jo grammatically saath jaate hain ("the cat" → "sat").
Head 2 un words ko connect kar sakta hai jo similar cheezein mean karte hain ("happy" ← "joyful").
Head 3 nearby words ko connect kar sakta hai.

Har head ek special talent waale dost ki tarah hai—ek grammar mein great hai, ek feelings samajhne mein, ek order yaad rakhne mein. In saare doston ko saath kaam karwa ke (wahi "multi" part hai), computer kahani ko kahin behtar samajhta hai jaise ki sirf ek dost help kar raha ho!

Ant mein "concatenate and project" step waise hai jaise saare doston ko baith ke share karwana ki unhone kya notice kiya, taaki tumhe poori picture mile.

Connections Scaled Dot-Product Attention: Har head ke andar use hone wala attention mechanism

Self-Attention: MHA typically self-attention mein use hota hai (Q, K, V same source se)
Cross-Attention: MHA encoder-decoder attention ke liye bhi use ho sakta hai (Q decoder se, K,V encoder se)
Transformer Encoder Layer: MHA ke baad feed-forward network use karta hai
Positional Encoding: Position information provide karta hai jise attention heads leverage kar sakte hain
Attention Visualization: Techniques jo interpret karti hain ki har head ne kya seekha
Query-Key-Value Intuition: Projections samajhne ki foundation
Linear Projections in Neural Networks: $W^{Q}, W^{K}, W^{V}, W^{O}$ matrices kya karte hain

#flashcards/ai-ml

Single-head attention ke mukable multi-head attention ka primary advantage kya hai? :: Multi-head attention multiple representation subspaces parallel mein seekhta hai, jisse model ek saath alag types ke relationships (syntactic, semantic, positional) capture kar sakta hai. Ek single head ki representational capacity limited hoti hai.

h

heads aur model dimension

d_{model}

wale multi-head attention mein, har head ki dimension kya hoti hai?

Har head dimension

d_{k} = d_{v} = d_{model} / h

mein operate karta hai. Yeh total computation constant rakhta hai jabki parallel specialization enable karta hai.

Har head ke liye alag projection matrices

W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}

kyun use karte hain?

Alag projections har head ko same input se alag features seekhne dete hain. Agar saare heads identical projections use karein, toh woh identical attention patterns compute karenge, koi faayda nahi.

Multi-head attention ka formula kya hai?

MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}

jahan

head_{i} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})

Head outputs ko sum karne ki jagah concatenate kyun karte hain?

Concatenation har head ki distinct information preserve karta hai. Summing patterns ko mix kar deta, aur har head ne jo specialized information seekhi hai woh kho jaati. Final

W^{O}

projection phir seekh sakta hai ki heads ko optimally kaise combine karein.

Model dimension

d_{model}

wale multi-head attention ka total parameter count kya hai?

4 d_{model}^{2}

parameters: saare heads mein

W^{Q}, W^{K}, W^{V}

projections ke liye

3 d_{model}^{2}

, plus output projection

W^{O}

ke liye

d_{model}^{2}

Multi-head attention ki computational complexity kya hai?

Dominant term

O (n^{2} d_{model})

hai jahan

n

sequence length hai, jo saari positions mein attention scores compute karne se aata hai. Yeh sequence length mein quadratic hai.

Multi-head attention mein final

W^{O}

projection kyun exist karta hai?

W^{O}

model ko heads ke across information mix karne deta hai. Concatenation ke baad, heads abhi bhi alag "channels" mein hain—

W^{O}

seekhta hai ki alag types ki attention (syntactic + semantic + positional) ko task ke liye optimally kaise combine karein.