Self-attention mechanism in detail — Hinglish notes

4.1.2 · AI-ML › Transformer Architecture

Overview

Self-attention woh core operation hai jo sequence mein har position ko baaki sabhi positions par attend karne deta hai, relevance ke basis par ek weighted representation compute karta hai. RNNs ke unlike jo sequentially process karte hain, self-attention parallel processing aur long-range dependencies enable karta hai bina vanishing gradient problem ke.

Figure — Self-attention mechanism in detail

Core Concepts

First Principles se Derivation

Step 1: Teen Matrices Kyun?

Problem: Hume compute karna hai ki har word kितना relevant hai har doosre word ke liye. Lekin raw embeddings directly use karne mein issues hain:

Embeddings meaning encode karti hain, "relevance" nahi
Hum apples ko oranges se compare kar rahe honge (position 1 ka role vs position 5 ka role)

Solution: Teen specialized representations banao:

Step 2: Attention Scores Compute Karna

Step 3: Scaling Factor

Formula

$d_{k}$ se Divide Kyun Karte Hain? $Scaled Score = \frac{Q K ^{T}}{d _{k}}$

Problem: Jaise-jaise $d_{k}$ badhta hai, dot products magnitude mein badhte hain kyunki hum zyada terms sum kar rahe hain. Bade $d_{k}$ ke liye, scores bahut bade positive/negative numbers ban sakte hain.

Yeh kyun matter karta hai: Jab hum bahut bade numbers par softmax apply karte hain, toh yeh extremely peaked ho jaata hai (almost one-hot). Near-zero regions mein gradients bahut tiny ho jaate hain → vanishing gradients.

Scaling ka Proof: Assume karo ki $q_{i d}$ aur $k_{j d}$ independent random variables hain mean 0 aur variance 1 ke saath.

$Var (q_{i} \cdot k_{j}) = Var (\sum_{d = 1}^{d_{k}} q_{i d} k_{j d}) = \sum_{d = 1}^{d_{k}} Var (q_{i d} k_{j d})$

Independent mean-zero variables ke liye: $Var (X Y) = Var (X) Var (Y) = 1 \cdot 1 = 1$

$Var (q_{i} \cdot k_{j}) = d_{k}$

Standard deviation $d_{k}$ ke roop mein badhti hai. $d_{k}$ se divide karne par unit variance normalize ho jaati hai, scores softmax ke liye ek reasonable range mein rehte hain.

Step 4: Softmax Normalization

Step 5: Weighted Aggregation

Complete Formula Derivation

Worked Examples

Worked example Example 1: Teen-Word Sequence

Input: "cat sat mat" with 4-dimensional embeddings (simplified). Suppose embedding ke baad: $X = 101011100010$ Hum $d_{k} = 2$ aur simplified weight matrices use karte hain: $W^{Q} = W^{K} = 10000100, W^{V} = 11000011$

Step 1: Q, K, V Compute Karo $Q = X W^{Q} = [110011], K = X W^{K} = [110011]$

$V = X W^{V} = 202021$

Step 2: Scores $Q K^{T}$ Compute Karo $Q K^{T} = 101011 [100111] = 101011112$

Yeh step kyun? Row $i$ , column $j$ batata hai ki position $i$ ko position $j$ par kitna attend karna chahiye. Jaise ki, score[2,2] = 2 row 2 mein sabse high hai, matlab "mat" sabse zyada khud par attend karta hai.

Step 3: $d_{k} = 2 \approx 1.41$ se Scale Karo $Scaled = [0.71 0.71 0 1.41 0.71 0 0.71 0.71]$

Step 4: Softmax Apply Karo (row-wise) Row 0 ke liye: $[0.71, 0, 0.71] \to [0.5, 0.18, 0.5]$ (exp aur normalize karne ke baad) Full attention matrix (approximate): $A \approx 0.40 0.20 0.23 0.20 0.40 0.23 0.40 0.40 0.54$

Yeh step kyun? Softmax raw scores ko probability distribution mein convert karta hai. "cat" (row 0) 40% khud par, 20% "sat" par, 40% "mat" par attend karta hai.

Step 5: V se Multiply Karo $Z = A V \approx 0.40 0.20 0.23 0.20 0.40 0.54 0.40 0.40 [220012]$

Row 0 output: $0.40 \cdot [2, 0] + 0.20 \cdot [0, 2] + 0.40 \cdot [2, 1] = [1.6, 0.8]$

Interpretation: "cat" ka output ek weighted mix hai—yeh primarily khud se aur "mat" se information incorporate karta hai, "sat" se kam.

Worked example Example 2: Distant Dependencies par Attention

Sentence: "The cat, which was very fluffy, sat on the mat."

Problem: "sat" samajhne ke liye, model ko jaanna chahiye ki subject "cat" hai (7 words dur), "fluffy" nahi (closer but irrelevant).

Self-attention yeh kaise solve karta hai:

"sat" (query) ke liye attention compute karte waqt, yeh sabhi previous words ke saath scores compute karta hai
"sat" (ek verb) ka query vector nouns (subjects) ke key vectors ke saath align karna seekh chuka hoga
"cat" ek noun aur main subject hai → high score
"fluffy" ek adjective hai → lower score
Softmax difference amplify karta hai → "sat" strongly "cat" par attend karta hai

Yeh step kyun? Seekhe gaye $W^{Q}$ aur $W^{K}$ matrices syntactic/semantic patterns encode karte hain. Verbs ke liye $W^{Q}$ subjects "maangna" seekhta hai; nouns ke liye $W^{K}$ potential subjects ke roop mein "advertise" karna seekhta hai.

RNN se comparison: Ek RNN "sat" aur "cat" ko connect karne ke liye 7 steps process karta, information potentially kho sakti thi. Self-attention direct gradient flow ke saath ek step mein connection banata hai.

Common Mistakes

Common mistake Mistake 1: "Self-attention sirf embeddings ke beech similarity compute karta hai"

Yeh sahi kyun lagta hai: Hum dot products compute kar rahe hain, jo similarity measure karte hain.

Yeh galat kyun hai: Hum raw embeddings use nahi karte! Hum $Q, K, V$ projections use karte hain. Model seekhta hai kis tarah ki similarity matter karti hai.

Example: Words "king" aur "queen" ke similar embeddings ho sakte hain (dono royalty), lekin sentence "The king married the queen," mein jab "married" query hai, toh use subject aur object ko alag tarike se attend karna hai. Seekhe gaye $W^{K}$ matrices subject-keys ko object-keys se distinguish karte hain, chahe underlying embeddings similar kyun na hon.

Fix: Yaad rakho ki teen projections task-specific similarity spaces create karte hain. $W^{Q}$ aur $W^{K}$ seekhte hain meaning ke kaun se aspects compare karne hain.

Common mistake Mistake 3: "Attention weights dikhate hain ki model kya 'samajhta' hai"

Yeh sahi kyun lagta hai: High attention weight = "model ko lagta hai yeh important hai."

Yeh galat kyun hai: Attention weights ek deep computation mein ek step hain. Baad ki layers attended information ko completely transform kar sakti hain. Low attention wala position phir bhi residual connections, layer norms, ya feedforward networks ke through output ko influence kar sakta hai.

Example: "The animal didn't cross the street because it was tired," mein "it" "animal" (80%) aur "street" (20%) dono par attend kar sakta hai. Lekin FFN "street" interpretation suppress karna seekh sakta hai, toh 20% ka matlab nahi ki model "confused" hai.

Fix: Attention weights ko information flow ka evidence maano, final decisions ka nahi. Attention pattern ek tool hai, poora reasoning process nahi.

Common mistake Mistake 4: "Self-attention ko word order ka koi notion nahi hai"

Yeh sahi kyun lagta hai: Formula permutation-invariant hai! $Attention (permute (X)) = permute (Attention (X))$ . Agar hum input positions shuffle karein, toh attention sirf outputs ko correspondingly shuffle karta hai.

Practice mein yeh galat kyun hai: Hum attention se pehle input embeddings mein positional encodings add karte hain. Yeh permutation invariance tod dete hain. Positional encoding ke bina: "dog bites man" aur "man bites dog" ko identical representations milenge (reordering tak).

Positional encoding ke saath: Har position ko uski embedding mein ek unique positional signal add hota hai. Ab position 0 ka "dog" ke liye query position 2 ke "dog" ke query se alag hai—attention positional patterns seekh sakta hai.

Fix: Self-attention akela order-agnostic hai. Positional encodings order-aware processing ke liye zaruri hain.

Active Recall Questions

Recall Self-attention ko ek 12-saal ke bachche ko explain karo

Socho tum ek mystery story padh rahe ho, aur tumhare saamne "he" word aata hai. "He" kaun hai? Tumhe answer dhundhhne ke liye pehle ke sentences mein wapas dekhna hoga—shayad yeh detective hai, ya shayad suspect.

Self-attention computer ka yahi karna ka tarika hai! Jab computer "he" padhta hai, toh yeh sabhi previous words ko dekhta hai aur poochhta hai: "Mujhe 'he' samajhne ke liye in mein se har word par kitna dhyan dena chahiye?"

Yeh har word ko ek score deta hai (jaise 0% se 100% tak). Agar "detective" recently mention hua tha aur context mein sense banta hai, toh use 80% jaisa high score milta hai. Agar "suspect" mention hua tha lekin utna fit nahi baith raha, toh use 20% milta hai. Phir yeh sabhi words ke meanings ko unke scores ke basis par combine karta hai—80% "detective" ki meaning + 20% "suspect" ki meaning—"he" samajhne ke liye.

Cool part? Yeh EVERY word ke liye karta hai! Har word har doosre word ko dekhta hai aur decide karta hai ki use kitna attend karna hai. Isliye ise self-attention kehte hain—sentence khud apne aap par attention de raha hai.

Computer lakhs sentences practice karke seekhta hai ki in words ko kaise score karna hai, jab tak yeh har word ko samajhne ke liye kaun se words matter karte hain yeh samajhne mein really good nahi ho jaata.

Connections

Transformer Architecture Overview - self-attention core building block hai
Multi-Head Attention - multiple self-attention operations parallel mein chalana
Positional Encoding - self-attention ko order information provide karta hai
Attention Mechanism Basics - encoder-decoder attention (self-attention se alag)
Computational Complexity - $O (n^{2})$ bottleneck self-attention se aata hai
Gradient Flow - self-attention RNNs se better gradient flow enable karta hai

#flashcards/ai-ml

Self-attention kya hai aur yeh encoder-decoder attention se kaise alag hai? :: Self-attention tab hota hai jab ek sequence khud par attend karta hai—har position same sequence mein sabhi doosre positions ko dekh sakta hai. Encoder-decoder attention ke unlike jahan queries decoder se aur keys/values encoder se aate hain, self-attention mein Q, K, V teeno same input sequence X se alag-alag learned projections ke through aate hain.

Teen separate matrices W^Q, W^K, W^V kyun chahiye seedha embeddings use karne ki jagah?

Teen projections specialized representations create karte hain: W^Q seekhta hai kaunse aspects search karne hain, W^K seekhta hai matching ke liye kaunse aspects offer karne hain, W^V seekhta hai kaunsa content retrieve karna hai. Yeh model ko similarity aur relevance ke task-specific notions seekhne deta hai, sirf raw embedding similarity use karne ki jagah.

Derive karo ki hum attention scores ko 1/sqrt(d_k) se kyun scale karte hain

Assume karo Q aur K entries mean 0, variance 1 ke saath random hain. Dot product q·k, d_k terms sum karta hai, har ek variance 1 ke saath, toh Var(q·k) = d_k. Standard deviation sqrt(d_k) ke roop mein badhti hai. Scaling ke bina, bada d_k bahut bade scores create karta hai → softmax almost one-hot ban jaata hai → gradients vanish ho jaate hain. sqrt(d_k) se divide karne par unit variance normalize hoti hai, softmax ko responsive region mein rakhta hai.

Self-attention ki computational complexity kya hai aur yeh kahan se aati hai?

O(n²d) jahan n sequence length hai aur d model dimension hai. n² sabhi pairwise attention scores compute karne se aata hai (har position har position par attend karta hai), ek n×n attention matrix chahiye hoti hai. Yeh long sequences ke liye main bottleneck hai.

Self-attention positional encodings ke bina word order kyun nahi seekh sakta?

Self-attention permutation-invariant hai: formula Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V positions ko ek set ki tarah treat karta hai. Agar tum inputs shuffle karo, outputs sirf correspondingly shuffle ho jaate hain. Embeddings mein positional encodings add kiye bina, "dog bites man" aur "man bites dog" identical representations produce karenge (reordering tak).

Formula softmax(QK^T/sqrt(d_k))V mein, explain karo ki har matrix multiplication kya compute karta hai :: QK^T (n×d_k)(d_k×n) = (n×n) sabhi-pairs ke similarity scores deta hai—har query har key ke against. Softmax rows ko probability distributions mein normalize karta hai. Result (n×n) times V (n×d_v) = (n×d_v) weighted aggregation perform karta hai—har output position sabhi value vectors ka ek weighted sum hai, weights attention scores se aate hain.

Jab d_k bada ho aur hum scale nahi karte, toh self-attention mein gradients ka kya hota hai?

Bade d_k ke liye, dot products bade ho jaate hain (std sqrt(d_k) ke roop mein badhta hai). Softmax ke bade inputs nearly one-hot outputs create karte hain (jaise [0.01, 0.98, 0.01]). Softmax ke through backpropagating karte waqt, near-zero probabilities wale positions ke near-zero gradients hote hain → model un positions par attention adjust karna nahi seekh sakta → vanishing gradient problem.

Self-attention us long-range dependency problem ko kaise solve karta hai jisse RNNs struggle karte hain?

RNNs sequentially process karte hain, isliye position 1 ki information position 100 tak pahunchne ke liye sabhi intermediate states se guzarni hoti hai (potential degradation ke 100 steps). Self-attention directly har position ko doosre se ek step mein connect karta hai—position 100, position 1 par direct gradient path ke saath attend kar sakta hai, sequential bottleneck aur vanishing gradients se bachta hai.