Scaled dot-product attention

4.1.4 · AI-ML › Transformer Architecture

Scaled Dot-Product Attention Kya Hai?

Components breakdown:

$Q K^{T}$ : Compatibility scores (har query kitna acha match karti hai har key se)
$\frac{1}{d _{k}}$ : Scaling taaki gradient saturation na ho
softmax: Scores ko probability distribution mein convert karta hai (weights ka sum 1 hota hai)
$V$ se multiply karo: Attention weights ke basis par values ka weighted sum

First Principles Se Derivation

Step 1: Similarity Ke Liye Dot Product Kyun?

Fundamental sawaal se shuru karo: Hum kaise measure karein ki query $q$ , key $k$ se match karti hai ya nahi?

Dot product $q \cdot k = \sum_{i = 1}^{d_{k}} q_{i} k_{i}$ alignment measure karta hai:

Agar $q$ aur $k$ same direction mein point karein: badi positive value
Agar orthogonal hain: zero
Agar opposite hain: badi negative value

Ye step kyun? Humein ek differentiable similarity metric chahiye. Dot product computationally cheap hai ( $O (d_{k})$ ) aur embedding spaces mein semantic similarity naturally capture karta hai.

Step 2: Batch Processing Ke Liye Matrix Form

$n$ queries aur $m$ keys ke liye: $S = Q K^{T} \in R^{n \times m}$

jahan $S_{ij} = q_{i} \cdot k_{j}$ query $i$ aur key $j$ ke beech compatibility score hai.

Ye step kyun? Modern hardware (GPUs/TPUs) matrix multiplication ke liye optimized hai. Saari queries ek saath batch karna looping se 100x faster hai.

Step 3: Scaling Problem

Observation: Agar $Q$ aur $K$ ke entries $N (0, 1)$ se draw ki gayi hain, tab:

$E [q \cdot k] = E [\sum_{i = 1}^{d_{k}} q_{i} k_{i}] = \sum_{i = 1}^{d_{k}} E [q_{i}] E [k_{i}] = 0$

$Var (q \cdot k) = \sum_{i = 1}^{d_{k}} Var (q_{i} k_{i}) = \sum_{i = 1}^{d_{k}} E [q_{i}^{2}] E [k_{i}^{2}] = d_{k}$

Toh standard deviation $d_{k}$ ke roop mein grow karta hai! Typical $d_{k} = 64$ ke liye, dot products ka std $\approx 8$ hota hai.

Ye kyun matter karta hai: Softmax hai $σ (z)_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$ . Jab inputs ki magnitude badi ho:

Badi positive values: softmax $\approx$ one-hot ho jaata hai (gradient $\approx 0$ )
Relative differences shrink ho jaate hain: seekhna mushkil ho jaata hai ki kaunsi key thodi si better hai

Fix: $d_{k}$ se divide karo taaki variance wapas 1 ho jaaye: $Var (\frac{q \cdot k}{d _{k}}) = \frac{d _{k}}{d _{k}} = 1$

Step 4: Softmax Normalization

Scores ko attention weights mein convert karne ke liye row-wise softmax apply karo: $A = softmax (\frac{Q K ^{T}}{d _{k}}) \in R^{n \times m}$

jahan $A_{ij} = \frac{e x p ( S _{ij} / d _{k} )}{\sum _{j^{'} = 1}^{m} e x p ( S _{i j^{'}} / d _{k} )}$ .

Ye step kyun?

Probabilistic interpretation: $A$ ki har row keys ke upar ek distribution hai ( $\sum_{j} A_{ij} = 1$ )
Differentiability: Backprop ke liye smooth function
Competition: Keys compete karti hain — ek ko zyada attend karne ka matlab doosre ko kam attend karna

Step 5: Weighted Value Aggregation

Aakhir mein, values ka weighted average lene ke liye attention weights use karo: $output_{i} = \sum_{j = 1}^{m} A_{ij} v_{j}$

Matrix form mein: $Attention (Q, K, V) = A V \in R^{n \times d_{v}}$

Ye step kyun? Query "sawaal poochti hai," keys "sawaal match karti hain," aur values "jawab deti hain." Hum relevance ke basis par ek soft-blended jawab retrieve karte hain.

Worked Examples

Worked example Example 1: Single Head, Chhoti Dimensions

Setup: $d_{k} = 4$ , $d_{v} = 3$ , ek query, do keys.

$Q = [1010], K = [100110], V = [500300]$

Step 1: $Q K^{T}$ compute karo $Q K^{T} = [1010] 10100101 = [20]$

Ye step kyun? Query, key 1 se perfectly match karti hai (dono ka pattern [1,0,1,0] hai), key 2 se bilkul match nahi karti.

Step 2: $d_{k} = 2$ se scale karo $\frac{Q K ^{T}}{d _{k}} = [10]$

Ye step kyun? Saturation prevent karta hai; ab scores order-1 hain.

Step 3: Softmax $A = softmax ([1, 0]) = [\frac{e ^{1}}{e ^{1} + e ^{0}} \frac{e ^{0}}{e ^{1} + e ^{0}}] \approx [0.731 0.269]$

Ye step kyun? Query, key 1 ko 73% attend karti hai, key 2 ko 27%. Step 4: Values ka weighted sum $output = 0.731 500 + 0.269 030 = 3.655 0.807 0$

Interpretation: Output mostly value 1 hai (logical hai — query ne key 1 ko best match kiya tha).

Feynman Explanation

Recall Ek 12-saal ke bacche ko samjhao

Socho tum homework kar rahe ho aur apni textbook se sawaal pooch sakte ho. Lekin tumhari textbook BAHUT BADI hai — hundreds of pages.

Attention ek smart search system jaisi hai:

Query (Q): Tum apna sawaal likhte ho: "Photosynthesis kya hai?"
Keys (K): Textbook ke har page ka ek title/summary hai (jaise "Plants," "Cells," "Energy," "History")
Comparison: Tum check karte ho ki tumhara sawaal har page title se kitna match karta hai. "Plants" page bahut acha match hai (score = 10), "Energy" theek hai (score = 5), "History" barely match karta hai (score = 1).
Scaling Trick: Agar tumhari textbook BAHUT BADI hoti (hazaaron pages), toh scores bahut bade ho jaate (best match ke liye score = 1000). Isse tumhara dimaag kehta "SIRF is ek page ko dekho" aur baaki sab ignore karo. Scaling matlab hai "chalao scores 1-10 ke beech rakhein taaki main multiple pages consider karun."
Softmax: Scores ko percentages mein convert karo: 70% Plants, 25% Energy, 5% History.
Values (V): Har page mein actual content hai (plants, energy, etc. ke baare mein paragraphs)
Output: Tum Plants page ka 70%, Energy page ka 25%, History page ka 5% padhte ho, aur unhe ek jawab mein blend karte ho.

Scaling kyun zaroori hai: Iske bina, tum ek robot ban jaate jo SIRF top match padhta hai aur baaki sab ignore karta hai — chahe second-best page mein bhi useful info ho! Scaling tumhare dimaag ko multiple sources ke liye open rakhti hai.

Connections

Multi-Head Attention — parallel mein multiple scaled dot-product attention operations use karta hai
Self-Attention — jab Q, K, V sab same sequence se aate hain
Cross-Attention — jab Q ek sequence se aata hai, K/V doosre se (encoder-decoder)
Softmax Function — normalization function jo scores ko probabilities mein convert karta hai
Gradient Descent — stable learning ke liye scaling kyun matter karta hai
Positional Encoding — sequence order information provide karta hai (attention permutation-invariant hai)
Query-Key-Value Model — conceptual framework jo information retrieval se liya gaya hai
Attention Mask — certain positions ko attend karne se rokne ke liye use hota hai (future tokens, padding)

Last updated: 2026-07-01 | Multi-head attention par jaane se pehle ye master karo