Transformers — attention mechanism, self-attention

5.6.14 · Coding › Machine Learning (Aerospace Applications)

Attention exist kyun karta hai?

Self-attention kya hai? (ke pieces)

Formula kaise banaate hain? (scratch se derivation)

Step 1 — Similarity measure karo. Position $i$ ko position $j$ kitna attend karna chahiye? Unke query aur key ka dot product use karo: $s_{ij} = q_{i} \cdot k_{j}$ Yeh step kyun? Dot product tab bada hota hai jab vectors ek hi direction mein point karte hain — ek natural "yeh dono ek doosre ke liye relevant hain" score.

Step 2 — Scale karo. Agar keys/queries ki dimension $d_{k}$ hai, toh $q_{i} \cdot k_{j}$ $d_{k}$ random products ka sum hai, jiska variance $d_{k}$ ke saath badhta hai. Badi values softmax ko saturated regions mein push karti hain (tiny gradients). Variance $\approx 1$ rakhne ke liye divide karo: $s_{ij} = \frac{q _{i} \cdot k _{j}}{d _{k}}$ $d_{k}$ kyun? Agar har component ka variance 1 hai, toh dot product ka variance $d_{k}$ hai, toh standard deviation $d_{k}$ . Isse divide karna re-normalise karta hai.

Step 3 — Scores ko weights mein badlo. Hume chahiye non-negative weights jo 1 mein sum hon (positions par ek probability distribution). Softmax exactly yahi karta hai: $α_{ij} = \frac{e ^{s_{ij}}}{\sum _{m} e ^{s_{im}}}$ Yeh step kyun? Ek weighted average ko aise weights chahiye jo 1 mein sum hon; softmax sabse bade score ko bhi smoothly aur differentiably amplify karta hai.

Step 4 — Values blend karo. Position $i$ ka output values ka weighted sum hai: $z_{i} = \sum_{j} α_{ij} v_{j}$ Yeh step kyun? Hum finally content (values) pull in karte hain, relevance se weighted — yeh hai "soft lookup."

Saare positions ko matrices $Q, K, V$ mein stack karo (rows = tokens):

Figure — Transformers — attention mechanism, self-attention

Multi-Head Attention (80/20 extension)

Worked Examples

Common Mistakes (Steel-man + fix)

Recall Feynman: ek 12-saal ke bachche ko explain karo

Imagine karo tum class mein ho aur teacher ek sawaal poochti hai (tumhara query). Har classmate ek card uthata hai jisme ek topic label hai (unki key) aur kuch facts jaanta hai (unki value). Tum jaldi saare labels dekhte ho, decide karte ho ki kaun se classmates most relevant hain, phir unke facts ko mix karte ho — sabse zyada un logon par attention dete ho jinke keys best match kiye. Woh mixing hi jawab hai. Self-attention poori class ka yeh kaam simultaneously karna hai, har koi har kisi ke baare mein poochh raha hai, toh kisi ki bhi information forget nahi hoti.

Active Recall

Self-attention har input se kaun se teen vectors banata hai, aur unke roles kya hain?

Query (main kya dhoondh raha hoon), Key (main kya offer karta hoon), Value (content jo main pass karunga agar select hua).

Scaled dot-product attention formula likhо.

softmax (\frac{Q K ^{⊤}}{d _{k}}) V

d_{k}

se divide kyun karte hain?

d_{k}

-dim vectors ke dot product ka variance

\approx d_{k}

hota hai;

d_{k}

se divide karna variance ko ~1 par normalise karta hai, softmax saturation aur vanishing gradients rokta hai.

Softmax kis axis par apply hota hai?

Keys/positions ke across (row-wise

Q K^{⊤}

par), taaki tokens par attention weights 1 mein sum hon.

Actual learned parameters kya hain (attention weights nahi)?

W_{Q}, W_{K}, W_{V}

aur output projection

W_{O}

Self-attention ko positional encodings ki zarurat kyun hai?

Yeh permutation-equivariant hai — isme koi built-in notion of order nahi hai, isliye position info add karni padti hai.

Attention RNNs ki kaun si problems solve karta hai?

Long-range forgetting aur lack of parallelism — har position doosri position se directly, ek parallel matrix multiply mein padh sakti hai.

Multi-head attention se kya fayda hota hai?

Multiple parallel attention subspaces, jo alag-alag tarah ke relationships mein specialise karte hain, phir concatenate hote hain.

n

tokens ke liye attention matrix

Q K^{⊤}

ki shape kya hai?

n \times n

(positions ke har pair ke liye).

"Self"-attention mein Q, K, V kahan se aate hain?

Teeno usi input sequence se.

Connections

Softmax function — scores ko probability distribution mein badalta hai
Dot product & cosine similarity — relevance score
Positional Encoding — sequence order inject karta hai
Multi-Head Attention — parallel attention subspaces
RNNs and LSTMs — woh predecessors jinhein attention ne replace kiya
Transformer Architecture — encoder/decoder inhi blocks se bana hai
Gradient vanishing/saturation — scaling kyun matter karta hai
Time-series anomaly detection (aerospace telemetry) — ek application