Transformers — attention mechanism, self-attention

WHY does attention exist?

WHAT is self-attention? (the pieces)

HOW do we build the formula? (derivation from scratch)

Step 1 — Measure similarity. How much should position $i$ attend to position $j$ ? Use the dot product of their query and key: $s_{ij} = q_i \cdot k_j$ Why this step? Dot product is large when vectors point the same way — a natural "these two are relevant to each other" score.

Step 2 — Scale it. If keys/queries have dimension $d_k$ , then $q_i\cdot k_j$ is a sum of $d_k$ random products, whose variance grows with $d_k$ . Large values push softmax into saturated regions (tiny gradients). Divide to keep variance $\approx 1$ : $s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}$ Why $\sqrt{d_k}$ ? If each component has variance 1, the dot product has variance $d_k$ , so standard deviation $\sqrt{d_k}$ . Dividing by it re-normalises.

Step 3 — Turn scores into weights. We want non-negative weights summing to 1 (a probability distribution over positions). Softmax does exactly this: $\alpha_{ij} = \frac{e^{s_{ij}}}{\sum_{m} e^{s_{im}}}$ Why this step? A weighted average needs weights that sum to 1; softmax also amplifies the biggest score smoothly and differentiably.

Step 4 — Blend the values. The output for position $i$ is the weighted sum of values: $z_i = \sum_j \alpha_{ij}\, v_j$ Why this step? We finally pull in content (values), weighted by relevance — this is the "soft lookup."

Stacking all positions into matrices $Q,K,V$ (rows = tokens):

Figure — Transformers — attention mechanism, self-attention

Multi-Head Attention (the 80/20 extension)

Worked Examples

Worked example Example 1 — tiny 2-token self-attention by hand

Let $d_k=2$ , two tokens with (already-projected) values: $q_1=(1,0),\ q_2=(0,1)$ ; $k_1=(1,0),\ k_2=(0,1)$ ; $v_1=(2,0),\ v_2=(0,3)$ .

Scores for token 1 (before scaling): $s_{11}=q_1\cdot k_1=1$ , $s_{12}=q_1\cdot k_2=0$ . Why? Token 1's query aligns with key 1, not key 2.

Scale by $\sqrt{d_k}=\sqrt2$ : $s_{11}=0.707,\ s_{12}=0$ . Softmax: $\alpha_{11}=\frac{e^{0.707}}{e^{0.707}+e^{0}}=\frac{2.028}{3.028}=0.670$ , $\alpha_{12}=0.330$ . Output: $z_1 = 0.670\,v_1 + 0.330\,v_2 = (1.34,\ 0.99)$ . Why this step? Token 1 mostly grabs $v_1$ but mixes in some $v_2$ — a soft blend.

Common Mistakes (Steel-man + fix)

Recall Feynman: explain it to a 12-year-old

Imagine you're in class and the teacher asks a question (your query). Every classmate holds up a card with a topic label (their key) and knows some facts (their value). You quickly glance at all the labels, decide which classmates are most relevant, and then mix their facts together — paying most attention to the best-matching ones. That mixing is the answer. Self-attention is the whole class doing this simultaneously, everyone asking about everyone, so no one's information gets forgotten.

Active Recall

What three vectors does self-attention create from each input, and their roles?

Query (what I seek), Key (what I offer), Value (content I pass on if selected).

Write the scaled dot-product attention formula.

\operatorname{softmax}\!\big(\frac{QK^\top}{\sqrt{d_k}}\big)V

Why divide by

\sqrt{d_k}

The dot product of

d_k

-dim vectors has variance

\approx d_k

; dividing by

\sqrt{d_k}

normalises variance to ~1, preventing softmax saturation and vanishing gradients.

Along which axis is softmax applied?

Across keys/positions (row-wise on

QK^\top

), so attention weights over tokens sum to 1.

What are the actual learned parameters (not the attention weights)?

W_Q, W_K, W_V

and the output projection

W_O

Why does self-attention need positional encodings?

It is permutation-equivariant — it has no inherent notion of order, so position info must be added.

What problem of RNNs does attention solve?

Long-range forgetting and lack of parallelism — every position reads every other directly, in one parallel matrix multiply.

What does multi-head attention buy you?

Multiple parallel attention subspaces, each specialising in different relationship types, then concatenated.

Shape of the attention matrix

QK^\top

for

n

tokens?

n\times n

(every pair of positions).

In "self"-attention, where do Q, K, V come from?

All from the same input sequence.

Connections

Softmax function — turns scores into a probability distribution
Dot product & cosine similarity — the relevance score
Positional Encoding — injects sequence order
Multi-Head Attention — parallel attention subspaces
RNNs and LSTMs — the predecessors attention replaced
Transformer Architecture — encoder/decoder built from these blocks
Gradient vanishing/saturation — why scaling matters
Time-series anomaly detection (aerospace telemetry) — an application

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, attention ka core idea bilkul simple hai: har token (ya har timestep) ko decide karna hota hai ki poori sequence me se kis-kis cheez pe dhyaan dena hai. Iske liye hum har input se teen versions banate hain — Query (main kya dhoond raha hoon), Key (main kya offer karta hoon), aur Value (agar mujhe choose kiya to main kya content doonga). Query aur Key ka dot product nikaal ke similarity score milta hai, phir softmax se woh score probabilities ban jaate hain (sum = 1), aur finally Values ka weighted average lete hain. Bas — yeh ek "soft database lookup" hai jo differentiable hai, matlab gradient descent se train ho sakta hai.

$\sqrt{d_k}$ se divide karna kyun zaroori hai? Kyunki jab dimension bada hota hai, dot product ki value bahut badi ho jaati hai, aur softmax "saturate" ho ke gradient ko zero kar deta hai. Divide karne se variance normal ho jaata hai aur training stable rehti hai. Yeh chhota sa step deep Transformers ke kaam karne ki badi wajah hai — pure 80/20 concept.

RNN/LSTM me information ek-ek step karke pass hoti thi, is liye door ki cheezein bhool jaati thi aur parallel training possible nahi thi. Attention me har position direct har doosri position ko dekh leti hai ek hi matrix multiply me — na koi bhoolna, na koi slowness. Aerospace me socho: engine ke telemetry data me abhi ka anomaly 60 timesteps pehle ke pressure drop se juda ho sakta hai — attention us purani ghatna ko high weight de deta hai bina signal kho aaye.

Ek important baat yaad rakho: attention ke pass order ka koi natural sense nahi hota (permutation-equivariant hota hai), is liye alag se positional encoding add karni padti hai. Aur jo "attention weights" dikhte hain woh learn nahi hote — woh input ke hisaab se on-the-fly compute hote hain; asli learnable cheezein $W_Q, W_K, W_V, W_O$ matrices hain.

Transformers — attention mechanism, self-attention

WHY does attention exist?

WHAT is self-attention? (the pieces)

HOW do we build the formula? (derivation from scratch)

Multi-Head Attention (the 80/20 extension)

Worked Examples

Common Mistakes (Steel-man + fix)

Active Recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Machine Learning (Aerospace Applications)

Connections