Self-attention mechanism in detail: Visual walkthrough

Step 1 — A word is an arrow

WHAT. Before any attention happens, each word in a sentence is turned into a fixed-length list of numbers called an embedding. A list of numbers of length $d_{m o d e l}$ is just a point — equivalently an arrow from the origin — in a space with $d_{m o d e l}$ axes.

WHY. We cannot do arithmetic on the letters "c-a-t". We can do arithmetic on arrows: add them, measure angles between them, stretch them. Turning words into arrows is what makes the whole machine differentiable and learnable.

PICTURE. Below, three words become three arrows in a (drawn as 2-D) embedding space. The symbol $X$ is just the table that stacks these arrows as rows.

Figure — Self-attention mechanism in detail

Step 2 — One arrow, three jobs

WHAT. From each word-arrow $x_{i}$ we make three new arrows by multiplying by three learned matrices:

$query q_{i} = x_{i} W^{Q}, key k_{i} = x_{i} W^{K}, value v_{i} = x_{i} W^{V}$

$q_{i}$ ::: "what am I looking for?" — the question this word asks.
$k_{i}$ ::: "what do I offer?" — the label this word advertises.
$v_{i}$ ::: "what do I actually carry?" — the content this word hands over if chosen.
$W^{Q}, W^{K}$ ::: rectangles of shape $d_{m o d e l} \times d_{k}$ that squish the arrow into a smaller "matching" space.
$W^{V}$ ::: shape $d_{m o d e l} \times d_{v}$ , squishes into a "content" space.

WHY three, not one? A word plays different roles depending on the moment. "cat" as a questioner ("who is my verb?") is a different need than "cat" as a thing to be found ("I am a noun subject"). One arrow cannot point in the right direction for two different jobs at once, so we let the network learn three separate rotations/squishes of the same starting arrow. These matrices are the only trainable parameters here — the model discovers what to ask and what to advertise.

PICTURE. One starting arrow $x_{i}$ (white) is sent by three different matrices into three different directions.

Step 3 — Measuring "do these two match?" with a dot product

WHAT. For word $i$ 's query and word $j$ 's key we compute a single number, the score:

$s_{ij} = q_{i} \cdot k_{j} = \sum_{d = 1}^{d_{k}} q_{i d} k_{j d}$

$q_{i} \cdot k_{j}$ ::: the dot product — multiply matching entries, add them all up.
$s_{ij}$ ::: how strongly word $i$ wants to listen to word $j$ . Big = strong match.

WHY the dot product and not something else? The dot product is the cheapest honest measure of alignment. Geometrically $q_{i} \cdot k_{j} = ∥ q_{i} ∥ ∥ k_{j} ∥ cos θ$ , where $θ$ is the angle between the two arrows.

Arrows point the same way ( $θ = 0$ , $cos θ = 1$ ) → large positive → "these belong together."
Arrows are perpendicular ( $θ = 9 0^{\circ}$ , $cos = 0$ ) → zero → "unrelated."
Arrows point opposite ( $θ = 18 0^{\circ}$ , $cos = - 1$ ) → large negative → "actively mismatched."

So the dot product answers exactly the question we care about — are these two aiming in the same direction? — with one multiply-add per dimension.

PICTURE. Same-direction, perpendicular, and opposite pairs, with the sign of the score each produces.

Step 4 — All queries vs all keys, at once: $Q K^{T}$

WHAT. Stack every query as a row of $Q$ and every key as a row of $K$ . Then

$S = Q K^{T} \in R^{n \times n}, S_{ij} = q_{i} \cdot k_{j}$

$Q$ ::: $n \times d_{k}$ , one query per row.
$K^{T}$ ::: $K$ flipped so keys become columns — shape $d_{k} \times n$ .
$(n \times d_{k}) (d_{k} \times n) = (n \times n)$ ::: the inner $d_{k}$ cancels; we're left with an $n \times n$ grid.
$S_{ij}$ ::: the score of Step 3, now living at row $i$ , column $j$ .

WHY the transpose? Matrix multiplication pairs rows of the left with columns of the right. We want every query (a row of $Q$ ) dotted with every key. Keys start as rows of $K$ ; transposing turns them into columns so the multiplication lines them up. The result $S$ is a full grid: every word checked against every word — that all-pairs grid is the $n^{2}$ that drives the cost in 4.2.02-Computational-complexity-of-transformers.

PICTURE. The grid: rows are the asking words, columns the offered words, each cell a match score.

Step 5 — Cooling the scores: divide by $d_{k}$

WHAT. Before going further we shrink every score:

$S^{'} = \frac{S}{d _{k}}$

$d_{k}$ ::: the square root of the query/key dimension — a single fixed number.
$S^{'}$ ::: the scaled scores.

WHY exactly $d_{k}$ ? The dot product in Step 3 adds $d_{k}$ terms. If each term is a random number with average size 1, then adding $d_{k}$ of them makes the total spread (standard deviation) grow like $d_{k}$ — not $d_{k}$ , because random terms partly cancel. Formally, for independent mean-0 variance-1 entries, $Var (q_{i} \cdot k_{j}) = \sum_{d = 1}^{d_{k}} Var (q_{i d} k_{j d}) = d_{k} \Rightarrow std = d_{k} .$ Dividing by $d_{k}$ pulls the spread back to 1 — no matter how large $d_{k}$ is.

Why do we care about the spread? The next step is softmax. Feed softmax huge numbers and it becomes almost one-hot: one weight ≈ 1, the rest ≈ 0. There, the slope is nearly flat, so gradients vanish and learning stalls (see 5.3.03-Gradient-flow-in-deep-networks). Scaling keeps softmax in its gentle, learnable middle.

PICTURE. Same scores before and after cooling, and what softmax does to each — peaked vs. balanced.

Step 6 — Turning scores into a budget: softmax per row

WHAT. Along each row we run softmax:

$α_{ij} = \frac{e x p ( s _{ij}^{'} )}{m = 1 \sum n exp ( s _{im}^{'} )}$

$exp (\cdot)$ ::: the exponential — turns any number into a positive one and stretches gaps.
numerator ::: word $i$ 's raw appetite for word $j$ .
denominator ::: total appetite in row $i$ , so the row sums to 1.
$α_{ij}$ ::: the final attention weight — a fraction of word $i$ 's attention spent on word $j$ .

WHY softmax and not plain normalising? Three properties we need, all delivered at once:

Positivity. $exp$ is always positive, so a weight can drop to ~0 (turn a word off) but never go negative.
Contrast. $exp$ amplifies gaps: a slightly higher score becomes a noticeably bigger slice.
Smoothness. It is differentiable everywhere, which backprop demands.

PICTURE. One row of the grid becoming a bar chart that sums to 1 — a "spending budget" of attention.

Step 7 — Spending the budget: $Z = A V$

WHAT. Each output arrow is the attention-weighted blend of all the value arrows:

$z_{i} = \sum_{j = 1}^{n} α_{ij} v_{j}, Z = A V$

$A$ ::: the $n \times n$ table of weights $α_{ij}$ from Step 6.
$V$ ::: the value arrows (Step 2) stacked as rows.
$z_{i}$ ::: word $i$ 's new representation — a mixture of everyone's content, in the proportions its budget dictated.

WHY values, not embeddings? Query/key decided who to listen to; the actual content pulled in is a separately learned view $V$ . Splitting "who to trust" from "what they say" gives the model an extra lever.

PICTURE. One output arrow drawn as a weighted sum of value arrows — the long weight bends the result toward the trusted word.

Step 8 — Edge & degenerate cases (never surprised)

WHAT & WHY. A robust reader must know what happens at the extremes:

Case	What the math does	Why it makes sense
All scores equal in a row	softmax → uniform $\frac{1}{n}$	No word stands out, so the output is a plain average — a sensible fallback.
One score huge, rest tiny	softmax → ~one-hot	Attention becomes a hard copy of one value arrow. This is why over-large scores (no scaling) hurt learning.
$n = 1$ (single word)	$A = [1]$ , $z_{1} = v_{1}$	With no one else to attend to, a word can only attend to itself. Perfectly defined.
Perfectly opposite $q_{i}, k_{j}$	score strongly negative → $α_{ij} \to 0$	A mismatched word is muted, never chosen.
Masked position (future token)	set $s_{ij}^{'} = - \infty$ before softmax	$exp (- \infty) = 0$ , so that word gets exactly 0 budget — the trick behind causal decoding.

PICTURE. Three softmax outputs side by side: uniform (flat), peaked (spike), and masked (a bar forced to zero).

The one-picture summary

Everything on this page is a single pipeline: arrows in → three roles → all-pairs match → cool → budget → blend → arrows out.

Recall Feynman retelling — say it back in plain words

Every word starts as an arrow. From that one arrow we build three: a question it asks, a label it wears, and the stuff it carries. To decide how much word A should listen to word B, we check whether A's question points the same way as B's label — that's a dot product, one number per pair, giving a full grid of "who-matches-whom." Big dot products from a big dimension get scary, so we cool them by $d_{k}$ to keep the softmax gentle. Softmax turns each row of the grid into a spending budget that adds up to 1: it must be positive, it exaggerates the winners, and it's smooth enough to train. Finally each word spends its budget by mixing everyone's stuff in those proportions — that blended arrow is the output. If everyone ties, you get a plain average; if one wins big, you get a copy; if a word is masked, its budget is forced to zero. Same idea, one word or a thousand.