Transformers — attention mechanism, self-attention: Visual walkthrough

We work a running toy example the whole way: three telemetry snapshots from an engine — call them token 1, token 2, token 3 (imagine: temperature reading, pressure reading, rpm reading at three moments). Everything stays 2-dimensional so we can literally draw the vectors on the board.

Step 1 — Start with the raw inputs (what is a "token vector"?)

WHAT. We place three little arrows on the board, one per snapshot. Each arrow is a point in a flat plane — two numbers, so two axes.

WHY. Before any clever machinery, attention needs something to compare. The comparison will happen between arrows, so we must first see the arrows. Nothing is learned yet; these are the plain inputs.

PICTURE. Look at the three chalk arrows below. They point in different directions — that difference in direction is the only thing attention will care about in the next steps.

Figure — Transformers — attention mechanism, self-attention

Step 2 — Split each input into three roles: Query, Key, Value

WHAT. Each single input arrow fans out into three arrows living in their own copies of the plane.

WHY. One number cannot play two contradictory jobs at once. "How relevant are you to me?" (matching) and "what information do you carry?" (content) are different questions — so we give each token a separate arrow for searching ( $q$ ), for being found ( $k$ ), and for being read ( $v$ ). This is exactly the search-request-vs-label idea.

PICTURE. Below, input $x_{2}$ (chalk white) splits into $q_{2}$ (yellow), $k_{2}$ (blue), $v_{2}$ (pink). Same origin, three destinations.

Step 3 — Measure similarity with the dot product

WHAT. For token 1's query, we compute one score against every key: $s_{11}, s_{12}, s_{13}$ .

WHY THIS TOOL (dot product, not something else)? We need a single number that is big when two arrows point the same way and small (or negative) when they don't. The dot product does exactly that: $q \cdot k = ∣ q ∣ ∣ k ∣ cos θ$ , where $θ$ is the angle between them. Same direction ⇒ $cos θ = 1$ ⇒ large score; perpendicular ⇒ $cos θ = 0$ ⇒ zero; opposite ⇒ negative. No other simple operation reads "alignment" so cheaply. See Dot product & cosine similarity.

PICTURE. The angle $θ$ between $q_{1}$ and each key controls the score. Notice the aligned pair (small angle) gets the fat score bar; the perpendicular pair gets ≈0.

Step 4 — Scale the scores by $d_{k}$

WHAT. Divide every raw score by the same constant $d_{k}$ . In our toy example $d_{k} = 2$ , so we divide by $2 \approx 1.414$ .

WHY. A dot product adds up $d_{k}$ little products. Add more terms and the sum swings wider — its spread grows like $d_{k}$ . If scores get huge, the next step (softmax) turns into a hard "winner takes all" that has flat gradients everywhere else — the model stops learning. See Gradient vanishing/saturation. Dividing by $d_{k}$ pulls the spread back to ≈1, keeping the next step responsive.

PICTURE. Two score profiles for the same directions: raw (tall, spiky) vs scaled (shorter, gentler). The spiky one is heading for saturation.

Step 5 — Turn scores into weights with softmax

WHAT. Convert token 1's three scaled scores into three non-negative numbers that add to 1.

WHY THIS TOOL (softmax, not "divide by the total")? We want a weighted average next, so the weights must be non-negative and sum to 1 — a probability spread over positions. Plain scores can be negative; softmax's exponential kills negatives (they map to small positives) and smoothly amplifies the leader while staying differentiable so gradients can flow. See Softmax function.

PICTURE. Three scaled scores go into the softmax box; out come three weight bars whose heights stack exactly to 1.

Step 6 — Blend the Values (the soft lookup)

WHAT. Stretch each value arrow by its weight and add them tip-to-tail. The landing point is $z_{1}$ .

WHY. Only now do we pull in actual content. Steps 3–5 decided how much to trust each token; this step finally reads them, most strongly the best match. This is the "soft, differentiable database lookup": no hard pick, a smooth blend.

PICTURE. $v_{1}$ scaled by 0.67 plus $v_{2}$ scaled by 0.33 — the pink resultant $z_{1}$ leans toward $v_{1}$ but is nudged by $v_{2}$ .

Step 7 — Stack every token: the whole thing is two matrix multiplies

WHAT. Do Steps 3–6 for all tokens simultaneously by replacing "one query dotted with all keys" with "matrix $Q$ times matrix $K^{⊤}$ ."

WHY. This is the whole selling point over RNNs and LSTMs: every position reads every other position in one parallel matrix product — no step-by-step loop, no long-range forgetting.

PICTURE. The $3 \times 3$ attention grid: row = asker, column = asked. Bright cells = high weight. Each row sums to 1.

Step 8 — Edge & degenerate cases (so no scenario surprises you)

WHAT / WHY. A derivation is only trustworthy if it survives the extremes. Four cases:

All scores equal (queries match nothing in particular). Then every $α_{ij} = 1/ n$ : attention becomes a plain average of all values. Nothing breaks — it degrades gracefully to "look at everyone equally."
One score dominates (near one-hot). Softmax → ≈ $(1, 0, 0)$ ; output ≈ that single value. This is hard lookup as a limiting case, and is exactly what Step 4's scaling keeps us from hitting too early.
Single token ( $n = 1$ ). $Q K^{⊤}$ is $1 \times 1$ ; softmax of one number is $1$ ; output is just $v_{1}$ . Self-attention on a lone element returns that element's value — sensible.
Negative / opposite arrows. $q \cdot k < 0$ gives a small (but positive!) softmax weight — never zero, never negative. So a token is never fully ignored, just heavily down-weighted.

PICTURE. Three weight profiles side by side: uniform (case 1), spiky (case 2), and the single-token bar (case 3).

Recall One more edge case: order-blindness

Shuffle the input tokens and the output rows shuffle identically — attention has no built-in sense of time or order. That is why real Transformers bolt on Positional Encoding before this whole machine runs.

The one-picture summary

Everything above, compressed into a single flow: raw arrows → split into Q/K/V → dot products → scale → softmax → weighted sum. Follow the arrows left to right.

Recall Feynman retelling of the whole walkthrough

Picture a small group of engine snapshots on a chalkboard, each a little arrow. Give every arrow three jobs: a search request (query), a name tag (key), and a fact sheet (value). To decide who each snapshot should listen to, compare its search request to everyone's name tag by seeing how much the arrows point the same way — that's the dot product. Those raw match numbers can get wildly big when the arrows are long, which would make the next step panic, so we shrink them by $d_{k}$ to keep them calm. Then we run them through softmax, which turns them into a fair set of percentages that add up to 100%. Finally, each snapshot builds its answer by mixing everyone's fact sheets in those percentages — mostly the best match, a little of the rest. Do that for every snapshot at once with two matrix multiplies, and you have the whole attention formula. The only things ever learned are the three projection matrices; the attention percentages are recomputed fresh for every new input. And because it treats the sequence as a bag, we must separately tell it the order — that's positional encoding. See Multi-Head Attention for running several of these lookup strategies in parallel, and Time-series anomaly detection (aerospace telemetry) for where a distant sensor spike gets pulled in with no information loss.

Active Recall

Which vector answers "what am I looking for?"

The Query

q_{i} = W_{Q} x_{i}

Why is the dot product the right similarity tool here?

Because

q \cdot k = ∣ q ∣∣ k ∣ cos θ

is large when the arrows point the same way — it directly measures directional alignment.

Why scale by

d_{k}

and not

d_{k}

The spread of a

d_{k}

-term dot product grows like

d_{k}

; dividing by

d_{k}

normalises the spread to ~1. Dividing by

d_{k}

would over-shrink toward uniform.

If all scaled scores are equal, what does attention output?

A plain uniform average of all value vectors (

α_{ij} = 1/ n

Is a token with a negative score ever fully ignored?

No — softmax makes even negative scores map to small positive weights, so its contribution is tiny but never zero.

Along which axis is softmax applied in

Q K^{⊤}

Row-wise (across keys), so each asking token's weights sum to 1.

What is the numeric output

z_{1}

in the worked example?

(1.340, 0.990)