Transformers — attention mechanism, self-attention: Worked examples

Before anything, one reminder of what each symbol is, in plain words, so a first-time reader can follow every line below:

The scenario matrix

Every situation attention can hand you falls into one of these cells. The examples below are tagged with the cell(s) they cover.

#	Case class	What is special	Covered by
A	Aligned query/key ( $s > 0$ )	query points toward a key	Ex 1
B	Opposed query/key ( $s < 0$ )	query points away — small weight	Ex 1, Ex 2
C	Orthogonal ( $s = 0$ )	dot product zero — neutral	Ex 1
D	Exact tie (all scores equal)	uniform weights, degenerate	Ex 3
E	Zero query ( $q_{i} = 0$ )	degenerate input	Ex 4
F	Scaling effect (large $d_{k}$ )	saturation vs healthy gradient	Ex 5
G	Limiting case (score $\to \infty$ )	softmax collapses to one-hot	Ex 6
H	Full $n$ -token matrix	row-wise softmax on $Q K^{⊤}$	Ex 7
I	Real-world word problem	aerospace telemetry attends across time	Ex 8
J	Exam twist (permutation)	shuffle inputs → what happens?	Ex 9

Example 1 — aligned, opposed, and orthogonal in one shot (cells A, B, C)

Figure — Transformers — attention mechanism, self-attention

Step 1 — raw scores $s_{j} = q \cdot k_{j}$ . $s_{1} = 1 \cdot 1 + 0 \cdot 0 = 1$ , $s_{2} = 1 \cdot (- 1) + 0 = - 1$ , $s_{3} = 1 \cdot 0 + 0 \cdot 1 = 0$ . Why this step? The dot product measures alignment; look at the figure — the amber arrow $q$ points right, so it agrees with $k_{1}$ (+1), fights $k_{2}$ ( $- 1$ ), ignores $k_{3}$ (0).

Step 2 — scale by $d_{k} = 2$ . $s_{1} = 1/ 2 = 0.7071$ , $s_{2} = - 0.7071$ , $s_{3} = 0$ . Why this step? Keeps the numbers in softmax's healthy range (parent Step 2).

Step 3 — softmax the row. Exponentials: $e^{0.7071} = 2.0281$ , $e^{- 0.7071} = 0.4931$ , $e^{0} = 1$ . Sum $= 3.5212$ . $α_{1} = 0.5760, α_{2} = 0.1400, α_{3} = 0.2840$ . Why this step? We need fractions that add to 1 so the output is a genuine average.

Step 4 — blend the values. $z = 0.5760 (3, 0) + 0.1400 (0, 3) + 0.2840 (0, 0) = (1.7279, 0.4201)$ . Why this step? This finally pulls in content, weighted by relevance.

Verify: weights sum $0.5760 + 0.1400 + 0.2840 = 1.000$ ✓. The aligned key won the most weight (0.576), the opposed key least (0.140) — matching the forecast. Orthogonal sits in the middle, as it should for score 0.

Example 2 — negative scores don't mean negative weights (cell B)

Step 1 — exponentiate. $e^{- 2} = 0.1353, e^{- 5} = 0.006738, e^{- 1} = 0.3679$ . Why this step? $e^{x}$ is always positive, even for negative $x$ — this is exactly why softmax can never output a negative weight.

Step 2 — normalise. Sum $= 0.5099$ . $α = (0.2653, 0.01321, 0.7214)$ . Why this step? Softmax cares only about differences between scores, not their absolute sign.

Verify: all $α > 0$ ✓, sum $= 1.000$ ✓. The least-negative score ( $- 1$ ) got the largest weight (0.721). So the answer to the forecast: false — negative scores are fine.

Example 3 — the perfect tie (cell D, degenerate)

Step 1 — notice the symmetry. All exponentials equal $e^{4}$ . Why this step? Identical scores means the model has no reason to prefer any position.

Step 2 — softmax. $α_{j} = \frac{e ^{4}}{4 e ^{4}} = \frac{1}{4} = 0.25$ for each. Why this step? The constant cancels top and bottom — proving softmax ignores an added constant.

Step 3 — output. $z = \frac{1}{4} (v_{1} + v_{2} + v_{3} + v_{4})$ : the plain average. Why this step? With uniform weights, "attention" degenerates into an ordinary mean.

Verify: $4 \times 0.25 = 1.000$ ✓. Note $s = (4, 4, 4, 4)$ and $s = (0, 0, 0, 0)$ give the same result — softmax is invariant to adding a constant to every score.

Example 4 — the zero query (cell E, degenerate input)

Step 1 — scores. $q_{i} \cdot k_{j} = 0$ for every $j$ , whatever the keys are. Why this step? The zero vector is orthogonal to everything.

Step 2 — softmax of all-zeros. $α_{j} = \frac{e ^{0}}{\sum e ^{0}} = \frac{1}{n}$ (here $\frac{1}{3}$ ). Why this step? All-equal scores → uniform weights (same mechanism as Ex 3).

Verify: $3 \times \frac{1}{3} = 1.000$ ✓. A zero query doesn't crash — it just averages everything equally. Attention gracefully degrades to "look at all timesteps the same."

Example 5 — scaling saves the gradient (cell F)

Step 1 — unscaled softmax. $e^{10} = 22026$ , $e^{0} = 1$ . Sum $= 22027$ . $α = (0.99995, 0.0000454)$ . Why this step? One score dominates so hard the other is starved — nearly one-hot.

Step 2 — scaled softmax ( $s \to (1, 0)$ ). $e^{1} = 2.7183, e^{0} = 1$ , sum $= 3.7183$ . $α = (0.7311, 0.2689)$ . Why this step? Bringing scores near variance 1 keeps both weights well away from 0/1, so gradients through both survive (avoids Gradient vanishing/saturation).

Verify: unscaled sum $= 1.000$ ✓, scaled sum $= 1.000$ ✓. The second weight jumps from $4.5 \times 1 0^{- 5}$ to $0.269$ — a $\sim$ 5900× revival of its gradient. That is why we divide by $d_{k}$ .

Example 6 — the infinite-score limit (cell G, limiting behaviour)

Step 1 — write the weight. $α_{1} = \frac{e ^{M}}{e ^{M} + 1} = \frac{1}{1 + e ^{- M}}$ . Why this step? Divide top and bottom by $e^{M}$ so we can read the limit cleanly.

Step 2 — take $M \to \infty$ . $e^{- M} \to 0$ , so $α_{1} \to 1$ , $α_{2} \to 0$ . Why this step? This is the exact "one-hot" collapse: attention picks a single token and ignores the rest — the hard lookup that soft attention approximates.

Step 3 — check $M \to - \infty$ . Then $α_{1} \to 0$ , $α_{2} \to 1$ (the mirror case). Why this step? Covers both ends of the limit so no scenario is left unshown.

Verify: at $M = 20$ , $α_{1} = 1/ (1 + e^{- 20}) = 0.99999999794 \approx 1$ ✓, confirming the limit.

Example 7 — full $n \times n$ attention matrix (cell H)

Step 1 — raw $Q K^{⊤}$ . Entry $(i, j) = q_{i} \cdot k_{j}$ : $Q K^{⊤} = 101011112$ Why this step? One matrix multiply gives all pairwise relevances at once — the parallelism attention is famous for.

Step 2 — scale by $2$ . Divide every entry by $1.4142$ : row 3 becomes $(0.7071, 0.7071, 1.4142)$ . Why this step? Same variance control as before, applied uniformly.

Step 3 — softmax row 3 (the interesting one). $e^{0.7071} = 2.0281$ (twice), $e^{1.4142} = 4.1133$ , sum $= 8.1695$ . $α_{3 \cdot} = (0.2483, 0.2483, 0.5035)$ . Why this step? Each row must sum to 1 because it's token 3's own distribution over who to read.

Step 4 — output for token 3. $z_{3} = 0.2483 (1, 0) + 0.2483 (0, 1) + 0.5035 (1, 1) = (0.7518, 0.7518)$ . Why this step? Token 3, pointing diagonally, sensibly blends toward the diagonal value.

Verify: row-3 weights sum $0.2483 + 0.2483 + 0.5035 = 1.000$ ✓. Rows 1 and 2 (scores $(0.7071, 0, 0.7071)$ ) are more spread than row 3, which self-peaks at $2/ 2$ — matching the forecast about spread.

Example 8 — aerospace telemetry across time (cell I, word problem)

Step 1 — exponentiate scaled scores. $e^{3} = 20.086$ , $e^{0} = 1$ , $e^{- 1} = 0.3679$ . Sum $= 21.454$ . Why this step? Softmax over the three source timesteps.

Step 2 — weights. $α_{120} = 0.9363, α_{60} = 0.04661, α_{10} = 0.01715$ . Why this step? The most-relevant past step dominates — but directly, with no RNN blurring over 60 steps.

Step 3 — pooled evidence. $z_{180} = 0.9363 (1) + 0.04661 (0) + 0.01715 (0) = 0.9363$ . Why this step? Token $t = 180$ pulls in $\approx 0.94$ units of anomaly evidence from the pressure drop.

Verify: weights sum $= 1.000$ ✓; pressure-drop share $= 0.936$ (≈94%) ✓ — the model correctly "remembers" a cause 60 steps back. Contrast RNNs and LSTMs, where that signal would fade.

Example 9 — the permutation twist (cell J, exam trap)

Step 1 — see what a swap does to $Q K^{⊤}$ . Reordering rows/columns of the score matrix by permutation $P$ gives $P (Q K^{⊤}) P^{⊤}$ ; softmax and the value-blend follow the same $P$ . Why this step? Every operation (dot, softmax row-wise, sum) treats positions symmetrically — none looks at index order.

Step 2 — conclude. New outputs $= (z_{C}, z_{A}, z_{B})$ : the same three vectors, permuted identically to the input. This is permutation-equivariance. Why this step? It exposes that attention has no built-in order — the trap the exam is testing.

Step 3 — the fix. Add Positional Encoding to each input so identical tokens at different positions get different vectors, breaking the symmetry. Why this step? Without it, "engine spun up then failed" and "failed then spun up" look identical.

Verify: conceptual — swapping inputs by $P$ yields outputs by the same $P$ , values unchanged. No numeric claim, but confirmed by the structure in Ex 7 (the score matrix carries no index information beyond the vectors themselves).

Active Recall

Do negative scores ever give negative attention weights?

No —

e^{x} > 0

always, so softmax weights are strictly positive; only differences between scores matter.

What does attention degenerate into when all scores are equal?

A plain uniform average of the values (

α_{j} = 1/ n

), because softmax is invariant to adding a constant to every score.

As one scaled score

M \to \infty

with others fixed, what happens to softmax?

It collapses to one-hot — that key gets weight

\to 1

, all others

\to 0

(hard lookup).

Swapping input order with no positional encoding does what to outputs?

Outputs permute identically (permutation-equivariance); the actual vectors are unchanged.

Why must softmax be applied row-wise on

Q K^{⊤}

Each row is one query's distribution over all keys, so the weights over which tokens to read must sum to 1.