Scaled dot-product attention: Worked examples

This is the "roll up your sleeves" child of Scaled dot-product attention. The parent told you what the formula is and why we scale. Here we throw every kind of input at it — matched keys, unmatched keys, ties, negative scores, a degenerate all-equal case, the large- $d_{k}$ saturation case, a real-world sentence, and an exam twist — and grind each one to a number.

Before we compute anything, one promise: every symbol is re-earned here, so you can read this page cold.

The scenario matrix

Every worked example below is tagged with the cell of this table it exercises. Together they touch all of them.

Cell	What makes it special	Where the danger is
A. Perfect match vs no match	one key aligns, one is orthogonal	baseline sanity
B. Tie (all scores equal)	softmax must return uniform weights	division-by-nothing intuition
C. Negative scores	query dislikes a key ( $q \cdot k < 0$ )	sign handling in $e^{z}$
D. Degenerate values	all values identical	output must ignore weights
E. Large $d_{k}$ saturation	unscaled vs scaled logits	vanishing softmax gradient
F. Many keys, one dominates	4 keys, softmax near one-hot	limiting behaviour
G. Real-world word problem	a 3-word sentence, self-attention	interpret the blend
H. Exam twist	masked key (position we must forbid)	$- \infty$ logit trick

The worked examples

Cell A — perfect match vs orthogonal key

Worked example A · one query, two keys,

$d_{k} = 4$

K=\begin{bmatrix}1&0&1&0\\[2pt]0&1&0&1\end{bmatrix},\quad V=\begin{bmatrix}5&0&0\\[2pt]0&3&0\end{bmatrix}$$ **Forecast:** the query row *is* key 1. Guess: output should be mostly value 1, i.e. close to $[5,0,0]$ but not exactly. **Step 1 — score against each key.** $QK^\top=[\,1\cdot1+0+1\cdot1+0,\;\;0+0+0+0\,]=[2,\,0]$. *Why this step?* The dot product is our relevance meter; $2$ vs $0$ says "loves key 1, neutral on key 2." **Step 2 — scale by $\sqrt{d_k}=\sqrt4=2$.** $[2,0]/2=[1,0]$. *Why this step?* Keeps logits order-1 so softmax stays sensitive (see Cell E for what goes wrong otherwise). **Step 3 — softmax.** $A=\left[\tfrac{e^1}{e^1+e^0},\tfrac{e^0}{e^1+e^0}\right]\approx[0.731,\,0.269]$. *Why this step?* Turns raw scores into a distribution — "listen 73\% to key 1, 27\% to key 2." **Step 4 — blend values.** $0.731[5,0,0]+0.269[0,3,0]=[3.655,\,0.807,\,0]$. **Verify:** weights sum to $1$ ✓; output leans hard toward $[5,0,0]$ as forecast ✓.

Cell B — the tie (all scores equal)

Cell C — negative scores (the query dislikes a key)

Worked example C · one attractive, one repulsive key

$Q = [10], K = [1 - 1 00], V = [4004], d_{k} = 2$

Forecast: key 2 points opposite the query, so its score is negative. It should get a small weight, but never a negative one (softmax can't).

Step 1 — scores. $Q K^{⊤} = [1, - 1]$ . Why this step? $q \cdot k_{2} = 1 \cdot (- 1) + 0 = - 1$ : opposite direction → negative relevance.

Step 2 — scale. $[1, - 1] / 2 \approx [0.707, - 0.707]$ .

Step 3 — softmax handles the sign. $e^{- 0.707}$ is small but positive, so $A = [\frac{e ^{0.707}}{e ^{0.707} + e ^{- 0.707}}, \frac{e ^{- 0.707}}{e ^{0.707} + e ^{- 0.707}}] \approx [0.803, 0.197] .$ Why this step? Exponentials are always positive, so a negative logit becomes a small share, not a negative one. Softmax squashes everything into $[0, 1]$ .

Step 4 — blend. $0.803 [4, 0] + 0.197 [0, 4] = [3.211, 0.789]$ .

Verify: disliked key still contributes a small positive amount; weights $\geq 0$ and sum to $1$ ✓.

Cell D — degenerate values (weights become irrelevant)

Cell E — large $d_{k}$ : why scaling saves the gradient

This is the quantitative heart of the parent's "volume knob" story. Learning is driven by gradient descent, which needs a non-tiny gradient to make progress.

Worked example E ·

$d_{k} = 64$ , one query, two keys, raw score $[8, 7.5]$ Forecast: unscaled, the logits $[8, 7.5]$ push softmax toward one-hot and shrink the learning signal. Scaled by $64 = 8$ , they become $[1, 0.9375]$ — near the sensitive middle.

Step 1 — unscaled softmax. $softmax ([8, 7.5]) = [\frac{e ^{8}}{e ^{8} + e ^{7.5}}, \frac{e ^{7.5}}{e ^{8} + e ^{7.5}}] \approx [0.6225, 0.3775] .$ So $σ \approx 0.6225$ and local gradient magnitude $σ (1 - σ) \approx 0.6225 \times 0.3775 = 0.2350$ .

Step 2 — scale by $d_{k} = 8$ . $[8, 7.5] /8 = [1, 0.9375]$ . Why this step? We proved (parent Step 3) that dot-product variance grows as $d_{k}$ ; dividing by $d_{k}$ pulls std back to $1$ .

Step 3 — scaled softmax. $softmax ([1, 0.9375]) \approx [0.5156, 0.4844], σ (1 - σ) \approx 0.2498.$

Verify: $0.2498 > 0.2350$ — the scaled version has the larger, healthier gradient, so it learns the subtle key preference faster ✓. (See figure: the scaled point sits nearer the peak of the $σ (1 - σ)$ curve.)

Cell F — four keys, one dominant (limiting toward one-hot)

Worked example F · softmax approaching a hard selection

Scaled logits (already divided by $d_{k}$ ): $z = [3, 1, 1, 0]$ , with values $V = [[1], [2], [3], [4]]^{⊤}$ i.e. scalar contents $1, 2, 3, 4$ .

Forecast: logit $3$ is far above the rest, so weight 1 dominates → output near value $1$ .

Step 1 — exponentiate each logit. $e^{3} \approx 20.0855, e^{1} \approx 2.7183, e^{1} \approx 2.7183, e^{0} = 1$ . Their sum is $\approx 26.5220$ . Why this step? Softmax needs positive, order-preserving amounts before it can normalise. Exponentiating turns any real logit (even a huge or negative one) into a positive "raw share," and it stretches gaps so that a larger logit grabs disproportionately more mass — that is exactly what produces the near-one-hot behaviour we're studying.

Step 2 — normalise (divide each by the sum). $A \approx [\frac{20.0855}{26.5220}, \frac{2.7183}{26.5220}, \frac{2.7183}{26.5220}, \frac{1}{26.5220}] \approx [0.7573, 0.1025, 0.1025, 0.0377] .$ Why this step? Dividing by the total forces the four shares to sum to exactly $1$ , so they become genuine probabilities. (These now sum to $1.0000$ — the earlier draft's rounding made them fall short.)

Step 3 — blend the scalar values $[1, 2, 3, 4]$ . $0.7573 (1) + 0.1025 (2) + 0.1025 (3) + 0.0377 (4) = 1.4260.$ Why this step? The output is the attention-weighted average of the contents; with mass piled on key 1, the answer sits just above value $1$ .

Verify: weights sum to $1$ ✓; output $\approx 1.426$ sits just above value $1$ , exactly where a near-one-hot weight on key 1 puts it ✓. Push logit 1 to $\infty$ and the output would tend to $1$ (pure one-hot).

Cell G — real-world word problem (self-attention on 3 words)

Here $Q, K, V$ come from the same sentence — this is self-attention. Each word produces its own query, key and value (via learned matrices, part of the Q-K-V model). We follow the query of the word "it" in "The cat slept" … pretend "it" refers back.

Worked example G · which word does "it" attend to?

Query for "it": $q = [1, 1]$ . Keys ( $d_{k} = 2$ ): "The" $= [0, 0]$ , "cat" $= [1, 1]$ , "slept" $= [1, - 1]$ . Values (1-D content scores): "The" $= 0$ , "cat" $= 10$ , "slept" $= 4$ .

Forecast: "it" should align with "cat" (same key direction) and pull mostly its value.

Step 1 — scores. $q \cdot The = 0, q \cdot cat = 2, q \cdot slept = 0$ . Why this step? "cat" shares the query's direction → highest relevance; "slept" is orthogonal.

Step 2 — scale by $2$ . $[0, 1.414, 0]$ .

Step 3 — softmax. $e^{0} = 1, e^{1.414} \approx 4.113, e^{0} = 1$ ; sum $\approx 6.113$ . $A \approx [0.1636, 0.6728, 0.1636]$ .

Step 4 — blend values $[0, 10, 4]$ . $0.1636 (0) + 0.6728 (10) + 0.1636 (4) = 7.383$ .

Verify: most weight ( $0.67$ ) on "cat" ✓; blended content $7.383$ leans strongly toward cat's $10$ , nudged down by the weak neighbours — the coreference "it → cat" is captured ✓.

Cell H — exam twist: a masked (forbidden) key

In decoders a query must not peek at future tokens. We forbid a key by adding $- \infty$ to its logit before softmax — this is an attention mask. $e^{- \infty} = 0$ , so that key gets exactly zero weight.

Worked example H · block key 2 with a mask

Scaled logits $z = [1, 0.5, 2]$ but key 2 is masked (future position). Values (scalar): $[3, 9, 6]$ .

Forecast: whatever key 2 wanted, its weight becomes $0$ ; the remaining mass redistributes over keys 1 and 3.

Step 1 — apply mask. $z \to [1, - \infty, 2]$ . Why this step? Adding $- \infty$ is the clean algebraic way to delete a key; keeps the operation differentiable everywhere the mask is $0$ .

Step 2 — exponentiate. $e^{1} \approx 2.718, e^{- \infty} = 0, e^{2} \approx 7.389$ . Sum $\approx 10.107$ .

Step 3 — softmax. $A \approx [0.2689, 0, 0.7311]$ . Why this step? Masked key contributes exactly $0$ ; the other two still sum to $1$ .

Step 4 — blend values $[3, 9, 6]$ . $0.2689 (3) + 0 (9) + 0.7311 (6) = 5.193$ .

Verify: masked weight is exactly $0$ ✓; unmasked weights sum to $1$ ✓; the "9" value (forbidden) never enters the answer ✓. This is how a Transformer keeps the future secret.

Recall Self-check

Softmax of two equal scaled logits gives what weights? ::: Exactly $[0.5, 0.5]$ — magnitude is irrelevant, only differences matter. A key with logit $- \infty$ contributes what weight after softmax? ::: Zero, because $e^{- \infty} = 0$ (the masking trick). If all value rows are identical, what is the attention output? ::: That same value row, regardless of the weights. Why does scaling by $d_{k}$ help learning and not just prevent overflow? ::: It keeps logit differences small so softmax stays near $σ \approx 0.5$ , where the gradient $σ (1 - σ)$ is largest.

Related depth: Multi-Head Attention, Cross-Attention, Positional Encoding.