Self-attention mechanism in detail: Worked examples

This page is a workout gym for the parent note. There we built the formula $SelfAttention (X) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V .$ Here we run it through every kind of input it can meet — big scores, tiny scores, ties, a single word, a degenerate all-zero query, a masked (causal) case, and a real sentence. If you have not yet seen why $Q$ , $K$ , $V$ exist, or why we divide by $d_{k}$ , read the parent first; this page assumes those pieces are already earned.

Two symbols we reuse constantly, restated in plain words:

A dot product $a \cdot b = \sum_{d} a_{d} b_{d}$ is one number that says "how much do these two arrows point the same way?" Big positive = aligned, $0$ = perpendicular (unrelated), negative = opposite. This is the similarity score of attention.
Softmax turns a list of numbers into a list of positive weights that sum to $1$ : $softmax (s)_{j} = \frac{e ^{s_{j}}}{\sum _{k} e ^{s_{k}}}$ . Think of it as "vote shares" — every position gets a slice of a pie that always totals $100%$ .

The scenario matrix

Every input a self-attention layer meets falls into one of these case classes. Below, each example is tagged with the cell(s) it covers.

#	Case class	What is special	Covered by
C1	Ordinary spread	Scores differ moderately → soft, mixed weights	Ex 1
C2	Tie / symmetry	Two keys equally relevant → equal split	Ex 2
C3	Large scores (peaky)	Big $d_{k}$ or big values → near one-hot; shows why we scale	Ex 3
C4	Degenerate query = 0	Query vector is all zeros → all scores equal → uniform average	Ex 4
C5	Single token, $n = 1$	Limiting smallest sequence → attends to itself, weight $= 1$	Ex 5
C6	Negative alignment	A key points opposite the query → down-weighted, never negative	Ex 6
C7	Causal mask	Future positions forbidden ( $- \infty$ ) → decoder-style attention	Ex 7
C8	Real word problem	A full sentence, semantic reasoning	Ex 8
C9	Exam twist	"What must $d_{k}$ be?" shape-inference puzzle	Ex 9

We visit all nine cells. Numbers in every worked example are checked in the appendix.

The picture above is the whole machine in one glance: a query arrow (red) is compared to each key arrow by dot product, those numbers go through softmax to become a pie of weights, and the weights mix the value vectors into one output. Every example below is just this pipeline with different arrows.

Example 1 — Ordinary spread (C1)

Forecast: Guess before computing — will "cat" attend mostly to itself, or spread out? Write down a guess.

Scores $s_{0 j} = q_{0} \cdot k_{j}$ : with $k_{0} = [1, 0], k_{1} = [0, 1], k_{2} = [1, 1]$ we get $s_{0 \cdot} = [1, 0, 1]$ . Why this step? The dot product is our similarity meter. $q_{0}$ aligns fully with $k_{0}$ and $k_{2}$ (both have a $1$ in the first slot), and is perpendicular to $k_{1}$ .
Scale by $d_{k} = 2 \approx 1.414$ : $s^{'} = [0.707, 0, 0.707]$ . Why this step? Without scaling, larger $d_{k}$ inflates dot products and softmax becomes razor-sharp; $d_{k}$ keeps variance near $1$ (proved in parent).
Softmax: $e^{0.707} = 2.028$ , $e^{0} = 1$ , $e^{0.707} = 2.028$ ; sum $= 5.056$ . $α_{0 \cdot} = [0.401, 0.198, 0.401] .$ Why this step? Turns raw alignment into a pie of weights totaling $1$ so we can average.
Aggregate: $z_{0} = 0.401 [2, 0] + 0.198 [0, 2] + 0.401 [2, 1]$ $= [0.802 + 0.802, 0.396 + 0.401] = [1.604, 0.797] .$ Why this step? The output is a weighted blend of values, pulling most from positions "cat" and "mat".

Recall Answer

Output for "cat" $\approx [1.60, 0.80]$ . It attended $40%$ / $20%$ / $40%$ — a genuine spread, exactly the "ordinary" case.

Verify: Weights sum to $0.401 + 0.198 + 0.401 = 1.000$ ✓. Output lies inside the convex hull of the rows of $V$ (each coordinate between the min and max of that column of $V$ ) — a weighted average can never escape its inputs. ✓

Example 2 — Tie / symmetry (C2)

Forecast: By symmetry, guess the split before reading on.

Scores: $s_{0} = q \cdot k_{0} = 1$ , $s_{1} = q \cdot k_{1} = 1$ . Identical. Why this step? The query leans equally toward both axes, so both keys score the same.
Scale & softmax: identical inputs → $α = [0.5, 0.5]$ regardless of scaling. Why this step? Softmax of equal numbers is always uniform; scaling a constant leaves it constant.

Recall Answer

A perfect $[0.5, 0.5]$ split.

Verify: For any two equal scores $s$ , $\frac{e ^{s}}{e ^{s} + e ^{s}} = \frac{1}{2}$ . ✓ This is the tie cell — ties always split evenly, no matter the magnitude.

Example 3 — Large scores show why we scale (C3)

Forecast: Which version gives sharper, more one-hot weights?

Unscaled softmax of $[8, 2]$ : $e^{8} = 2981, e^{2} = 7.39$ , so $α \approx [0.9975, 0.0025]$ . Why this step? Big raw scores make softmax nearly one-hot — almost all weight on one key.
Scaled by $64 = 8$ : inputs become $[1.0, 0.25]$ . $e^{1} = 2.718, e^{0.25} = 1.284$ , sum $= 4.002$ , $α \approx [0.679, 0.321]$ . Why this step? Scaling pulls scores back to a healthy range, so weights stay soft and gradients survive (near-one-hot rows have almost-zero gradient — see gradient flow).

Recall Answer

Unscaled: $[0.9975, 0.0025]$ (peaky, gradient-starved). Scaled: $[0.679, 0.321]$ (healthy). Same information, very different learnability.

Verify: $e^{8} / (e^{8} + e^{2}) = 0.99753$ and $e^{1} / (e^{1} + e^{0.25}) = 0.67917$ . ✓ The scaling is not cosmetic — it is the difference between a trainable and a frozen layer.

Example 4 — Degenerate query $= 0$ (C4)

Forecast: A zero query has no direction to prefer. Guess the weights.

Scores: $q \cdot k_{j} = 0$ for every $j$ (zero dotted with anything is $0$ ). Why this step? This is the degenerate boundary — the query expresses no preference at all.
Softmax of $[0, 0, 0]$ $= [\frac{1}{3}, \frac{1}{3}, \frac{1}{3}]$ . Why this step? Equal scores → uniform weights; the output becomes the plain average of all values.

Recall Answer

Uniform $[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}]$ — self-attention gracefully falls back to "average everyone" when the query says nothing. No crash, no divide-by-zero.

Verify: $softmax (0, 0, 0) = (1/3, 1/3, 1/3)$ and $3 \times \frac{1}{3} = 1$ . ✓ This proves the layer is defined even on degenerate input.

Example 5 — Single token, $n = 1$ (C5)

Forecast: With only itself to look at, where can attention go?

Scores: a single number $s = q \cdot k$ . Why this step? There is exactly one key, so the score list has length $1$ .
Softmax of a single value is always $1$ : $α = [1]$ . Why this step? $\frac{e ^{s}}{e ^{s}} = 1$ — the pie has one slice, so it takes the whole pie regardless of $s$ .
Aggregate: $z = 1 \cdot v = [7, - 3]$ . Why this step? The output equals the token's own value — self-attention on a lone token is the identity on $V$ .

Recall Answer

Output $= v = [7, - 3]$ . The smallest sequence is a fixed point: a single token passes its value straight through.

Verify: $e^{s} / e^{s} = 1$ for any $s$ ; output $= v$ . ✓ Limiting-case sanity confirmed.

Example 6 — Negative alignment (C6)

Forecast: Can a weight ever be negative? Guess yes or no.

Scores: $s_{0} = q \cdot k_{0} = 1$ , $s_{1} = q \cdot k_{1} = - 1$ . Why this step? Opposite arrows give a negative dot product — the meter reads "anti-aligned".
Softmax: $e^{1} = 2.718$ , $e^{- 1} = 0.368$ , sum $= 3.086$ , so $α = [0.881, 0.119]$ . Why this step? The exponential of a negative number is a small positive number — softmax can shrink a weight toward zero but can never make it negative.

Recall Answer

$[0.881, 0.119]$ . The opposing key is heavily down-weighted yet keeps a small positive share — attention is a soft gate, not a switch.

Verify: $e^{1} / (e^{1} + e^{- 1}) = 0.88080$ and both weights $> 0$ . ✓ Weights are always in $(0, 1)$ ; this is why softmax and not a raw signed normalization.

Example 7 — Causal mask (C7)

Forecast: What happens to the weight on the (forbidden) future position $2$ ?

Mask the future: set $s_{12} = - \infty$ , giving $s_{1 \cdot} = [2, 3, - \infty]$ . Why this step? The mask enforces that a token predicts using only past+present — this is how the decoder stays causal.
Softmax: $e^{- \infty} = 0$ , so position $2$ gets weight $0$ . On the survivors $[2, 3]$ : $e^{2} = 7.389, e^{3} = 20.086$ , sum $= 27.475$ , $α = [0.269, 0.731, 0]$ . Why this step? Setting the score to $- \infty$ before softmax is the clean way to drop a position — it exits the pie entirely, and the remaining weights still sum to $1$ .

Recall Answer

$[0.269, 0.731, 0]$ . The forbidden future is exactly zero and the past re-normalizes among itself.

Verify: $e^{2} / (e^{2} + e^{3}) = 0.26894$ , $e^{3} / (e^{2} + e^{3}) = 0.73106$ , sum $= 1$ , future weight $= 0$ . ✓

Example 8 — Real word problem (C8)

Forecast: Which noun wins — trophy or suitcase?

Softmax of $[4.0, 1.0, 0.5]$ : $e^{4} = 54.60$ , $e^{1} = 2.718$ , $e^{0.5} = 1.649$ , sum $= 58.97$ . $α = [0.926, 0.0461, 0.0280] .$ Why this step? The learned projections have made the query for "it" align with the subject noun; softmax then amplifies that lead.
Read the winner: "trophy" holds $92.6%$ of the weight. Why this step? Coreference resolution is the argmax of the attention row — the output for "it" becomes almost entirely the value vector of "trophy".

Recall Answer

"it" resolves to trophy with $\approx 93%$ attention. Swap "big" → "small" and the trained scores would flip to favor suitcase — the meaning drives the weights.

Verify: $e^{4} / (e^{4} + e^{1} + e^{0.5}) = 0.92594$ . ✓ The largest score dominates but never fully — the smaller words keep tiny positive shares (soft attention).

Example 9 — Exam twist: shape inference (C9)

Forecast: Guess $d_{k}$ before computing.

Split rule: $d_{k} = d_{m o d e l} / h = 512/8 = 64$ . Why this step? Multi-head attention divides the embedding across heads so total compute matches a single full-size head.
Score shape: $Q \in R^{10 \times 64}$ , $K^{⊤} \in R^{64 \times 10}$ , so $Q K^{⊤} \in R^{10 \times 10}$ . Why this step? Inner dimension $64$ cancels; the surviving $n \times n$ is every-query-vs-every-key.

Recall Answer

$d_{k} = 64$ ; score matrix is $10 \times 10$ . The $n^{2} = 100$ entries are exactly the $O(n^2)$ cost of attention.

Verify: $512/8 = 64$ and $(10 \times 64) \cdot (64 \times 10) \to 10 \times 10$ . ✓

Recall drills

What weights does softmax give when all input scores are equal (say three zeros)?

Uniform

[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}]

— no preference, plain average.

Can an attention weight ever be negative?

No — softmax outputs

e^{s} > 0

always, so weights live in

(0, 1)

Why divide raw scores by

d_{k}

before softmax?

Dot products grow like

d_{k}

; scaling keeps variance

\approx 1

so softmax stays soft and gradients flow.

How does a causal mask remove a future position?

Set its score to

- \infty

before softmax, so

e^{- \infty} = 0

and it drops out of the pie.

For a single token (

n = 1

), what is the output?

Its own value

v

, since softmax of one number is

1