Worked examples — The attention mechanism intuition

You have met the three-line recipe on the parent note: score every encoder state, squash the scores with softmax, then blend. This page drills that recipe until every possible shape of input feels routine — the sharp cases, the flat cases, the degenerate one-word case, the negative-score case, and the exam trap where someone hands you scores that are already probabilities and dares you to softmax them twice.

Before we compute anything, let us name the pieces one more time in plain words, because we will use these symbols in every single example.

The only new machine we lean on is the softmax. Here is why it is the right tool and not, say, "just divide each score by the total."

One trap to flag now, so you recognise it when it arrives: softmax is for turning scores into weights, never for re-processing weights that are already normalised. If someone hands you three numbers that already add to $1$, running softmax again would flatten them toward uniform and destroy the focus. We show exactly how to handle that in Example 9.

---

The scenario matrix

Every attention computation you will ever be asked to do falls into one of these cells. Here $T_x$ is the input sequence length defined just above, and $\beta$ (introduced in Example 8) is a scaling factor we multiply scores by — it is not a sequence length, it is a "sharpness dial". The examples below are labelled with the cell(s) they cover, and together they hit all of them.

#	Case class	What is unusual about it	Example
A	One sharp peak	one score clearly biggest → focused $\alpha$	Ex 1
B	All scores equal	degenerate → uniform $\alpha =1/T_x$	Ex 2
C	Negative scores present	naive "divide by sum" breaks; softmax survives	Ex 3
D	Shifted scores ($+k$ to all)	should NOT change $\alpha$ — invariance check	Ex 4
E	Single input word ($T_x=1$)	degenerate → $\alpha =1$ always, context $=h_1$	Ex 5
F	Two near-tied peaks	soft blend of two boxes, entropy is high	Ex 6
G	Word-problem / reordering	real translation, non-monotonic alignment	Ex 7
H	Limiting behaviour (scores $\times \beta$, $\beta \to \infty$)	attention becomes a hard argmax	Ex 8
I	Exam twist: already-normalised input	do NOT softmax twice; compute entropy	Ex 9

---

Example 1 — Cell A: one sharp peak (the French "am ↔ suis" case)

Forecast: "suis" should win big — guess its weight before reading on. Above $0.6$? Above $0.7$?

Step 1 — exponentiate every score. $\exp (0.3)=1.350$, $\exp (2.1)=8.166$, $\exp (0.5)=1.649$. Why this step? Softmax needs positive quantities before it can form fractions; $\exp$ delivers that and blows up the gap so the biggest score dominates.

Step 2 — add them for the denominator. $Z=1.350+8.166+1.649=11.165$. Why this step? Dividing by this total is exactly what forces the weights to sum to $1$ — it is the "share of the whole" step.

Step 3 — divide. ${\alpha }_{21}=\frac{1.350}{11.165}=0.121,{\alpha }_{22}=\frac{8.166}{11.165}=0.731,{\alpha }_{23}=\frac{1.649}{11.165}=\mathrm{0.148.}$ Why this step? These are the fractions of focus.

Figure — what to see: the three coloured bars are the weights ${\alpha }_{2i}$. The coral bar for "suis" (height $0.731$) towers over the lavender "Je" and mint "étudiant" bars (each near $0.1$). Notice the visual of amplification: the raw score gap was only $2.1$ vs $0.5$, yet after $\exp$ the winning bar is roughly five times taller — that height ratio is exactly $\exp (2.1-0.5)=\exp (1.6)\approx 4.95$. The arrow marks the $73\%$ of focus poured onto "suis" the moment the decoder emits "am".

Verify: $0.121+0.731+0.148=1.000$ ✓ (sums to one, as any probability distribution must). "suis" carries $73\%$ — correct, since "am" translates "suis".

---

Example 2 — Cell B: all scores equal → uniform attention

Forecast: If the model can't tell the boxes apart, what should it do? Guess the weight of each.

Step 1 — notice the scores are equal. Why this step? When inputs to softmax are identical, symmetry forces identical outputs; no arithmetic needed yet, but let's confirm.

Step 2 — exponentiate and divide. Each $\exp (1.7)=5.4739$; sum $=4\times 5.4739=21.8956$. ${\alpha }_{ti}=\frac{5.4739}{21.8956}=0.25\text{for every }i.$ Why this step? Confirms the symmetry: equal scores give the uniform distribution $1/T_x=1/4$.

Verify: $4\times 0.25=1.000$ ✓. This is the "no focus" degenerate case — softmax does not invent focus out of nothing (Mistake 3 on the parent note). The context becomes a plain average $c_t=\frac{1}{4}(h_1+h_2+h_3+h_4)$.

---

Example 3 — Cell C: negative scores present

Forecast: The naive division uses $\sum e=-1.0$ — a negative denominator. Predict what goes wrong, then predict the real winner (position 2).

Step 1 — show why naive division fails. Naive: ${\alpha }_1=\frac{-2}{-1}=2.0$. A "weight" of $2.0$ means $200\%$ focus — impossible, and it is negative-denominator nonsense. Why this step? This is exactly the failure softmax was built to avoid; seeing it cements why we exponentiate.

Step 2 — exponentiate (kills the sign problem). $\exp (-2.0)=0.1353$, $\exp (1.0)=2.7183$, $\exp (0.0)=1.0000$. All positive. Why this step? $\exp$ maps every real number, negative included, to a positive one — the whole point.

Step 3 — normalise. $Z=0.1353+2.7183+1.0000=3.8536$. ${\alpha }_{t1}=\frac{0.1353}{3.8536}=0.0351,{\alpha }_{t2}=\frac{2.7183}{3.8536}=0.7054,{\alpha }_{t3}=\frac{1.0000}{3.8536}=\mathrm{0.2595.}$ Why this step? Dividing each exponentiated score by their total $Z$ is the softmax normalisation: it turns the positive numbers from Step 2 into fractions of focus that add to $1$. This is the same "share of the whole" move as Example 1, Step 2 — the only reason it deserves its own step here is to show the machinery survives negative inputs untouched.

Verify: all in $[0,1]$, and $0.0351+0.7054+0.2595=1.000$ ✓. Position 2 wins with $\approx 70.5\%$ — softmax handled the negative score gracefully.

---

Example 4 — Cell D: shift-invariance (add $k$ to every score)

Forecast: Will the weights change, shrink, or stay exactly the same? Commit to an answer.

Step 1 — write the shifted softmax and factor out the constant. ${\alpha }_i=\frac{\exp (e_i+k)}{{\sum }_j\exp (e_j+k)}=\frac{\exp (k)\exp (e_i)}{\exp (k){\sum }_j\exp (e_j)}=\frac{\exp (e_i)}{{\sum }_j\exp (e_j)}.$ Why this step? The $\exp (k)$ cancels top and bottom — algebraic proof that only score differences matter.

Step 2 — check numerically. $\exp (3)=20.086$, $\exp (6)=403.429$, $\exp (5)=148.413$; $Z=571.928$. ${\alpha }_1=20.086/571.928=0.0351$, ${\alpha }_2=403.429/571.928=0.7054$, ${\alpha }_3=148.413/571.928=0.2595$. Why this step? Confirms the algebra matches Example 3 to the digit.

Verify: identical to Ex 3 (0.0351, 0.7054, 0.2595) ✓. This is the fact real libraries use to subtract ${\max }_j{e}_{tj}$ before exponentiating — it prevents $\exp$ from overflowing without changing the answer.

---

Example 5 — Cell E: a single input word ($T_x=1$)

Forecast: With only one box, how much focus can it possibly get? Answer before computing.

Step 1 — apply softmax over a one-element list. ${\alpha }_{t1}=\frac{\exp (4.2)}{\exp (4.2)}=1.$ Why this step? The denominator is the sum over the only term, so it equals the numerator — the score value cancels entirely.

Step 2 — build the context. $c_t={\alpha }_{t1}{h}_1=1\cdot h_1=h_1$. Why this step? With no choice to make, attention degenerates to "just use the one state you have" — it can never do worse than vanilla seq2seq here.

Verify: ${\alpha }_{t1}=1.000$ regardless of the score ✓. Edge case handled: attention on a length-1 input is a no-op that returns $h_1$.

---

Example 6 — Cell F: two near-tied peaks (diffuse, high-entropy)

Forecast: Will the focus split roughly 50/50 between positions 2 and 3? Will entropy be higher or lower than the sharp Example 1?

Step 1 — softmax. $\exp (0.2)=1.2214$, $\exp (2.5)=12.1825$, $\exp (2.4)=11.0232$, $\exp (0.1)=1.1052$; $Z=25.5323$. $\alpha =(0.0478,0.4771,0.4317,0.0433).$ Why this step? Standard recipe; note positions 2 and 3 nearly split the mass.

Step 2 — sanity on the split. ${\alpha }_2+{\alpha }_3=0.9088$ — over $90\%$ shared between the two tied boxes. Why this step? Confirms this is a "blend two sources" situation, not a single-winner one.

Step 3 — compute entropy. $H=-(0.0478\ln 0.0478+0.4771\ln 0.4771+0.4317\ln 0.4317+0.0433\ln 0.0433)$. Term by term: $0.0478\cdot (-3.041)=-0.1454$, $0.4771\cdot (-0.7401)=-0.3531$, $0.4317\cdot (-0.8401)=-0.3627$, $0.0433\cdot (-3.139)=-0.1359$. $H=0.1454+0.3531+0.3627+0.1359=0.9971$ nats. Why this step? Entropy measures how spread-out the focus is: bigger = more diffuse.

Verify: weights sum $0.0478+0.4771+0.4317+0.0433=0.9999\approx 1$ ✓. $H\approx 0.997$ nats here versus a much lower entropy for Example 1's sharp peak — confirming "two tied peaks" is genuinely more diffuse. (Uniform-4 would be the maximum, $\ln 4=1.386$.)

---

Example 7 — Cell G: word problem with reordering (English → French)

Forecast: French output order is 3-2-1 relative to English. So the first output word should attend to English word... which one? Guess before Step 3.

Step 1 — softmax the three scores. $\exp (0.1)=1.1052$, $\exp (0.4)=1.4918$, $\exp (3.0)=20.0855$; $Z=22.6825$. Why this step? Turn relevance scores into focus fractions.

Step 2 — divide. $\alpha =(0.0487,0.0658,0.8855).$ Why this step? Position 3 ("translation") gets $\approx 88.6\%$ — the model reversed the order, exactly what French word-order demands.

Step 3 — read the alignment as a heatmap cell. Output word 1 → input word 3. On an attention heatmap this is an off-diagonal dot (top-right).

Figure — what to see: rows are the three French output words (top to bottom, in the order they are generated), columns are the three English input words. Colour is the attention weight — darker means more focus. The dark cells do not run down the main diagonal (top-left to bottom-right); instead they run down the anti-diagonal (top-right to bottom-left), the coral arrow pointing at the darkest cell where "traduction" (row 1) locks onto "translation" (column 3). That anti-diagonal streak is the visual fingerprint of reordering: had the languages shared word order, the dark cells would have formed the main diagonal instead.

Verify: $0.0487+0.0658+0.8855=1.000$ ✓. Dominant weight $0.8855$ on "translation" ✓ — the non-monotonic (reversed) alignment the parent note predicted for English↔French.

---

Example 8 — Cell H: limiting behaviour, scores scaled by a sharpness dial $\beta$

Forecast: As we exaggerate the gap between scores, does attention get sharper or flatter? What is the extreme limit?

Step 1 — $\beta =1$: scores $(0,1)$. $\alpha =\left(\frac{1}{1+e},\frac{e}{1+e}\right)=(0.2689,0.7311)$. Why this step? This is our baseline — the raw gap of $1$ between the two scores, un-amplified. Everything after this shows what turning the dial up does to that same gap.

Step 2 — $\beta =5$: scores $(0,5)$. $\alpha =\left(\frac{1}{1+e^5},\frac{e^5}{1+e^5}\right)=(0.0067,0.9933)$. Why this step? Multiplying by $\beta =5$ stretches the gap from $1$ to $5$; since softmax amplifies gaps through $\exp$, the winning weight jumps from $0.73$ to $0.99$ — we are watching focus sharpen as the dial turns.

Step 3 — $\beta =20$: scores $(0,20)$. $\alpha =(2.06\times 10^{-9},\approx 1)\to (0,1)$. Why this step? Pushing $\beta$ very large makes the gap enormous, and in the limit $\beta \to \infty$ softmax becomes a hard argmax — it collapses to weight $1$ on the single largest score and $0$ on everything else. This is the boundary case that connects soft attention to plain "pick the best".

Verify: at each $\beta$ the pair sums to $1$: $0.2689+0.7311=1$ ✓, $0.0067+0.9933=1$ ✓. Sharpness rises monotonically toward the argmax limit ✓. (Shrinking $\beta \to 0$ would instead flatten toward uniform — the opposite extreme, matching Example 2.)

---

Example 9 — Cell I: exam twist — do NOT softmax already-normalised numbers

Forecast: Do these three numbers already sum to $1$? If so, what would a second softmax do to them — sharpen, flatten, or leave them alone?

Step 1 — check normalisation before touching softmax. $0.1+0.7+0.2=1.0$ and all are in $[0,1]$: they are already valid attention weights. Why this step? Softmax is for turning scores into weights. These are already weights, so softmax has no job here. To see the damage it would do: $\exp (0.1)=1.105,\exp (0.7)=2.014,\exp (0.2)=1.221$, sum $4.340$, giving $(0.2546,0.4640,0.2814)$ — the sharp $0.7$ has been flattened toward uniform and position 2 lost half its lead. Double-softmax destroys focus; never do it.

Step 2 — compute the context as a weighted sum of the genuine weights. $c_t=0.1[1,0]+0.7[0,2]+0.2[3,3].$ $x$-coordinate: $0.1(1)+0.7(0)+0.2(3)=0.1+0+0.6=0.7$. $y$-coordinate: $0.1(0)+0.7(2)+0.2(3)=0+1.4+0.6=2.0$. So $c_t=[0.7,2.0]$. Why this step? $c_t$ is just the weighted average of the boxes — Step 3 of the recipe, using the weights we were given, with no softmax involved.

Step 3 — compute the attention entropy of the true weights. $H=-(0.1\ln 0.1+0.7\ln 0.7+0.2\ln 0.2)$. Term by term: $0.1\cdot (-2.3026)=-0.2303$, $0.7\cdot (-0.3567)=-0.2497$, $0.2\cdot (-1.6094)=-0.3219$. $H=0.2303+0.2497+0.3219=0.8018$ nats. Why this step? Entropy quantifies how focused these given weights are; at $0.80$ nats they are moderately focused — well below the uniform-3 maximum $\ln 3=1.0986$, confirming position 2's $0.7$ dominates.

Verify: weights already sum to $1$ ($0.1+0.7+0.2=1$) ✓. Context $c_t=[0.7,2.0]$: each coordinate lies inside the min–max of that coordinate across $\{h_i\}$ — $x\in [0,3]$ holds ($0.7$) and $y\in [0,3]$ holds ($2.0$) ✓, as any convex combination must. Entropy $H\approx 0.802$ nats ✓, less than $\ln 3\approx 1.099$. Trap avoided.

---

Recall Self-check before you leave

Equal scores give which attention distribution? ::: Uniform, ${\alpha }_i=1/T_x$ (Ex 2). Why exponentiate before normalising? ::: $\exp$ makes every score positive so negative scores don't break the fractions (Ex 3). Adding the same constant to all scores does what to $\alpha$? ::: Nothing — softmax is shift-invariant (Ex 4). With one input word, what is ${\alpha }_{t1}$? ::: Always $1$, and $c_t=h_1$ (Ex 5). As scores are scaled up without bound by $\beta$, softmax approaches what? ::: A hard argmax — weight $1$ on the largest score (Ex 8). If you are given numbers that already sum to $1$, should you softmax them? ::: No — they are already weights; softmax would flatten them. Just take the weighted sum (Ex 9).

Related reading in the vault: Transformers and self-attention (where these same softmax weights power self-attention), Neural machine translation (the reordering of Ex 7 in the wild), Encoder-decoder architecture and Seq2seq models (why the context vector matters), and Information bottleneck (the very problem attention dissolves).