The attention mechanism intuition: Exercises — AI-ML deep dive

A quick reminder of the machinery you will use (all from the parent):

Here $e_{t i}$ is a raw alignment score (a plain number saying "how relevant is input position $i$ right now"), $α_{t i}$ is an attention weight (a fraction between $0$ and $1$ ), and $c_{t}$ is the context vector (a custom summary of the input for decoder step $t$ ). $T_{x}$ is the input length. One more symbol you will meet below: $y_{t - 1}$ is the previously emitted output token (the word the decoder produced at the last step), fed back in so the decoder knows what it just said.

Level 1 — Recognition

Exercise 1.1 (L1)

Match each symbol to its plain-English meaning.

$s_{t - 1}$ , $h_{i}$ , $α_{t i}$ , $c_{t}$ , $e_{t i}$ — pair each with: (a) weighted summary fed to the decoder, (b) raw relevance number, (c) previous decoder memory, (d) encoder memory at input position $i$ , (e) normalized focus fraction.

Recall Solution

$s_{t - 1}$ → (c) previous decoder memory (the state we align from, because $s_{t}$ doesn't exist yet).
$h_{i}$ → (d) encoder memory at input position $i$ .
$e_{t i}$ → (b) raw relevance number (unbounded, not yet a probability).
$α_{t i}$ → (e) normalized focus fraction (in $[0, 1]$ , sums to $1$ ).
$c_{t}$ → (a) weighted summary fed to the decoder.

Exercise 1.2 (L1)

True or false: Vanilla seq2seq uses all encoder hidden states to build its context, while attention uses only the last one.

Recall Solution

False — it's the reverse. Vanilla seq2seq squeezes everything into the last state $h_{n}$ (the bottleneck). Attention uses all of $h_{1}, \dots, h_{T_{x}}$ , weighted by the $α_{t i}$ . See Seq2seq models for the vanilla baseline.

Exercise 1.3 (L1)

Which quantity must sum to exactly $1$ across all input positions $i$ , for a fixed decoder step $t$ ? The scores $e_{t i}$ , the weights $α_{t i}$ , or the states $h_{i}$ ?

Recall Solution

The weights $α_{t i}$ . That is precisely what the softmax in Step 2 guarantees: $\sum_{i} α_{t i} = 1$ . The scores $e_{t i}$ are unbounded and generally do not sum to $1$ ; the $h_{i}$ are vectors, not fractions.

Level 2 — Application

Exercise 2.1 (L2)

Decoder step $t = 2$ , input length $T_{x} = 3$ . Raw scores are $e_{21} = 1.0$ , $e_{22} = 3.0$ , $e_{23} = 1.0$ . Compute all three attention weights $α_{21}, α_{22}, α_{23}$ .

Recall Solution

Exponentiate: $exp (1) = 2.71828$ , $exp (3) = 20.0855$ , $exp (1) = 2.71828$ . Sum (denominator): $2.71828 + 20.0855 + 2.71828 = 25.5221$ . $α_{21} = \frac{2.71828}{25.5221} = 0.1065, α_{22} = \frac{20.0855}{25.5221} = 0.7870, α_{23} = \frac{2.71828}{25.5221} = 0.1065.$ Check: $0.1065 + 0.7870 + 0.1065 = 1.000$ . ✓ The middle word grabs $\approx 79%$ of the focus.

Exercise 2.2 (L2)

Using the weights from 2.1, and encoder states $h_{1} = [20]$ , $h_{2} = [04]$ , $h_{3} = [- 2 0]$ , compute the context vector $c_{2}$ .

Recall Solution

Why a weighted sum and not a max? Before the arithmetic, ask why attention blends states with weights instead of just picking the single highest-scoring state (a "max"/hard pick). Two reasons. (1) Differentiability: a hard argmax has zero gradient almost everywhere — you couldn't train it by backprop. The weighted sum is smooth, so gradients flow into every $h_{i}$ and the alignment can be learned. (2) Blending genuinely helps: a French article may need both gender info from one word and number info from another; a hard pick throws one away, the weighted sum keeps a mix. So the sum is a soft, trainable lookup — the best of "focus" (via large weights) and "flexibility" (via keeping everything). Now the arithmetic: $c_{2} = 0.1065 [20] + 0.7870 [04] + 0.1065 [- 2 0] .$ First component: $0.1065 (2) + 0.7870 (0) + 0.1065 (- 2) = 0.2130 - 0.2130 = 0$ . Second component: $0 + 0.7870 (4) + 0 = 3.1480$ . $c_{2} = [0 3.148] .$ Notice $c_{2}$ leans hard toward $h_{2}$ 's direction (the up axis) because $α_{22}$ dominates — a soft pick of the middle word.

Figure — The attention mechanism intuition

Exercise 2.3 (L2)

Same scores as 2.1, but now add $5$ to every score: $e_{21} = 6, e_{22} = 8, e_{23} = 6$ . Do the weights change?

Recall Solution

No — the weights are identical to 2.1. Softmax is invariant to adding the same constant to every score: $\frac{e x p ( e _{i} + c )}{\sum _{j} e x p ( e _{j} + c )} = \frac{e ^{c} e x p ( e _{i} )}{e ^{c} \sum _{j} e x p ( e _{j} )} = \frac{e x p ( e _{i} )}{\sum _{j} e x p ( e _{j} )} .$ The $e^{c}$ factor cancels top and bottom. So $α_{21} = 0.1065, α_{22} = 0.7870, α_{23} = 0.1065$ again. Only the differences between scores matter, never their absolute size.

Level 3 — Analysis

Exercise 3.1 (L3)

An attention distribution over $T_{x} = 4$ positions is perfectly uniform: $α_{t i} = 0.25$ for all $i$ . Compute the attention entropy $H = - \sum_{i} α_{t i} lo g α_{t i}$ (natural log). Then compute $H$ for a spiked distribution $(0.97, 0.01, 0.01, 0.01)$ . Interpret.

Recall Solution

Uniform: $H = - 4 \times (0.25 ln 0.25) = - 4 \times (0.25 \times (- 1.38629)) = - 4 \times (- 0.346574) = 1.38629$ . This equals $ln 4$ — the maximum possible entropy for 4 options. Spiked: $H = - (0.97 ln 0.97 + 3 \times 0.01 ln 0.01)$ $= - (0.97 \times (- 0.030459) + 3 \times 0.01 \times (- 4.60517))$ $= - (- 0.029545 - 0.138155) = 0.167700$ . Interpretation: high entropy ( $1.386$ ) = diffuse, "looking everywhere"; low entropy ( $0.168$ ) = sharp, "focused on one word." As the parent note warns, both can be correct — summarization wants diffuse, word-alignment wants sharp.

Exercise 3.2 (L3)

Why does Bahdanau (additive) attention use $s_{t - 1}$ in its score, while Luong (multiplicative) uses $s_{t}$ ? Explain the ordering, not just the names.

Recall Solution

It comes down to when the context vector is needed in the update equation. (Recall $y_{t - 1}$ = the previously emitted output token, defined at the top of this page.)

Bahdanau: the context $c_{t}$ is an input to the RNN update $s_{t} = f (s_{t - 1}, y_{t - 1}, c_{t})$ . To build $c_{t}$ you need scores, but $s_{t}$ doesn't exist yet — it's what you're about to compute. So you must align from the already-available $s_{t - 1}$ .
Luong: the RNN update runs first to produce $s_{t}$ (context-free), then attention is applied to refine the output. By that point $s_{t}$ already exists, so the score $e_{t i} = s_{t}^{T} W_{a} h_{i}$ can legally use it. The lesson: the available state depends on whether attention feeds the RNN (Bahdanau) or follows it (Luong). See Encoder-decoder architecture for the surrounding wiring.

Exercise 3.3 (L3)

You permute the encoder outputs: swap $h_{2}$ and $h_{3}$ . Assuming the scores are computed purely from content (no positional info added), what happens to (a) each weight $α_{t i}$ and (b) the final context vector $c_{t}$ ?

Recall Solution

(a) The weights permute along with the states: whatever weight was attached to $h_{2}$ now attaches to $h_{3}$ and vice versa. The score function $e_{t i} = v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$ depends only on the content $h_{i}$ , so swapping contents swaps the scores identically. (b) The context vector $c_{t} = \sum_{i} α_{t i} h_{i}$ is a sum, and addition is commutative, so $c_{t}$ is unchanged. This is the "permutation invariance" property from the parent note — a warning that pure content attention has no built-in notion of order, which is exactly why Transformers add positional encodings.

Level 4 — Synthesis

Exercise 4.1 (L4)

A decoder emits $T_{y} = 40$ words; the input has $T_{x} = 60$ words. How many alignment scores $e_{t i}$ are computed over the whole translation? Then explain in one sentence why this is the cost the fixed-vector bottleneck was trading away.

Recall Solution

One score per (decoder step, encoder position) pair: $T_{x} \times T_{y} = 60 \times 40 = 2400$ scores. Trade-off: vanilla seq2seq computes essentially $0$ alignment scores by forcing all information through one vector (cheap, lossy); attention pays $O (T_{x} T_{y})$ score computations to avoid that loss — spending computation to buy back information. This is the seed of the quadratic cost that self-attention later scales up.

Exercise 4.2 (L4)

Sketch (describe) the attention heatmap for translating "neural machine translation" → "traduction automatique neuronale" (French reverses the order). Rows are the 3 French output words, columns are the 3 English input words. Where are the hot cells?

Recall Solution

French reverses the noun-modifier order, so the heatmap has a anti-diagonal (bottom-left to top-right of a top-down reading) hot streak:

output ↓ / input →	neural(1)	machine(2)	translation(3)
traduction (1)	·	·	hot
automatique (2)	·	hot	·
neuronale (3)	hot	·	·

"traduction"↔"translation" (col 3), "automatique"↔"machine" (col 2), "neuronale"↔"neural" (col 1). A reordering pattern, not a diagonal — contrast with speech/OCR where alignment is monotonic (main diagonal). See Neural machine translation.

Exercise 4.3 (L4)

Argue why attention improves gradient flow compared to a long RNN encoder chain.

Recall Solution

In a plain RNN, the gradient from output $t$ to an early input must pass through every intermediate state — a long chain of multiplications that shrinks (vanishing gradient). Attention creates a direct, weighted shortcut: $c_{t}$ connects straight to each $h_{i}$ through a single weight $α_{t i}$ . During backprop, the gradient reaches $h_{i}$ in essentially one hop, skipping the long recurrent chain. Fewer multiplicative steps → less vanishing → early words stay learnable even at step $t = 40$ .

Level 5 — Mastery

Exercise 5.1 (L5, degenerate case)

Input length $T_{x} = 1$ (a one-word source). Show that attention becomes trivial, and state what $c_{t}$ equals for every decoder step.

Recall Solution

With one position, softmax over a single score is forced: $α_{t 1} = \frac{e x p ( e _{t 1} )}{e x p ( e _{t 1} )} = 1 for all t .$ The score value is irrelevant — the denominator is the same single term. So $c_{t} = α_{t 1} h_{1} = h_{1}$ for every decoder step. Attention degenerates exactly into vanilla seq2seq's single fixed context. Takeaway: attention only earns its keep when $T_{x} > 1$ ; with one source token there is nothing to choose between.

Exercise 5.2 (L5, limiting behaviour)

Take two scores $e_{t 1} = 0$ and $e_{t 2} = Δ$ . As $Δ \to + \infty$ , what do $α_{t 1}$ and $α_{t 2}$ approach? As $Δ \to 0$ ? This shows softmax interpolating between "hard pick" and "uniform."

Recall Solution

$α_{t 2} = \frac{e ^{Δ}}{1 + e ^{Δ}}, α_{t 1} = \frac{1}{1 + e ^{Δ}} .$

$Δ \to + \infty$ : $α_{t 2} \to 1, α_{t 1} \to 0$ — a hard one-hot pick of position 2 (this is the "argmax" limit).
$Δ \to 0$ : $α_{t 2} \to \frac{1}{2}, α_{t 1} \to \frac{1}{2}$ — perfectly uniform, no focus.
$Δ \to - \infty$ : mirror image, position 1 wins. So the score gap $Δ$ is a continuous knob from diffuse to razor-sharp attention. Why this matters in practice: when scores are computed by a dot product over vectors of dimension $d_{k}$ (here $d_{k}$ = the length of the query/key vectors, i.e. how many numbers each such vector holds), the dot product's typical size grows with $d_{k}$ , so the gaps $Δ$ blow up and softmax saturates into a near one-hot hard pick — killing the gradient into every non-winning position. To prevent that saturation, scaled-dot-product attention divides each score by $d_{k}$ , keeping the gaps moderate and the softmax soft. This is the detail explored in Transformers and self-attention.

Exercise 5.3 (L5, numeric synthesis)

Scores $e = (2.0, 2.0, 2.0, 2.0)$ over $T_{x} = 4$ . Compute the weights and the entropy $H$ . Then a single score jumps: $e = (2.0, 2.0, 2.0, 6.0)$ . Recompute weights and $H$ . Confirm entropy dropped.

Recall Solution

Uniform case: all four scores equal ⇒ $α_{i} = 0.25$ each. Entropy $H = ln 4 = 1.38629$ (maximum, as in 3.1). Spiked case: $exp (2) = 7.38906$ (three times), $exp (6) = 403.4288$ . Denominator $= 3 (7.38906) + 403.4288 = 22.1672 + 403.4288 = 425.5960$ . $α_{1} = α_{2} = α_{3} = \frac{7.38906}{425.5960} = 0.017360, α_{4} = \frac{403.4288}{425.5960} = 0.947919.$ Check sum: $3 (0.017360) + 0.947919 = 0.052081 + 0.947919 = 1.000$ . ✓ Entropy: $H = - (3 \times 0.017360 ln 0.017360 + 0.947919 ln 0.947919)$ $= - (3 \times 0.017360 \times (- 4.05334) + 0.947919 \times (- 0.053470))$ $= - (- 0.211108 - 0.050686) = 0.261794$ . $0.2618 < 1.3863$ ✓ — the spike sharpened attention, entropy fell as predicted.

Recall Self-test checklist

Can you, from memory: Restate the 3-step pipeline (score → softmax → blend)? ::: Yes — $e_{t i}$ , then $α_{t i}$ , then $c_{t} = \sum_{i} α_{t i} h_{i}$ . Explain why softmax ignores a shared additive constant? ::: The $e^{c}$ factor cancels in numerator and denominator; only score differences matter. Say what happens to attention when $T_{x} = 1$ ? ::: $α = 1$ always, $c_{t} = h_{1}$ — degenerates to vanilla seq2seq. Explain low vs high entropy? ::: Low = focused/sharp, high = diffuse/broad; both can be task-correct. Distinguish Bahdanau ( $s_{t - 1}$ ) from Luong ( $s_{t}$ ) ordering? ::: Context feeds the RNN in Bahdanau (state not yet built), context follows the RNN in Luong (state already built).