Bahdanau and Luong attention: Worked examples

Before starting, one reminder of what each symbol means in plain words:

The scenario matrix

Every attention computation you will ever face falls into one of these cells. The rest of the page fills each one.

#	Cell (scenario class)	What makes it tricky	Covered by
A	Dot-product scoring, aligned spaces ( $d_{s} = d_{h}$ )	Softmax on hand-computed dot products	Ex 1
B	General/bilinear scoring ( $d_{s} \neq = d_{h}$ )	The matrix $W_{a}$ bridges two sizes	Ex 2
C	Bahdanau additive scoring	$tanh$ non-linearity + $v_{a}^{⊤}$ read-out	Ex 3
D	All scores equal (degenerate)	Uniform attention — the "no-info" case	Ex 4
E	One score $\to + \infty$ (limiting)	Attention collapses to one word (hard attention)	Ex 5
F	Negative & zero scores, sign handling	$exp$ of negatives stays positive	Ex 6
G	Real-world word problem (translation)	Reading the $α$ matrix as an alignment	Ex 7
H	Exam twist: Bahdanau vs Luong timing	Which decoder state feeds the score?	Ex 8

(All of $d_{s}$ , $d_{h}$ , $d_{a}$ are defined in the callout just above.)

Forecast: $s_{t}$ points purely along the first axis. Which $h_{i}$ do you think wins the attention — and by how much? Guess before computing.

Compute the raw scores. Why this step? The dot product $s^{⊤} h$ measures how much two vectors point the same way — that is exactly "relevance" when both live in the same space. $e_{1} = (1) (1) + (0) (0) = 1, e_{2} = (1) (0) + (0) (2) = 0, e_{3} = (1) (2) + (0) (0) = 2.$
Exponentiate. Why? Softmax needs positive, order-preserving numbers; $exp$ makes negatives positive and amplifies gaps. $e^{1} = 2.7183, e^{0} = 1, e^{2} = 7.3891, sum = 11.1074.$
Normalise. Why? Dividing by the sum forces the weights to add to $1$ — a genuine probability split of "where my attention goes". $α_{1} = 0.2447, α_{2} = 0.0900, α_{3} = 0.6652.$
Blend the memories. Why? The context vector is the mix the decoder actually reads. Add coordinate-by-coordinate: $first coord: 0.2447 (1) + 0.0900 (0) + 0.6652 (2) = 0.2447 + 1.3304 = 1.5751,$ $second coord: 0.2447 (0) + 0.0900 (2) + 0.6652 (0) = 0.1800,$ $c_{t} = (1.5751, 0.1800) .$

Verify: $h_{3}$ had the biggest score ( $2$ ) and wins the most weight ( $0.665$ ) — matches the forecast. Weights sum: $0.2447 + 0.0900 + 0.6652 = 0.9999 \approx 1$ . ✓

Forecast: $W_{a}$ reshapes $s_{t}$ into the 3-D encoder space. Will the two words end up sharing attention roughly equally, or lopsidedly?

Map the decoder into encoder space. Why? $s_{t}^{⊤} W_{a}$ is a $3$ -vector that lives where the $h_{i}$ live — now a dot product is legal. $s_{t}^{⊤} W_{a} = (1, 2) (100110) = (1, 2, 1) .$
Dot with each memory. Why? Same relevance idea as Ex 1, now in the shared 3-D space. $e_{1} = (1, 2, 1) \cdot (1, 0, 0) = 1, e_{2} = (1, 2, 1) \cdot (0, 0, 1) = 1.$
Softmax. Why? Turn scores into a probability split. $α_{1} = \frac{e ^{1}}{e ^{1} + e ^{1}} = 0.5, α_{2} = 0.5.$

Verify: The two scores came out equal, so attention splits $50/50$ — the bilinear form correctly bridged the size gap and the symmetry is preserved. Note $W_{a}$ has $d_{s} \cdot d_{h} = 2 \times 3 = 6$ parameters, matching the parent's count. ✓

Worked example Example 3 — Cell C: Bahdanau additive scoring with

$tanh$ The three pieces of the additive scorer (defined first). Bahdanau's score is $e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$ . Each learned piece has a job:

$W_{a}$ — projects the decoder state into the shared attention space (size $d_{a}$ ).
$U_{a}$ — projects the encoder state into that same shared space, so the two can be added.
$v_{a}$ — a single read-out vector that collapses the $d_{a}$ -length blend down to one number, the score.

Recall $s_{t - 1}$ is the previous decoder thought (see the symbol callout) — Bahdanau scores before its RNN step, so only $s_{t - 1}$ exists yet.

Setup. $s_{t - 1} = (1)$ , encoder $h_{1} = (1), h_{2} = (- 1)$ (all scalars, $d_{a} = 1$ for arithmetic clarity, so each of $W_{a}, U_{a}, v_{a}$ is a single number). $W_{a} = 2, U_{a} = 1, v_{a} = 1.$ Find $α$ .

Forecast: Both inputs get pushed through $tanh$ , which saturates near $\pm 1$ . Do you expect the weights to be extreme or gentle?

Pre-activation for each. Why? Bahdanau projects decoder ( $W_{a} s_{t - 1}$ ) and encoder ( $U_{a} h_{i}$ ) into the shared space and adds them — the "additive" name. $z_{1} = 2 (1) + 1 (1) = 3, z_{2} = 2 (1) + 1 (- 1) = 1.$
Apply $tanh$ . Why $tanh$ and not just $z$ ? The non-linearity lets the network learn alignments that a plain sum cannot; it also bounds values in $(- 1, 1)$ so no score can explode. $tanh (3) = 0.9951, tanh (1) = 0.7616.$
Read out with $v_{a}$ (here $= 1$ ). So $e_{1} = 0.9951, e_{2} = 0.7616$ .
Softmax. $α_{1} = \frac{e ^{0.9951}}{e ^{0.9951} + e ^{0.7616}} = 0.5581, α_{2} = 0.4419.$

Verify: $z_{1} > z_{2}$ so $α_{1} > α_{2}$ — order preserved. Because $tanh$ squashed the raw gap ( $3$ vs $1$ became $0.995$ vs $0.762$ ), the weights are gentle, not extreme — exactly the saturation effect we forecast. ✓

Forecast: If nothing is more relevant than anything else, where should attention land?

Softmax of equal scores. Why? $exp$ of the same number gives the same value; the sum just multiplies by the count. $α_{i} = \frac{e ^{1.5}}{4 e ^{1.5}} = \frac{1}{4} = 0.25 for every i .$
Context vector. Why? With uniform weights the blend is a plain average. $c_{t} = \frac{1}{4} (h_{1} + h_{2} + h_{3} + h_{4}) .$

Verify: Weights sum to $4 \times 0.25 = 1$ . ✓ Interpretation: attention has learned nothing selective — it degrades gracefully to the old fixed-vector average, which is the seq2seq bottleneck the parent note warned about. Notice the score value $1.5$ never mattered: softmax is shift-invariant. ✓

Forecast: One word gets shouted louder and louder. Where does attention concentrate?

Write the weight. Why? Track the winning word's share as $z$ grows. $α_{1} = \frac{e ^{z}}{e ^{z} + e ^{0} + e ^{0}} = \frac{e ^{z}}{e ^{z} + 2} = \frac{1}{1 + 2 e ^{- z}} .$
Take the limit. Why? $e^{- z} \to 0$ as $z \to \infty$ . $lim_{z \to \infty} α_{1} = \frac{1}{1 + 0} = 1, α_{2} = α_{3} \to 0.$

Verify: At $z = 5$ : $α_{1} = 1/ (1 + 2 e^{- 5}) = 1/ (1 + 0.01347) = 0.9867$ — already near $1$ . This is hard attention: the soft blend collapses to picking one word. The opposite limit ( $z \to - \infty$ ) drives $α_{1} \to 0$ , handing all attention to the others. ✓

Forecast: Negatives look dangerous. Can a probability come out negative? (No — but see why.)

Exponentiate — signs vanish here. Why? $exp$ maps every real number, negative or positive, to a positive value. That is precisely why softmax can eat any score. $e^{- 2} = 0.1353, e^{0} = 1, e^{3} = 20.0855, sum = 21.2208.$
Normalise. $α_{1} = 0.00638, α_{2} = 0.04712, α_{3} = 0.94650.$

Verify: All three weights are positive despite a negative score. Sum $= 0.00638 + 0.04712 + 0.94650 = 1.0000$ . ✓ The most-negative score $e_{1}$ got the tiniest (but non-zero!) weight — attention never fully ignores anything unless a score is $- \infty$ . ✓

Forecast: French says the noun first. Which English word should attention pick for output word 1?

Softmax the scores. Why? Convert relevance into a readable percentage split. $α_{1} = \frac{e ^{0.2}}{e ^{0.2} + e ^{2.5}} = \frac{1.2214}{1.2214 + 12.1825} = 0.0911,$ $α_{2} = 0.9089.$
Read the alignment. Why? The largest $α$ tells us which source word the decoder is "flipping back to."

The model puts $90.9%$ of its attention on $h_{2}$ = "car" while emitting "voiture" (= car). ✓

Verify: French voiture rouge reorders adjective and noun; attention has correctly aligned output-1 "voiture" to input-2 "car" across the reordering — the exact skill a fixed bottleneck could not do (see the attention overview). Weights sum: $0.0911 + 0.9089 = 1.000$ . ✓

Now do the second output word "rouge" (= red) the same way. Suppose its scores are $e_{1} = 2.3, e_{2} = 0.5$ : $α_{1} = \frac{e ^{2.3}}{e ^{2.3} + e ^{0.5}} = 0.8581, α_{2} = 0.1419.$ So output-2 "rouge" puts $85.8%$ of its attention on $h_{1}$ = "red" — the other diagonal. Together the two rows form the alignment picture below.

Forecast: Same formula shape, different decoder state. Can the timing alone flip which word wins? Guess before computing.

Bahdanau uses the PREVIOUS state $s_{t - 1}$ . Why this step? Bahdanau computes attention before the RNN step (parent note, "input feeding"), so at scoring time only $s_{t - 1}$ exists yet. $e_{1} = (1, 0) \cdot (1, 0) = 1, e_{2} = (1, 0) \cdot (0, 1) = 0.$ $α_{1} = \frac{e ^{1}}{e ^{1} + e ^{0}} = \frac{2.7183}{2.7183 + 1} = 0.7311, α_{2} = 0.2689.$ So Bahdanau's blend leans on $h_{1}$ .
Luong uses the CURRENT state $s_{t}$ . Why this step? Luong runs the RNN first, then scores with the freshly-updated $s_{t}$ (parent note, "attention after the step"). $e_{1} = (0, 1) \cdot (1, 0) = 0, e_{2} = (0, 1) \cdot (0, 1) = 1.$ $α_{1} = \frac{e ^{0}}{e ^{0} + e ^{1}} = 0.2689, α_{2} = \frac{e ^{1}}{e ^{0} + e ^{1}} = 0.7311.$ So Luong's blend leans on $h_{2}$ instead.
Name each recipe. Why? The whole point of the twist is to attach the right label. The recipe that scores with the older $s_{t - 1}$ is Bahdanau (additive origins, attention before the step); the one that scores with the current $s_{t}$ is Luong (multiplicative, attention after the step).

Verify: The winning word flips — $h_{1}$ under Bahdanau, $h_{2}$ under Luong — purely from which decoder state feeds the score, which is the single most-tested distinction. Both weight-pairs sum to $0.7311 + 0.2689 = 1.0000$ . ✓ See 3.5.11-Self-Attention-and-Transformers for where this timing question disappears entirely (queries and keys are computed in parallel).

Recall Self-check

Softmax is invariant to adding a constant to every score. True or false? ::: True — that is why the score value $1.5$ was irrelevant in Ex 4, and why the max-subtraction stability trick is safe. Why can't Luong dot-product scoring be used when $d_{s} \neq = d_{h}$ ? ::: The dot product needs both vectors the same length; the bilinear form $s^{⊤} W_{a} h$ inserts $W_{a}$ to bridge the sizes. In Ex 7, which input word did output-1 "voiture" align to, and with what weight? ::: "car" ( $h_{2}$ ), with $α_{2} \approx 0.909$ . As one score $\to + \infty$ , soft attention becomes _____ attention. ::: hard (one-hot) attention. What is $d_{a}$ ? ::: The attention hidden size — the length of the shared space Bahdanau projects both states into before $tanh$ .

See also: 3.5.8-Encoder-Decoder-Architecture · 5.1.2-Word-Embeddings · 3.3.5-Vanishing-and-Exploding-Gradients (why attention beats long-range recurrence).