Exercises — Bahdanau and Luong attention

(a) Bahdanau, the alignment score step. Tell-tales: the additive + inside tanh, three parameter groups $W_a,U_a,v_a$, and it uses the previous state ${\mathbf{s}}_{t-1}$. (b) Luong (general/bilinear), the alignment score step. A single matrix $W_a$ sandwiched between two vectors, uses the current state ${\mathbf{s}}_t$. (c) Luong, the attentional hidden state step — it blends the decoder state with the context AFTER attention. Bahdanau has no such line.

(i) $e_{ti}$ — a raw score, a single real number, any sign. (ii) ${\alpha }_{ti}$ — a softmax output, so $0\le {\alpha }_{ti}\le 1$. (iii) ${\mathbf{c}}_t$ — a weighted average of the ${\mathbf{h}}_i$, so it lives in ${\mathbb{R}}^{d_h}$, exactly the size of one encoder state.

Softmax exponentiates then normalises. $\exp (1)=2.718$, $\exp (3)=20.086$, $\exp (1)=2.718$; sum $=25.522$. $\alpha =(0.107,0.787,0.107)$ The middle word wins with $\approx 78.7\%$ of the attention. Note the two equal scores get equal weight — softmax preserves ties.

${\mathbf{c}}_t=0.107(1,0)+0.787(0,1)+0.107(1,1)$ First coordinate: $0.107+0+0.107=0.214$. Second coordinate: $0+0.787+0.107=0.894$. ${\mathbf{c}}_t\approx (0.214,0.894).$ It leans hard toward ${\mathbf{h}}_2$, exactly the dominant word — the picture below shows ${\mathbf{c}}_t$ sitting near ${\mathbf{h}}_2$.

${\mathbf{s}}_t^{\top }{\mathbf{h}}_i$ is a dot product: it needs both vectors to have the same length. With $512\ne 256$ the multiplication is undefined. The general/bilinear variant ${\mathbf{s}}_t^{\top }{W}_a{\mathbf{h}}_i$ fixes it: $W_a\in {\mathbb{R}}^{512\times 256}$ turns the 256-length ${\mathbf{h}}_i$ into a 512-length vector so the dot product lines up. It's the minimal fix — just one matrix, no tanh, no concat.

Bahdanau uses $W_a\in {\mathbb{R}}^{d_a\times d_s}$, $U_a\in {\mathbb{R}}^{d_a\times d_h}$, $v_a\in {\mathbb{R}}^{d_a}$: $d_a{d}_s+d_a{d}_h+d_a=128\cdot 256+128\cdot 512+128=32768+65536+128=98432.$ Luong-general uses $W_a\in {\mathbb{R}}^{d_s\times d_h}$ and $W_c\in {\mathbb{R}}^{d_s\times (d_s+d_h)}$: $d_s{d}_h+d_s(d_s+d_h)=256\cdot 512+256\cdot 768=131072+196608=327680.$ Here Luong is bigger (327,680 vs 98,432) — because $W_c$, the attentional-hidden-state matrix, is large. The common claim "Luong has fewer parameters" refers only to the scoring matrix, not the whole layer. Read the tables carefully.

The state that is available is dictated by when the RNN runs.

• Bahdanau computes attention before the RNN step, feeding ${\mathbf{c}}_t$ into the RNN. At that moment the only decoder state that exists is the previous one, ${\mathbf{s}}_{t-1}$ — the current ${\mathbf{s}}_t$ hasn't been produced yet.
• Luong runs the RNN first (${\mathbf{s}}_t=\text{RNN}({\mathbf{y}}_{t-1},{\mathbf{s}}_{t-1})$), so by scoring time the fresh ${\mathbf{s}}_t$ is already available and is the natural query. Cause → effect: the timing fixes the state, not the other way around.

• Bahdanau score: tanh is the non-linearity of a one-hidden-layer network. Without it, $v_a^{\top }(W_a\mathbf{s}+U_a\mathbf{h})$ collapses to a single linear map $v_a^{\top }{W}_a\mathbf{s}+v_a^{\top }{U}_a\mathbf{h}$ — that is just an additive dot-product-style score with no capacity to learn interactions. tanh buys expressive alignment.
• Luong ${\tilde{\mathbf{s}}}_t$: tanh bounds the blended vector into $(-1,1)$ per coordinate, so concatenating ${\mathbf{s}}_t$ and ${\mathbf{c}}_t$ (which can be large) doesn't hand an exploding vector to the output softmax. This ties to 3.3.5-Vanishing-and-Exploding-Gradients.

Consistency check: $W_a$ must be ${\mathbb{R}}^{d_s\times d_h}$ so that ${\mathbf{s}}_{t-1}^{\top }{W}_a$ has length $d_h$ and dots cleanly with ${\mathbf{h}}_i$. Because we score with ${\mathbf{s}}_{t-1}$, no ${\tilde{\mathbf{s}}}_t$ step is needed — context already entered through the RNN. This is essentially how many modern seq2seq toolkits actually wire attention.

Softmax is invariant to adding a constant and strictly monotone, so equal weights require equal scores: $e_{t1}=e_{t2}=e_{t3}$. Take $(0,0,0)$: ${\alpha }_i=\frac{\exp (0)}{\exp (0)+\exp (0)+\exp (0)}=\frac{1}{3}.$ Any constant triple $(c,c,c)$ works identically, because $\exp (c)$ cancels top and bottom. Uniform attention = "the model found no reason to prefer any word."

(a) If ${\mathbf{c}}_t\approx {\mathbf{h}}_T$ always, the decoder is effectively reading only the encoder's final state — the classic fixed-length bottleneck of the plain 3.5.8-Encoder-Decoder-Architecture that attention was invented to escape. Attention is present but dead. (b) Likely the scores $e_{ti}$ are near-constant across $i$ except for ${\mathbf{h}}_T$, or the score magnitudes are so large that softmax saturates (one score dominates and its gradient vanishes) — a saturation cousin of the issues in 3.3.5-Vanishing-and-Exploding-Gradients. A common trigger: the query ${\mathbf{s}}_{t-1}$ carries little positional signal, and ${\mathbf{h}}_T$ (having seen the whole sentence) always scores highest under a dot product. (c) Fixes: (i) scale the scores, e.g. divide by $\sqrt{d_h}$ to de-saturate softmax — the move that later defines scaled dot-product attention in 3.5.11-Self-Attention-and-Transformers; (ii) add positional information to encoder states so ${\mathbf{h}}_i$ differ meaningfully; (iii) use Bahdanau's richer tanh score to break the last-state dominance.

By construction ${\mathbf{c}}_t={\sum }_{i=1}^T{\alpha }_{ti}{\mathbf{h}}_i$ with ${\alpha }_{ti}\ge 0$ (softmax outputs are positive) and ${\sum }_i{\alpha }_{ti}=1$ (softmax normalises). A sum of vectors with non-negative coefficients summing to 1 is by definition a convex combination — it lies inside the convex hull of $\{{\mathbf{h}}_1,\dots ,{\mathbf{h}}_T\}$. Why desirable: ${\mathbf{c}}_t$ can never invent a direction absent from the encoder — no hallucinated features, no magnitude blow-up. The summary is always a "blend of things the encoder actually saw," which stabilises training. Illustrated in the s02 figure: ${\mathbf{c}}_t$ sits between the ${\mathbf{h}}_i$ arrows, never beyond them.

Bahdanau scores with the ::: previous decoder state ${\mathbf{s}}_{t-1}$, before the RNN step Luong scores with the ::: current decoder state ${\mathbf{s}}_t$, after the RNN step The context vector is always a ::: convex combination of encoder states (weights $\ge 0$, sum to 1) Removing tanh from Bahdanau's score collapses it into ::: a single linear map with no interaction capacity