Bahdanau and Luong attention: Question bank — AI-ML deep dive

This page is a trap-hunter. Every line below is a claim, a broken statement, or a "why" that a student who half-learned Bahdanau and Luong attention would stumble on. Cover each one, then reveal. If a reveal surprises you, re-read the matching section of the parent note before moving on.

Before you start, make sure the symbols we lean on constantly actually mean something concrete to you.

Definition The symbols this page uses

$h_{i}$ — the encoder hidden state at input position $i$ : a list of numbers (a vector) that the encoder produced after reading the $i$ -th input word. Think "the encoder's memory of word $i$ , coloured by everything around it." There are $T$ of them, $h_{1} \dots h_{T}$ .
$s_{t}$ — the decoder hidden state at output step $t$ : the decoder's own running memory as it generates word $t$ . $s_{t - 1}$ is simply that memory one step earlier.
$d_{h}$ — the length of an encoder state vector $h_{i}$ (how many numbers it holds). " $h_{i} \in R^{d_{h}}$ " just reads " $h_{i}$ is a list of $d_{h}$ real numbers."
$d_{s}$ — the length of a decoder state vector $s_{t}$ . It need not equal $d_{h}$ .
$d_{a}$ — the attention dimension: the length of the shared scratch-space Bahdanau projects both states into before comparing them (a chosen hyper-parameter, often 128–512).

The two score functions this page quizzes you on are worth pinning down before the traps, so each item reads on its own.

Formula The two scoring rules, and their trainable matrices

Bahdanau (additive) runs a tiny feedforward net over each pair: $e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$

$W_{a} \in R^{d_{a} \times d_{s}}$ — learned matrix that projects the decoder state into the shared $d_{a}$ -space.
$U_{a} \in R^{d_{a} \times d_{h}}$ — learned matrix that projects each encoder state into that same $d_{a}$ -space (so the two can be added).
$v_{a} \in R^{d_{a}}$ — ==learned vector that collapses the $d_{a}$ -length blend down to one scalar score==.

Luong (general/multiplicative) compares them with one matrix: $e_{t i} = s_{t}^{⊤} W_{a} h_{i}$

here $W_{a} \in R^{d_{s} \times d_{h}}$ maps encoder space into decoder space before the dot product.

After scoring, both merge context. Luong's merge uses:

$W_{c} \in R^{d_{s} \times (d_{s} + d_{h})}$ — learned matrix that combines the decoder state and context into the attentional state $\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])$ .

Two pictures anchor everything below — glance at them before the questions, and we point back to them by name inside the reveals.

The heatmap above is the attention weights $α_{t i}$ : one row per decoder step, one column per encoder word, each cell's brightness is how much that word was attended to. The softmax curve on the right shows why a small lead in score becomes a big lead in weight.

The alignment picture shows the encoder states $h_{1} \dots h_{T}$ along the bottom, the decoder walking left-to-right on top, and the bright links whose thickness is $α_{t i}$ — the decoder "flipping to the relevant page" at each step.

True or false — justify

TF1. "Attention removes the encoder's final hidden state entirely."

False. Attention removes the reliance on a single fixed vector, but the encoder still produces all states

h_{1} \dots h_{T}

; the decoder is often still initialised from the last one. It stops being the only channel of information, not the only vector.

TF2. "Because attention weights sum to 1, the context vector can never point somewhere no encoder state reached."

True, and the proof is coordinate-wise. Fix any coordinate

k

c_{t, k} = \sum_{i} α_{t i} h_{i, k}

is a convex combination (weights

\geq 0

summing to 1) of the numbers

h_{1, k} \dots h_{T, k}

, so

min_{i} h_{i, k} \leq c_{t, k} \leq max_{i} h_{i, k}

— every coordinate lies inside the range the encoder states span. (The Euclidean norm is then bounded too:

∥ c_{t} ∥ \leq \sum_{i} α_{t i} ∥ h_{i} ∥ \leq max_{i} ∥ h_{i} ∥

by the triangle inequality.) This is exactly why the bright blend in figure s02 always lands between the pink encoder nodes, never outside them.

TF3. "Bahdanau uses the current decoder state

s_{t}

to score."

False. In the score

e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})

the decoder term is the previous state

s_{t - 1}

, because attention is computed before the RNN step — the new state

s_{t}

does not exist yet when scoring happens.

TF4. "Luong's dot-product score works for any encoder/decoder dimensions."

False. Plain dot product

s_{t}^{⊤} h_{i}

requires

d_{s} = d_{h}

, because you can only dot two vectors of equal length. The general variant

s_{t}^{⊤} W_{a} h_{i}

was invented precisely to lift that restriction.

TF5. "Softmax makes attention weights positive, so a low-relevance encoder state gets weight exactly 0."

False.

exp (\cdot)

is always strictly positive, so every weight is strictly greater than 0 — the model can shrink an irrelevant state's weight toward zero but never reaches exactly zero (soft attention, not hard). In figure s01 even the dimmest heatmap cell is a small positive number, never pure black.

TF6. "Additive and multiplicative attention differ mainly in whether they add or multiply the context into the output."

False. The "additive/multiplicative" names describe the score function — Bahdanau adds projected states inside a tanh (

W_{a} s_{t - 1} + U_{a} h_{i}

), Luong multiplies them via a dot product (

s_{t}^{⊤} W_{a} h_{i}

). It has nothing to do with how the context is later merged.

TF7. "Removing the tanh from Bahdanau's score would leave the mechanism unchanged in expressive power."

False. Without tanh,

v_{a}^{⊤} (W_{a} s_{t - 1} + U_{a} h_{i})

collapses to a single linear function of

s_{t - 1}

and

h_{i}

; the tanh non-linearity is what lets it learn alignments that aren't just linear similarity.

TF8. "Luong's general score

s_{t}^{⊤} W_{a} h_{i}

is symmetric: swapping

s

and

h

gives the same value."

False.

W_{a}

is generally not symmetric, so

s^{⊤} W_{a} h \neq = h^{⊤} W_{a} s

in general. Encoder and decoder play different roles, which is the point.

Spot the error

SE1. "Bahdanau:

c_{t} = \sum_{i} α_{t i} e_{t i}

The context is a weighted sum of encoder states

h_{i}

, not of the scalar scores

e_{t i}

. Correct:

c_{t} = \sum_{i} α_{t i} h_{i}

SE2. "

α_{t i} = \frac{exp ( e _{t i} )}{\sum _{j} exp ( e _{j t} )}

The normalising sum runs over the same decoder step

t

across all encoder positions

j

\sum_{j} exp (e_{t j})

. Writing

e_{j t}

sums over the wrong axis (different decoder steps), which is meaningless. In figure s01 this is normalising along a row, not down a column.

SE3. "In Luong, we first compute the context, then run the RNN to get

s_{t}

Reversed. Luong runs the RNN first to get

s_{t}

, then uses that

s_{t}

to score and build

c_{t}

. Doing it the Bahdanau way here breaks the definition.

SE4. "Luong output:

p (y_{t}) = softmax (W_{o} s_{t})

Luong feeds the attentional state

\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])

(that's the

W_{c}

combination matrix), not raw

s_{t}

, into the output layer. Skipping the combine step throws away the context entirely.

SE5. "Dot-product attention has a weight matrix

W_{a} \in R^{d_{s} \times d_{h}}

The dot-product variant has no parameters at all. The matrix

W_{a} \in R^{d_{s} \times d_{h}}

belongs to the general variant. Confusing them mis-counts the model's parameters.

SE6. "Bahdanau's parameters:

W_{a} \in R^{d_{a} \times d_{s}}

U_{a} \in R^{d_{s} \times d_{h}}

v_{a} \in R^{d_{s}}

The shapes must all funnel into the attention dimension

d_{a}

U_{a} \in R^{d_{a} \times d_{h}}

and

v_{a} \in R^{d_{a}}

. Otherwise the sum

W_{a} s_{t - 1} + U_{a} h_{i}

wouldn't be a vector of the same length.

SE7. "Since

v_{a}^{⊤} (\dots)

produces a scalar, we can skip softmax and use

e_{t i}

directly as the weight."

The raw scores can be negative and don't sum to 1, so using them directly would let the context vector blow up or point in nonsensical directions. Softmax is what turns scores into a valid blend.

SE8. "To compute softmax we exponentiate the raw scores directly:

exp (e_{t i})

In practice you first subtract the max score,

exp (e_{t i} - max_{j} e_{t j})

, which is mathematically identical (the constant cancels in numerator and denominator) but prevents

exp

from overflowing to infinity or underflowing to zero on large scores.

Why questions

WHY1. Why does Bahdanau project both states into a shared

d_{a}

-space before comparing them?

The decoder space (length

d_{s}

) and encoder space (length

d_{h}

) may be different sizes and mean different things; the matrices

W_{a}

and

U_{a}

project both into a common

d_{a}

-space so the sum

W_{a} s_{t - 1} + U_{a} h_{i}

is even well-defined, and the model learns the comparison rather than assuming the two spaces already line up.

WHY2. Why does softmax use the exponential rather than just dividing scores by their sum?

Exponentiation makes every weight positive (scores can be negative, so plain division would break) and it amplifies differences — a slightly higher score gets a disproportionately larger weight, sharpening focus. That amplification is exactly the softmax curve in figure s01. See 2.4.7-Softmax-Function.

WHY3. Why is Luong called "cheaper" than Bahdanau despite both using softmax and a context vector?

The difference is only in the score: Bahdanau runs a small feedforward net (

W_{a}

U_{a}

, tanh,

v_{a}

) per encoder position, while Luong's general score is a single matrix multiply

s_{t}^{⊤} W_{a} h_{i}

. Fewer operations and often fewer parameters.

WHY4. Why does attention help with the vanishing-gradient problem of long sequences?

It creates direct weighted paths from every encoder state to the decoder (the links in figure s02), so gradient can flow back to early inputs without passing through every intermediate RNN step — shortening the effective path length that gradients must survive.

WHY5. Why does Bahdanau feed the context into the RNN (input feeding) while Luong combines it after via

W_{c}

Bahdanau lets the context shape the hidden-state evolution itself for tighter integration; Luong keeps the RNN attention-free and merges context afterward through

\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])

for a cleaner, easier-to-implement separation. Neither is universally "correct".

WHY6. Why must the alignment scores be recomputed at every decoder step?

Because relevance changes as you generate: the word you need to attend to for output token 2 (e.g. the subject) differs from token 5 (e.g. the verb). You can see this in figure s01 — the bright cell moves across the row as

t

increases. Freezing the weights would defeat the entire purpose.

WHY7. Why does the context vector have the same dimension as a single encoder state, not

T

times bigger?

It is a weighted average (a blend), not a concatenation, of the

T

states — averaging preserves the shape of one state regardless of how many you blend. That is what keeps the decoder input a fixed size no matter the sequence length.

WHY8. Why does Luong wrap the combined state in a

tanh

(

\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])

The concatenation-and-

W_{c}

-multiply step can produce large values; tanh bounds the output to

(- 1, 1)

per component, stabilising the signal fed to the final softmax and letting the layer learn how much to trust context versus state.

WHY9. Why is attention a stepping stone to self-attention?

Both compute weighted blends via learned similarity; self-attention just makes a sequence attend to itself (queries, keys, values all from the same sequence) and drops the RNN, so the scoring intuition here transfers directly.

Edge cases

EC1. What are the attention weights when all alignment scores are equal?

Softmax of equal scores gives the uniform distribution

α_{t i} = 1/ T

for every

i

, so the context becomes a plain average of all encoder states — the model is "undecided" and looks everywhere equally (a heatmap row of identical mid-tone cells in figure s01).

EC2. What happens to the context vector when the sequence has length

T = 1

With one encoder state, softmax over a single score always gives

α_{t 1} = 1

, so

c_{t} = h_{1}

exactly — attention degenerates to just copying that one state, and the whole mechanism adds nothing.

EC3. What about the truly empty source,

T = 0

(no encoder states at all)?

There is nothing to score, the softmax denominator

\sum_{j = 1}^{0} exp (\cdot) = 0

, and

c_{t} = \sum_{i = 1}^{0} (\dots)

is an empty sum — attention is undefined. In practice you never let this happen: an empty input is rejected or a special start/end token is inserted so

T \geq 1

EC4. If one score is enormously larger than the rest (say

e_{t 2} \to + \infty

), what does attention become?

Softmax saturates toward a one-hot vector:

α_{t 2} \to 1

, all others

\to 0

, so

c_{t} \to h_{2}

. This is the "hard attention" limit — soft attention can approach but never exactly hit it. Numerically this is where the subtract-the-max trick matters, or

exp

overflows.

EC5. What does a negative alignment score mean for a state's weight?

A negative score simply means "less relevant than the others", but after

exp (\cdot)

it still yields a positive weight (just a small one). Negativity is relative, not a veto.

EC6. Two encoder states

h_{i} = h_{j}

are identical — what happens?

They receive identical scores and therefore identical weights; their combined contribution is

(α_{t i} + α_{t j}) h_{i}

. The model can't distinguish them, which is fine — duplicate information is just pooled.

EC7. In Luong dot-product attention with

d_{s} \neq = d_{h}

, what happens if you run it anyway?

The dot product is undefined — you cannot multiply mismatched-length vectors. The model won't even build; you must switch to the general variant (whose

W_{a} \in R^{d_{s} \times d_{h}}

bridges the gap) or reshape. This is a hard error, not a soft degradation.

EC8. What if every embedding/encoder state is the zero vector?

All scores become equal (dot products and projections of zero vanish), so weights are uniform and

c_{t} = 0

— the context carries no information, and the decoder falls back to whatever the RNN state alone provides.

Recall Rebuild the framework from memory

Name the four shared steps of both mechanisms, in order. Answer ::: (1) alignment scoring — compute a scalar $e_{t i}$ for each encoder state; (2) softmax to weights — turn scores into a probability distribution $α_{t i}$ summing to 1; (3) context vector — blend the encoder states, $c_{t} = \sum_{i} α_{t i} h_{i}$ ; (4) output generation — use the context (Bahdanau feeds it into the RNN; Luong combines it via $\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])$ ) to predict $y_{t}$ .