Bahdanau and Luong attention: Foundations — AI-ML deep dive

Before you can read the Bahdanau/Luong recipes, you need to see what every letter means. This page builds each one from nothing, in an order where each piece rests on the previous. Nothing here contradicts the parent — it is the ground floor beneath it. See the parent Bahdanau and Luong attention for the full recipes.

0. The scene: an encoder and a decoder

The whole setup lives inside an Encoder–Decoder machine. Picture two conveyor belts.

The encoder reads the input sentence one word at a time and, after each word, produces a little bundle of numbers describing "everything I understood up to here." Each bundle is a hidden state.
The decoder writes the output sentence one word at a time, also carrying its own running bundle of numbers.

Figure 1 — The two belts. Teal boxes are the encoder states $h_{1} .. h_{4}$ , one per input word. The plum box is the decoder state $s_{t}$ writing one output word. The orange arrows are the attention bridge: the decoder reaches back to every encoder box, not just the last. This picture is the whole reason the rest of the symbols exist.

Attention is the bridge between the belts: it lets the writing belt (decoder) reach back to any bundle produced by the reading belt (encoder).

1. A vector — a bundle of numbers with a direction

The picture. A list of 2 numbers is an arrow on a flat page: $h = (3, 1)$ is an arrow reaching 3 right and 1 up. A list of 3 is an arrow in a room. A list of 500 is an arrow in a space we can't draw — but every rule below still holds. In this topic each hidden state is an arrow whose direction encodes the meaning of a word in context.

2. Subscripts and the sequence $h_{1}, \dots, h_{T}$

The picture. Lay the encoder bundles in a row: $h_{1}, h_{2}, \dots, h_{T}$ — one arrow per input word. The decoder wants to choose among this row. $i$ is the pointing finger; $j$ is the same kind of finger used when we sum over the whole row (we use a fresh letter so it doesn't clash with $i$ ).

$d_{h}$ ::: the length of each encoder bundle (how many numbers per arrow).
$d_{s}$ ::: the length of each decoder bundle.

3. The decoder state $s_{t}$ and the two clocks

Why two subscripts matter here. The parent page hinges on a subtle timing difference:

Bahdanau uses $s_{t - 1}$ — the previous decoder state, computed before this step's RNN update.
Luong uses $s_{t}$ — the current state, computed after the RNN update.

Keep two clocks in mind: $t$ counts output words; $i$ (or $j$ ) counts input words. They are independent.

4. Multiplying vectors: the dot product

This is the first real tool, so we earn it carefully.

Figure 2 — Dot product as an agreement meter. The teal arrow $h$ is fixed. The plum arrow $s$ points the same way, so $s^{T} h$ is large positive. The orange arrow $s^{'}$ points opposite, so its dot with $h$ is large negative. The dashed arrow at a right angle to $h$ gives dot $= 0$ . This is exactly the "relevance" number Luong's dot-product score reads off.

Why the topic needs it. Luong's simplest score is literally $s_{t}^{T} h_{i}$ : "how much does the decoder's current thought agree with input word $i$ ?" High agreement → this word is relevant now.

5. Matrices — machines that reshape vectors

The picture. A matrix is a machine that can stretch, rotate, and re-project an arrow into a possibly different space. The shape of the matrix tells you what lengths go in and out — and different attention scores use differently-shaped matrices, so read shapes carefully.

The Luong-general parameters we now have:

$W_{a}$ (Luong general) ::: a learned $d_{s} \times d_{h}$ machine that re-projects each encoder arrow into the decoder's space before the dot product.

The Bahdanau-family symbols ( $U_{a}$ , $v_{a}$ , $d_{a}$ ) are introduced with their formula in the next section, so you meet each symbol exactly where it is used.

6. $tanh$ — the gentle bender (and the additive score)

Figure 3 — The $tanh$ bend. The teal dashed line is the straight map $y = x$ ; the orange curve is $tanh$ . Near the origin they agree (near-linear), but $tanh$ flattens toward the plum dotted ceilings at $+ 1$ and $- 1$ . This flattening is what both bends the network's alignment rule and caps its output size.

Now the additive score, with its symbols defined on the spot. Bahdanau's alignment score is

$e_{t i} = v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$

Reading it left to right, meeting each symbol as it appears:

$W_{a} \in R^{d_{a} \times d_{s}}$ ::: projects the decoder arrow $s_{t - 1}$ into the attention space of length $d_{a}$ .
$U_{a} \in R^{d_{a} \times d_{h}}$ ::: projects the encoder arrow $h_{i}$ into that same attention space of length $d_{a}$ , so the two can be added.
$d_{a}$ ::: the size of that shared attention space (a hyperparameter, often 128–512).
$v_{a} \in R^{d_{a}}$ ::: a learned vector that, via a dot product, collapses the bent length- $d_{a}$ bundle down to one scalar score.

So the recipe is: project both arrows into one space, add them, bend with $tanh$ , then collapse to a single number. That "add-then-bend" is exactly why Bahdanau is called additive attention — and it needs no equal-length constraint, because $W_{a}$ and $U_{a}$ each map into the shared length $d_{a}$ .

7. Turning scores into weights: softmax

We now have one raw score $e_{t i}$ per input word. Raw scores can be any size, positive or negative. We need weights: all positive, summing to 1, so they act like "shares of attention."

Figure 4 — Softmax at work. Left (teal): three raw scores for "The / cat / sat"; note "cat" is only somewhat higher. Right (orange): after softmax the weights are $0.13, 0.79, 0.09$ — the exponential has sharpened the modest lead of "cat" into a decisive $0.79$ share, and the three bars sum to exactly $1$ . That sharpening is why attention can commit to one input word.

$e_{t i}$ ::: raw alignment score between decoder step $t$ and input word $i$ .
$α_{t i}$ ::: the attention weight — word $i$ 's share of the decoder's focus at step $t$ ; all $α_{t i}$ sum to 1.

Recall Every case of the weights

If all scores are equal, softmax gives a flat $α = 1/ T$ each (no focus). If one score dominates, its $α \to 1$ and the rest $\to 0$ (sharp focus). Weights are never negative and never exceed 1. There is no input that breaks it.

8. The context vector $c_{t}$ — the weighted average

The picture. Each encoder arrow $h_{i}$ is scaled down by its share $α_{t i}$ , then all the scaled arrows are placed tip-to-tail. The resulting single arrow leans toward whichever inputs got the biggest shares. Because the shares sum to 1, $c_{t}$ always lands inside the cloud of encoder arrows — never outside it.

Why the topic needs it. $c_{t}$ is the "custom summary of the input for right now." It replaces the single fixed bottleneck the parent's intuition callout warned about — a fresh summary for every output word.

9. Concatenation, the attentional state, and the output

Bahdanau side (input feeding). Here the context is glued to the previous output word and fed into the RNN, so it steers the state update itself:

$s_{t} = RNN ([y_{t - 1}; c_{t}], s_{t - 1})$

Luong side (the attentional state $\tilde{s}_{t}$ ). Here the context is combined after the RNN. Luong glues the current decoder state and the context, then re-mixes them with a learned matrix $W_{c}$ and a $tanh$ bend:

$\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])$

Meeting each new symbol on the spot:

$[s_{t}; c_{t}]$ ::: the glued vector of length $d_{s} + d_{h}$ .
$W_{c} \in R^{d_{s} \times (d_{s} + d_{h})}$ ::: a learned machine that eats that length- $(d_{s} + d_{h})$ glued vector and outputs a length- $d_{s}$ vector — it learns how much to trust context versus the decoder's own state.
$\tilde{s}_{t}$ ::: the attentional hidden state — the $tanh$ -bent blend of decoder state and context. The tilde ("~") just marks it as the attention-adjusted version of $s_{t}$ .

Output, both mechanisms. A final machine $W_{o}$ turns the relevant state into a score per vocabulary word, and softmax turns those into probabilities:

p(y_t \mid y_{<t}, \mathbf{x}) = \text{softmax}(W_o\,\tilde{\mathbf{s}}_t)\quad\text{(Luong)}$$ - $\mathbf{y}_{t-1}$ ::: the previous output word, itself a [[5.1.2-Word-Embeddings|word embedding]] vector. - $W_o$ ::: the final machine turning a decoder bundle into scores over the whole vocabulary, which softmax converts to word probabilities. The whole idea sits under the broader [[3.5.9-AttentionMechanism-Overview|Attention Mechanism overview]], and grows into full [[3.5.11-Self-Attention-and-Transformers|self-attention and Transformers]]. --- ## Prerequisite map ```mermaid graph TD A["Vector = arrow of numbers"] --> B["Sequence h1..hT and subscripts"] A --> C["Decoder state s and clocks t and i"] A --> D["Dot product = agreement meter"] D --> E["Matrix W reshapes and re-projects"] E --> F["tanh bends for curved alignments"] D --> G["Raw scores e_ti"] F --> G G --> H["Softmax makes weights alpha_ti"] H --> I["Context vector c_t weighted average"] B --> I C --> G I --> J["Attentional state and output word y_t"] J --> K["Bahdanau and Luong attention"] ``` --- ## Equipment checklist Cover the right side; answer, then reveal. What does bold $\mathbf{h}$ mean and where does it live? ::: An ordered list of $d_h$ numbers — an arrow in the space $\mathbb{R}^{d_h}$. What do $T$, $i$, and $j$ each count? ::: $T$ = number of input words; $i$ and $j$ = which input word (two clock-fingers over the same row). Difference between $\mathbf{s}_{t-1}$ and $\mathbf{s}_t$, and who uses which? ::: Previous vs current decoder state; Bahdanau uses $\mathbf{s}_{t-1}$ (before RNN), Luong uses $\mathbf{s}_t$ (after RNN). What does the dot product $\mathbf{s}^T\mathbf{h}$ measure, and what does 0 mean? ::: Agreement/similarity between two arrows; 0 means they point at right angles (unrelated). Needs equal lengths. When is the plain Luong score $\mathbf{s}_t^T\mathbf{h}_i$ even defined? ::: Only when $d_s = d_h$; otherwise the slots don't line up and you must use $\mathbf{s}_t^T W_a \mathbf{h}_i$. What shape is $W_a$ in Luong-general, and what shape in Bahdanau? ::: Luong-general: $d_s \times d_h$ (encoder → decoder space). Bahdanau: $d_a \times d_s$ (decoder → attention space, with $U_a$ being $d_a \times d_h$). Which mechanism uses $d_a$, and what is it? ::: Only Bahdanau/additive; $d_a$ is the size of the shared attention space where $W_a\mathbf{s}$ and $U_a\mathbf{h}$ are added. Why is $\tanh$ needed at all? ::: Without a bend, stacked matrices collapse to one straight map; $\tanh$ adds curvature (and caps output size). State the two jobs of softmax. ::: Make all weights positive and make them sum to 1 (a probability distribution), while amplifying the gap toward the top score. What is $\mathbf{c}_t$ in one sentence? ::: The attention-weighted average of all encoder states — a custom input summary for output step $t$. What is $\tilde{\mathbf{s}}_t$ and what shape is $W_c$? ::: The Luong attentional state $\tilde{\mathbf{s}}_t=\tanh(W_c[\mathbf{s}_t;\mathbf{c}_t])$; $W_c$ is $d_s \times (d_s+d_h)$. What does $[\mathbf{a};\mathbf{b}]$ do to lengths? ::: Glues them into one vector of length (length of a) + (length of b).