Bahdanau and Luong attention: Visual walkthrough

Step 1 — The problem we are solving, as a picture

WHAT. A translator model reads a sentence word by word. Each word it reads leaves behind a little "memory tag" — a list of numbers we call a hidden state. We write these tags as $h_{1}, h_{2}, \dots, h_{T}$ , one per input word, where $T$ is how many words there are.

The old (encoder–decoder) trick was: throw away every tag except the very last one, $h_{T}$ , and force the whole meaning of the sentence through it. That single tag is the bottleneck.

WHY it fails. One fixed-size list of numbers cannot hold a 40-word sentence any more than one sticky note can hold a whole book. Long sentences get crushed (this is a cousin of information decay over long chains).

PICTURE. In the figure, the blue tags on the left are ALL the memories. The old model (red) keeps only the last. Attention (green) keeps a doorway to every single one.

Step 2 — The one question attention keeps asking

WHAT. At each moment the decoder (the part writing the translation) has its own memory tag, the decoder state $s$ , a list of $d_{s}$ numbers ( $s \in R^{d_{s}}$ ). For every input tag $h_{i}$ it asks one question:

"How relevant is input word $i$ to the word I am about to write?"

The answer is a single number, the alignment score $e_{i}$ . Bigger score = more relevant.

WHY a number. We need something we can compare and rank across all input words. A single number per word is the simplest comparable thing.

PICTURE. The orange decoder tag shines a "relevance beam" at each blue input tag; the brightness of each beam is the score $e_{i}$ .

Step 3 — Two ways to compute the score (the whole Bahdanau-vs-Luong story)

WHAT. How do we turn " $s$ meets $h_{i}$ " into a number? Two recipes.

Recipe A — Luong (multiplicative). Multiply and add, slot by slot. This is the dot product:

$e_{t i} = decoder tag s_{t}^{⊤} input tag h_{i} = \sum_{k} s_{t, k} h_{i, k}$

$s_{t}^{⊤}$ — the $⊤$ ("transpose") just means "lay the list on its side so we can multiply it against the other list."
$\sum_{k}$ — add up over every slot $k$ .
Each term $s_{t, k} h_{i, k}$ — multiply the $k$ -th number of each tag.

WHY a dot product measures relevance. When two lists point the same way (agree slot by slot), their products are all positive and the sum is large. When they disagree, positives and negatives cancel and the sum is near zero. So the dot product is a built-in "how much do these two agree?" meter.

Recipe B — Bahdanau (additive). Squeeze both tags through a tiny neural net first, then read off a score:

$e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$

Before we read it, here are the three learnable pieces with their exact shapes (they must be defined before use):

Now the term-by-term reading:

$W_{a} s_{t - 1}$ and $U_{a} h_{i}$ — reshape each tag into the shared space of size $d_{a}$ , so tags of different lengths can meet.
$+$ — add them (this "adding" is why it is called additive attention).
$tanh$ — a soft squashing function that bends everything into $(- 1, 1)$ , letting the network learn curved, non-straight-line notions of relevance.
$v_{a}^{⊤}$ — a final dot product that collapses the $d_{a}$ numbers down to one score.

WHY two recipes. Luong's dot product assumes $s$ and $h$ already live in the same space (needs $d_{s} = d_{h}$ ). Bahdanau learns the translation between the two spaces with $W_{a}, U_{a}$ , so it is more flexible but has more parameters.

PICTURE. Left: two arrows and the dot product as "shadow of one arrow on the other." Right: both tags funneled through boxes $W_{a}, U_{a}$ , added, bent by $tanh$ , squeezed to a number.

Step 4 — Turning scores into weights with softmax

WHAT. We now have raw scores $e_{t 1}, e_{t 2}, \dots, e_{tT}$ . Some may be negative, some large — they do not form a tidy set of proportions. We convert them to attention weights $α_{t i}$ that are all positive and add up to exactly $1$ :

$α_{t i} = \frac{e x p ( e _{t i} )}{j = 1 \sum T exp ( e _{t j} )}$

$exp (\cdot)$ — the exponential $e^{x}$ . It is always strictly positive (even when its input is negative) and grows fast, so bigger scores get amplified.
Numerator — how much word $i$ "shouts."
Denominator — the total shouting of everyone, so we are taking each word's share.

WHY softmax and not just "divide by the sum". Two solid reasons. First, guaranteed positivity: $exp$ turns every score into a positive number, so every weight is genuinely a fraction of a whole — no matter the signs of the raw scores. (Dividing raw scores by their sum can misbehave: if the scores happen to sum to zero you divide by zero, and mixed signs give a messy, hard-to-interpret split.) Second, sharper gradients and focus: the exponential magnifies gaps, so a score of $2.1$ against $0.5$ becomes a clearly dominant weight, and during training the derivatives push weight decisively toward the winning word. (This is the same softmax used everywhere in classification.)

PICTURE. A bar chart: raw scores on the left, then after $exp$ all positive, then after dividing, bars that stack to a full unit.

Step 5 — Building the context vector

WHAT. The weights tell us how much of each input memory to keep. We now blend all the input tags in those proportions to build the context vector $c_{t}$ :

$c_{t} = \sum_{i = 1}^{T} α_{t i} h_{i}$

$α_{t i} h_{i}$ — shrink input tag $i$ by its weight.
$\sum_{i}$ — add all the shrunk tags together.

Because the weights sum to $1$ , $c_{t}$ is a weighted average — it lives among the input tags, leaning toward the ones with high weight.

WHY a weighted average. It is a differentiable way to "mostly look at cat, glance at the others." We could have picked only the top word (a hard choice), but averaging is smooth, so gradients can flow and the model can learn where to look.

PICTURE. Three blue input arrows scaled to lengths $0.155, 0.768, 0.077$ , then chained tip-to-tail; the resulting green arrow $c_{t}$ points almost exactly at $h_{2}$ (cat).

Step 6 — Two different moments to use the context (the timing split)

WHAT. Now the context $c_{t}$ enters the decoder. The two mechanisms differ only in when.

Bahdanau — context goes IN before the step. Glue the previous word's embedding and the context together and feed the pair to the RNN:

$s_{t} = RNN ([y_{t - 1}; c_{t}], s_{t - 1})$

$[\cdot; \cdot]$ — concatenation: stack two lists into one longer list.
Here the context shapes the new decoder memory itself (called input feeding).

Luong — context is combined AFTER the step. Run the RNN first, then mix:

$s_{t} = RNN (y_{t - 1}, s_{t - 1}), \tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])$

$\tilde{s}_{t}$ — the "attentional state," a learned blend of raw decoder memory and context.
$W_{c} \in R^{d_{s} \times (d_{s} + d_{h})}$ — a combining matrix that takes the stacked $(d_{s} + d_{h})$ -long list $[s_{t}; c_{t}]$ back down to a $d_{s}$ -long state. It is its own parameter, distinct from $W_{g}$ and $W_{a}$ .
$tanh$ — bounds the blend so it cannot blow up.

WHY it matters. Bahdanau uses the previous state $s_{t - 1}$ to attend, so context steers the RNN update — more integrated. Luong uses the current state $s_{t}$ to attend, so scoring sees a fresher memory and the code is cleaner.

PICTURE. Two flow lanes. Top (Bahdanau): context arrow enters the RNN box. Bottom (Luong): RNN runs, then context merges downstream through a $tanh$ box.

Step 7 — Edge and degenerate cases

WHAT & WHY. A picture-based derivation must survive the corners:

One input word ( $T = 1$ ). Softmax of a single score is always $1$ . So $c_{t} = h_{1}$ exactly — attention politely collapses to "look at the only thing there."
All scores equal. Softmax gives $α_{t i} = 1/ T$ for all $i$ : the context becomes the plain average of every input tag — no focus, maximum spread. This is what an untrained model does at step zero.
One score dominates hugely. As one $e_{t i} \to + \infty$ relative to others, its weight $\to 1$ and all others $\to 0$ : soft attention becomes hard selection of a single word.
$d_{s} \neq = d_{h}$ . The bare Luong dot product is undefined (can't multiply mismatched lists); you must use Bahdanau, or Luong-general with its translator matrix $W_{g}$ .

PICTURE. Three mini weight-bars: uniform (flat), peaked (one tall bar), and single-word ( $T = 1$ , one full bar).

Definition The large-

$T$ edge case: local (windowed) attention Everything above is global attention — every output word scores all $T$ input tags. When $T$ is very large (long documents), that is $T$ score computations per output word, which is slow, and the softmax spreads weight thinly over hundreds of positions. Luong's second variant, local attention, fixes this: first predict a single centre position $p_{t}$ in the input, then only score input tags inside a small window $[p_{t} - D, p_{t} + D]$ (width set by a fixed $D$ ). WHY it helps: cost drops from $O (T)$ to $O (2 D + 1)$ per step and attention stays sharp. WHEN to use it: long sequences where relevance is roughly monotonic (translation, where output word $t$ usually aligns near input position $t$ ). Global attention remains the default for short-to-medium sequences and is what this page derives.

Recall Check the corners

When there is exactly one input word, the context vector equals ::: $h_{1}$ (softmax of one number is 1) When every alignment score is identical, each weight equals ::: $1/ T$ , giving a plain average of all encoder states The pure dot-product score needs which condition on dimensions ::: $d_{s} = d_{h}$ (equal decoder and encoder tag lengths) For very large $T$ , which cheaper variant scores only a window of input tags ::: local (windowed) attention, cost $O (2 D + 1)$ per step instead of $O (T)$

The one-picture summary

WHAT. The whole pipeline as one river: input tags → scores → softmax weights → context → decoder output. Bahdanau and Luong are the same river; they only differ where the context pours in.

Recall Feynman retelling — say it to a 12-year-old

The model reads a sentence and leaves a sticky note ( $h_{i}$ ) for every word. To write each output word, it holds up its own current thought ( $s$ ) and asks each sticky note, "how much do you matter right now?" That question is answered by a single number, the score — either by checking how much the two lists agree (dot product, Luong) or by pushing both through a tiny learn-able mixer (Bahdanau). The scores get passed through softmax, which turns them into a set of percentages that add to 100%. The model then makes a blended memory — the context vector — by mixing all the sticky notes in exactly those percentages, so it mostly reads the important word and only glances at the rest. Finally it feeds that blend to the decoder, along with the embedding of the word it just wrote ( $y_{t - 1}$ ): Bahdanau slips it in before the memory updates, Luong stirs it in after. If the sentence is enormous, it switches to local attention and only reads the sticky notes near where it is currently looking. Repeat for every output word, and the book stays open the whole time — no more single-sticky-note bottleneck.

See also: 3.5.9-AttentionMechanism-Overview, 3.5.11-Self-Attention-and-Transformers, 5.1.2-Word-Embeddings, and the parent topic.