The attention mechanism intuition: Visual walkthrough

We will translate the tiny sentence "Je suis étudiant" → "I am a student" the whole way through, because a small concrete case is easier to see than symbols.

Step 0 — The problem we are trying to fix

Figure — The attention mechanism intuition

WHAT the picture shows: three encoder boxes producing three hidden vectors, all funneling into one thin pipe (the fixed context vector) before the decoder.

WHY it fails: the pipe is a fixed size. A 3-word sentence and a 50-word sentence both get the same thickness of pipe. Long sentences overflow. (See also Gradient flow in deep networks for why long paths are hard to train.)

The plan: instead of one pipe, let the decoder reach back and look at each encoder vector directly, mixing them freshly at every output step.

Step 1 — The raw materials: encoder states $h_{i}$

For our sentence we get three vectors:

$"Je" h_{1} "suis" h_{2} " \overset{e}{ˊ} tudiant" h_{3}$

WHAT the picture shows: three coloured arrows, one per input word — these are our raw ingredients. Attention never rewrites them; it only decides how much of each to use.

WHY keep all three: the failure in Step 0 was throwing $h_{1}$ and $h_{2}$ away and keeping only $h_{3}$ . Attention keeps the whole shelf of ingredients available.

h_{i}

the encoder's hidden vector after reading input word

i

— a summary of the input up to that word.

Step 2 — The decoder's question: state $s_{t - 1}$

At the moment we are about to produce output word $t$ , the decoder holds $s_{t - 1}$ . Think of $s_{t - 1}$ as a question: "Given what I've said so far, which input word do I need next?"

WHAT the picture shows: the decoder having just said "I", now holding $s_{1}$ and looking back at the shelf of $h_{1}, h_{2}, h_{3}$ , asking "who's next?"

WHY $s_{t - 1}$ and not $s_{t}$ : we are about to build $s_{t}$ , and we will need the answer to this question to build it. So $s_{t}$ does not exist yet. We must ask using the state we already have, $s_{t - 1}$ . This ordering is the Bahdanau convention.

Step 3 — Scoring: how well does each $h_{i}$ answer the question?

We now compare the question $s_{t - 1}$ against each ingredient $h_{i}$ and produce a single number — a score — saying "how relevant is input word $i$ right now?"

Formula Alignment score (additive / Bahdanau)

$e_{t i} = squash to 1 number v_{a}^{⊤} tanh (rotate the question W_{a} s_{t - 1} + rotate the ingredient U_{a} h_{i})$

$e_{t i}$ — the raw score for using input word $i$ at output step $t$ . Bigger = more relevant. Can be any real number.
$W_{a} s_{t - 1}$ — take the question and multiply by a learned matrix $W_{a}$ ; this rotates and reshapes it into a shared comparison space of size $d_{a}$ .
$U_{a} h_{i}$ — do the same to the ingredient with matrix $U_{a}$ , landing in the same space so the two can be added.
$tanh (\dots)$ — a smooth squashing function (output between $- 1$ and $1$ ); it lets the network learn nonlinear matches, not just straight-line ones.
$v_{a}^{⊤} (\dots)$ — dot with a learned vector $v_{a}$ to collapse the whole thing down to one scalar.

WHAT the picture shows: the question and one ingredient both being projected into a shared plane, added, bent by $tanh$ , then flattened to a single dot on a number line — the score.

WHY two separate matrices: $s_{t - 1}$ and $h_{i}$ may have different lengths (dimensions $d_{s}$ and $d_{h}$ ). You cannot add vectors of different lengths. $W_{a}$ and $U_{a}$ first map both to the same length $d_{a}$ , then addition makes sense.

WHY $tanh$ and not just addition: pure addition can only detect "point in the same direction" matches. The task "does this ingredient answer this question?" is subtler, so we let a nonlinearity learn it.

For our example at $t = 2$ (producing "am"), suppose the three scores come out: $e_{21} = 0.3 ("Je"), e_{22} = 2.1 ("suis"), e_{23} = 0.5 (" \overset{e}{ˊ} tudiant") .$

Step 4 — Softmax: turn scores into a focus distribution

Scores are unbounded and don't sum to anything meaningful. We want a set of weights that behave like percentages of attention: each between 0 and 1, all summing to 1.

WHAT the picture shows: the three flat scores on the left; after $exp$ -and-normalise they become three bars of a probability distribution summing to 1 — a tall bar over "suis".

Working the numbers for our example: $exp (0.3) = 1.350, exp (2.1) = 8.166, exp (0.5) = 1.649, sum = 11.165$ $α_{21} = \frac{1.350}{11.165} = 0.12, α_{22} = \frac{8.166}{11.165} = 0.73, α_{23} = \frac{1.649}{11.165} = 0.15$

WHY this makes sense: "am" is the translation of "suis", and 73% of the focus landed on "suis". The model found that alignment.

Step 5 — The context vector: a custom-mixed summary

Now we use the weights to build one vector — a fresh summary of the input, tailored to this exact output step.

For our step ( $t = 2$ ): $c_{2} = 0.12 h_{1} + 0.73 h_{2} + 0.15 h_{3}$

WHAT the picture shows: the three ingredient arrows scaled to their weights (a short one, a long one, a short one) then summed tip-to-tail into one resulting arrow $c_{2}$ that leans heavily toward $h_{2}$ .

WHY a weighted sum and not just "pick the biggest": picking the single top word is a hard choice — it has no smooth slope for learning to follow. A weighted blend is a soft lookup: nudging a score slightly nudges the blend slightly, so gradient-based training can improve it. (This soft/smooth property is what makes attention trainable — compare Gradient flow in deep networks.)

Step 6 — Feed context back into the decoder

The blend $c_{t}$ finally lets the decoder move forward one step.

WHAT the picture shows: $c_{t}$ entering the decoder cell twice — once into the state update ( $f$ ) and once into the output ( $g$ ).

WHY twice: once to decide what to remember (state), once to decide what to say right now (output). Empirically both help.

And repeat: with $s_{t}$ now built, we loop back to Step 3 for step $t + 1$ — new question, new scores, new focus, new context.

Step 7 — Edge and degenerate cases

Never let the reader hit a scene we did not show.

WHAT the picture shows: four small attention heatmaps side by side — uniform (flat grey), sharp diagonal, single hot cell, and anti-diagonal — one per case above.

The one-picture summary

WHAT the picture shows the whole pipeline in one flow: the question $s_{t - 1}$ meets each ingredient $h_{i}$ → a score $e_{t i}$ → softmax bars $α_{t i}$ → weighted sum $c_{t}$ → back into the decoder to produce $y_{t}$ and $s_{t}$ → loop.

Recall Feynman retelling — say it in plain words

The decoder just finished saying a word, and it holds a little "state" that carries a question: what do I need next? On a shelf sit summaries of every input word — one arrow each. We hold the question up against each arrow and read off a number: how well does this arrow answer the question? Those numbers are raw, so we run them through softmax, which turns them into percentages that add up to 100. Now we build one custom arrow: take each shelf-arrow, shrink it to its percentage, and add them all up. The word that scored highest dominates this new arrow — that's the context. We hand the context to the decoder, which uses it both to update its memory and to pick the next word. Then it asks a fresh question and we do it all again. No book got squeezed into one sentence; the whole book stays open, and at every step we glance at exactly the page we need.

Recall Symbol checklist

$h_{i}$ ::: encoder summary of input up to word $i$ $s_{t - 1}$ ::: decoder state before producing output word $t$ — carries the "question" $e_{t i}$ ::: raw relevance score of input word $i$ for output step $t$ (unbounded) $α_{t i}$ ::: attention weight — focus fraction, in $[0, 1]$ , summing to 1 over $i$ $c_{t}$ ::: context vector — the weighted blend $\sum_{i} α_{t i} h_{i}$

For the fully-parallel, RNN-free version of exactly this idea, continue to Transformers and self-attention.