The attention mechanism intuition: Question bank

Definition Symbols used on this page (reminder)

All of these come straight from the parent note; keep them handy while you answer.

$h_{i}$ — the encoder hidden state at input position $i$ (a vector summarizing that word plus its context). There are $T_{x}$ of them.
$s_{t}$ — the decoder hidden state at output step $t$ ; $s_{t - 1}$ is the previous one. $s_{0}$ is the very first decoder state (see below).
$e_{t i}$ — the raw (unnormalized) alignment score between decoder step $t$ and input position $i$ , produced by the score function $v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$ before softmax. It is a single real number that can be any size, positive or negative; larger means "more relevant".
$α_{t i}$ — the attention weight (the score $e_{t i}$ after softmax): how much output step $t$ leans on input position $i$ . Lives in $(0, 1)$ and sums to 1 across $i$ .
$c_{t}$ — the context vector, the weighted sum $\sum_{i} α_{t i} h_{i}$ .
$\tilde{y}_{t}$ — the decoder's predicted output at step $t$ (a probability distribution over the vocabulary from which the emitted word is chosen), produced by $g (s_{t}, c_{t}, y_{t - 1})$ ; $y_{t - 1}$ is the previously emitted word.
$T_{x}$ — the input (encoder) sequence length; $T_{y}$ — the output (decoder) sequence length.
$d_{s}$ — the number of components in $s_{t - 1}$ ; $d_{h}$ — the number of components in $h_{i}$ . These can differ, which is why we need projections.
$W_{a}, U_{a}, v_{a}$ — the learned parameters of the additive score: $W_{a}$ maps $s_{t - 1}$ and $U_{a}$ maps $h_{i}$ into a shared space of size $d_{a}$ , and $v_{a}$ (a vector of length $d_{a}$ ) collapses the $tanh$ output to one scalar.
$s_{0}$ — the initial decoder state, not emitted by any word. It is usually produced from the encoder (e.g. the last encoder state $h_{n}$ passed through a small learned layer), so $s_{t - 1}$ is always defined even at $t = 1$ .
$H$ — the attention entropy $H = - \sum_{i} α_{t i} lo g α_{t i}$ , measured in nats (natural-log units) or bits (log base 2). It is one number per decoder step that measures how spread out the weights are: $H = 0$ means all mass on one position (razor focus), $H = lo g T_{x}$ means perfectly uniform (maximum spread).

Figure — The attention mechanism intuition

Intuition Why the softmax

denominator is not optional The score $e_{t i}$ alone tells you a raw relevance, but raw relevances live on an arbitrary scale — the network could double all of them and nothing would change about "which is biggest". To combine the $h_{i}$ into one context vector we need weights that behave like proportions: non-negative and summing to 1, so no information leaks in or out. The exponential $exp (e_{t i})$ makes every term positive; dividing by $\sum_{j} exp (e_{t j})$ rescales the whole set so the parts add to a whole. Drop that denominator and the "weights" no longer form a distribution — $c_{t}$ could blow up or shrink arbitrarily just because the scores drifted. The two figures below show the same three scores before normalization (bars of any height) and after (bars that fill exactly one unit).

True or false — justify

Attention removes the need for an RNN encoder entirely in a Bahdanau-style seq2seq.

False — without the RNN the states

h_{i}

collapse to bare word embeddings with no surrounding context, so attention would be looking up context-free vectors. See RNN and LSTM fundamentals.

The attention weights

α_{t i}

always sum to exactly 1 for a fixed decoder step

t

True — softmax normalizes over all

T_{x}

encoder positions, so

\sum_{i} α_{t i} = 1

by construction; that is what lets us read them as "percent of focus".

A high

α_{t i}

proves that source word

i

causes the output word at step

t

False — attention is correlation, not causation; a high weight only means

h_{i}

was useful for the prediction, which could be agreement, co-occurrence, or a spurious shortcut.

Softmax guarantees the model focuses sharply on one input position.

False — if all scores

e_{t i}

are roughly equal, softmax returns a near-uniform

α_{t i} \approx 1/ T_{x}

, which is deliberately diffuse and can be the correct behaviour.

In Bahdanau attention the alignment score uses the current decoder state

s_{t}

False — it uses

s_{t - 1}

, because

c_{t}

is needed to compute

s_{t}

; you cannot align with a state that does not exist yet.

Attention makes the fixed-length context bottleneck of vanilla seq2seq disappear.

True — instead of one frozen vector

h_{n}

, each decoder step builds a fresh weighted summary of all encoder states, so long-sequence detail no longer has to squeeze through one vector. See Information bottleneck.

Multiplicative (Luong) and additive (Bahdanau) attention always give identical weights.

False — they use different score functions (

s_{t}^{T} W_{a} h_{i}

v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})

) and even different timing of

s

, so their learned alignments differ.

Because attention adds

O (T_{x} \times T_{y})

scores, it is asymptotically cheaper than the RNN itself.

False — the quadratic attention cost is often the dominant term for long sequences; that quadratic growth is exactly why sparse/efficient attention was later invented.

Permuting the encoder outputs

h_{i}

leaves the final translation unchanged.

False — the weights permute along with the states so the weighted sum is preserved, but the encoder RNN itself is order-sensitive, so in practice reordering the inputs changes the

h_{i}

and thus the output.

Spot the error

"We compute

α_{t i} = exp (e_{t i})

directly and feed it to the decoder."

Missing the normalization denominator — you must divide by

\sum_{j} exp (e_{t j})

so the weights are non-negative and sum to 1; without it they are unbounded and do not form a distribution, letting

c_{t}

scale arbitrarily.

"The context vector is

c_{t} = \sum_{i} α_{t i} s_{t - 1}

Wrong operand — the weighted sum is over the encoder states

h_{i}

, not the decoder state;

c_{t} = \sum_{i} α_{t i} h_{i}

"Since the decoder-state length

d_{s}

and encoder-state length

d_{h}

differ, we just add

s_{t - 1} + h_{i}

inside the score."

You cannot add vectors of different lengths (

d_{s} \neq = d_{h}

); that is precisely why the learned matrices

W_{a}

and

U_{a}

project both into a shared space of size

d_{a}

before the

tanh

"Attention weights are learned parameters we store after training."

They are computed at inference time for every step from the current states; the learned parameters are the projection matrices

W_{a}, U_{a}

and the score vector

v_{a}

, not the

α

's themselves.

"A diagonal attention heatmap means the model is broken."

A clean diagonal means monotonic alignment (natural for OCR or speech), which is healthy; scattered patterns appear for reordering languages. See Neural machine translation.

"We drop

v_{a}^{T}

and use

tanh (W_{a} s_{t - 1} + U_{a} h_{i})

as the score."

That leaves a vector of length

d_{a}

, but a score must be a single scalar; the dot product with the learned vector

v_{a}

is what collapses it to one number.

Why questions

Why is

c_{t}

fed into the decoder both to update

s_{t}

and to generate the prediction

\tilde{y}_{t}

Once to decide what information to store in the state and once to decide what to emit now; empirically using it in both places improves output quality.

Why do we pass alignment through a

tanh

before scoring?

The

tanh

builds a learned nonlinear "compatibility space" so the model can encode relationships more expressive than a raw linear match.

Why is softmax preferred over simply picking the single highest-scoring

h_{i}

Hard selection is non-differentiable and brittle; softmax gives a smooth, differentiable soft lookup that can blend several positions and be trained by gradient descent. See Gradient flow in deep networks.

Why does attention help most on long sequences specifically?

Short inputs fit comfortably in one context vector, but long inputs overflow the fixed bottleneck; attention grants each decoder step a targeted view, so information loss no longer scales with length.

Why can two different tasks both be "correct" with opposite attention entropies

H

Focused (low-

H

) attention fits word-alignment tasks like translation, while diffuse (high-

H

) attention fits tasks like summarization that must blend many positions — the right shape depends on the task, not on a universal rule.

Why does attention count as interpretability but not proof?

The weights show where the model looked, giving a readable window, but "looked at" does not equal "was caused by", so patterns must be checked against domain knowledge.

Edge cases

If every encoder score

e_{t i}

is identical, what is

c_{t}

The weights become uniform

1/ T_{x}

, so

c_{t}

is the plain average of all

h_{i}

— a completely unfocused summary, equivalent to no selective attention.

For an input of length

T_{x} = 1

, what does attention reduce to?

With one position, softmax forces

α_{t 1} = 1

, so

c_{t} = h_{1}

every step — attention degenerates to just reusing that single state.

What is the attention entropy

H = - \sum_{i} α_{t i} lo g α_{t i}

at its maximum, and when?

It is maximal,

lo g T_{x}

(in nats), exactly when attention is uniform; it drops toward

0

as the mass concentrates on one position.

Can an attention weight ever be exactly

0

or exactly

1

Not with a plain softmax on finite scores — outputs are strictly in

(0, 1)

; you only approach

0

1

as a score's gap grows without bound.

What happens to alignment for the very first decoder step

t = 1

when there is no emitted word yet?

You still have the initial decoder state

s_{0}

— typically built from the encoder (e.g. the last state

h_{n}

through a small learned layer) — so scoring proceeds against

s_{0}

; there is no missing-state problem because

s_{t - 1}

is always defined, and

s_{0}

exists by that initialization.

For English→Japanese where word order reverses, what does the heatmap look like and is it wrong?

An anti-diagonal or scattered pattern appears because the languages reorder; this is correct behaviour, not a bug, and is exactly the reordering case attention was meant to handle. See Seq2seq models and Transformers and self-attention.