The attention mechanism intuition: Foundations

This page assumes you know nothing. Every letter, arrow, and squiggle that appears in the parent note is unpacked below, in an order where each idea only uses ideas already built.

0. The picture we keep returning to

Everything in attention lives on top of one image: an input sentence turned into a row of boxes, and an output being written one box at a time while looking back at the input row.

Figure — The attention mechanism intuition

Keep that picture in your head. Every symbol below is a label on some part of it.

1. A sequence, and its length

The input sequence is the sentence we read (e.g. "Je suis étudiant").
The output sequence is what we produce (e.g. "I am a student").

We need names for how many items each has:

Why the topic needs this: attention computes one score for every input position at every output step. So the total amount of work is "output steps $\times$ input positions" $= T_{y} \times T_{x}$ . You cannot even count the work without these two symbols.

The subscript $i$ will always mean "which input position" ( $i$ runs $1, 2, \dots, T_{x}$ ), and the subscript $t$ will always mean "which output step" ( $t$ runs $1, 2, \dots, T_{y}$ ). Two different letters because they count two different things.

2. A vector — a box that holds many numbers

Each word can't be stored as one number; a word carries many features (is it a noun? is it plural? is it about animals?). So we store it as a list of numbers.

Why the topic needs this: the parent note writes $h_{i} \in R^{d_{h}}$ and $s_{t - 1} \in R^{d_{s}}$ . That notation is read aloud as " $h_{i}$ is a vector of length $d_{h}$ ." $d_{h}$ and $d_{s}$ are just how many numbers are in an encoder box vs a decoder box — they can differ, which is exactly why a "projection" step will later be needed. See RNN and LSTM fundamentals for where these vectors come from.

3. Hidden states: $h_{i}$ and $s_{t}$

A network reads words one at a time and, after each word, holds a summary of "everything I've understood so far." That running summary is a vector called a hidden state.

Why the topic needs this: attention asks "for output step $t$ , which input words matter?" To answer it needs (a) a description of what the writer is currently doing — that's $s_{t - 1}$ — and (b) a description of each input word — those are the $h_{i}$ . Both are vectors, and the whole mechanism is a way to compare them. Building these $h_{i}$ is the encoder's job; see Encoder-decoder architecture.

4. A score is a single number: $e_{t i}$

We want to turn "how well does input word $i$ match what the writer needs at step $t$ ?" into one number. A big number = strong match.

Think of a whole grid of these numbers — one per (output step, input word) pair. That grid is exactly the heatmap the parent note draws.

Why a scalar and not a vector? Because next we want to rank and mix the input words, and you can only sort by a single number. A vector has no natural "bigger."

5. The tools that build the score

The score formula $e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$ uses three tools. Let's earn each.

5a. A matrix, and what " $W_{a} s_{t - 1}$ " means

$s_{t - 1}$ has $d_{s}$ numbers; $h_{i}$ has $d_{h}$ numbers. To compare them we must first drag both into a common-size space of length $d_{a}$ . The device that reshapes a length- $d_{s}$ vector into a length- $d_{a}$ vector is a matrix.

So $W_{a} s_{t - 1}$ and $U_{a} h_{i}$ are both length- $d_{a}$ vectors. Now they are the same size and can be added (add matching slots). That is why two different matrices $W_{a}$ and $U_{a}$ appear: one resizes the writer's state, the other resizes an input word.

5b. $tanh$ — the gentle squashing curve

After adding, we pass the result through $tanh$ applied to each number.

Why this tool and not just adding? A plain sum $W_{a} s_{t - 1} + U_{a} h_{i}$ can only express straight-line (linear) relationships. Real language relevance is bendy: "matches only if it's a plural noun near the verb." $tanh$ introduces a bend (nonlinearity), letting the model carve out such curved regions. It also keeps numbers bounded so nothing blows up.

5c. $v_{a}^{⊤}$ — collapsing a vector to one number (dot product)

After $tanh$ we still have a vector of length $d_{a}$ , but we promised $e_{t i}$ would be a single number. The tool that turns a vector into one number is the dot product with a learned vector $v_{a}$ .

Why this tool? We need exactly one scalar to rank input words. The dot product is the simplest, learnable way to compress a length- $d_{a}$ vector into one number, and $v_{a}$ is learned so the model decides which combination of features signals "relevant."

Putting 5a–5c together, in order:

resize both vectors to length $d_{a}$ ( $W_{a} s_{t - 1}$ , $U_{a} h_{i}$ ),
add them,
bend with $tanh$ ,
collapse to one number with $v_{a}^{⊤}$ . That whole pipeline is $e_{t i} = v_{a}^{⊤} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$ .

6. $exp$ and softmax — turning scores into fair shares

The scores $e_{t i}$ can be any numbers (negative, huge, tiny). We want instead weights that are all positive and add up to exactly $1$ , so they read like "percent of attention."

Why $exp$ and not just "divide by the total"? Because raw scores can be negative — dividing negatives by a total gives negative "weights," which is nonsense for a share of attention. $exp$ first lifts everything to be positive, then division makes fair shares. It also sharpens gaps: a slightly bigger score becomes a distinctly bigger share.

7. The context vector $c_{t}$ — a custom-mixed summary

Now we have weights. We use them to blend the input word-vectors.

Why the topic needs this: $c_{t}$ is the whole payoff. Instead of one frozen summary of the sentence for every output word (the old Seq2seq models approach), the writer gets a fresh, focused summary rebuilt for each step $t$ . That is attention in one line.

8. How the writer uses $c_{t}$

Prerequisite map

Each box only depends on boxes above it — that is the order this page built them in.

Worked micro-check (numbers from the parent's Example 1)

At output step $t = 2$ the three scores are $e_{21} = 0.3, e_{22} = 2.1, e_{23} = 0.5$ .

$exp (0.3) \approx 1.350$ , $exp (2.1) \approx 8.166$ , $exp (0.5) \approx 1.649$ .
Sum $= 1.350 + 8.166 + 1.649 = 11.165$ .
$α_{21} = 1.350/11.165 \approx 0.12$ , $α_{22} = 8.166/11.165 \approx 0.73$ , $α_{23} = 1.649/11.165 \approx 0.15$ .
Check they add to $1$ : $0.12 + 0.73 + 0.15 = 1.00$ . ✓

The biggest score ("suis") wins the biggest share — exactly the behaviour softmax promises. This same machinery, stacked and generalised, becomes Transformers and self-attention and powers modern Neural machine translation.

Equipment checklist

What does

T_{x}

count?

The number of items (words) in the input sequence.

What does the notation

h_{i} \in R^{d_{h}}

say in plain words?

h_{i}

is a vector — a list of

d_{h}

real numbers."

Why do we align using

s_{t - 1}

instead of

s_{t}

Because

s_{t}

can't be computed until we have

c_{t}

, and

c_{t}

is built from the writer's state — so we must use the previous one.

What single-number output does the dot product

v_{a}^{⊤} u

give, and why do we want a single number?

One scalar; we need a scalar so we can rank input words (only single numbers have a natural "bigger/smaller").

Why must scores pass through

exp

before dividing?

To make every value positive first, otherwise negative scores would produce meaningless negative "shares."

What two guarantees does softmax give the weights

α_{t i}

Each is in

[0, 1]

, and together they sum to

1

c_{t} = \sum_{i} α_{t i} h_{i}

, what happens if one weight is near

1

and the rest near

0

c_{t}

becomes almost exactly that one

h_{i}

— a near-hard lookup.

Why is

tanh

(a bend) in the score formula rather than plain addition?

A plain sum is linear;

tanh

adds nonlinearity so the model can express curved "relevant only if..." conditions.