Foundations — Multi-head attention

This page assumes you know nothing about the notation used in the parent Multi-Head Attention note. We build every symbol from the ground up, in the order they depend on each other. Nothing is used before it is drawn.

0. What are we even looking at?

A sentence like "the cat sat" is, to a computer, just words. But neural networks only eat numbers. So the very first thing that happens is: each word becomes a list of numbers. Everything else is arithmetic on those lists.

Keep that picture in your head: words in a row, each word its own list of numbers. Every symbol below is a name for some part of this picture.

1. The number $n$ — how many words

Picture: in figure s01 above, we had 3 words, so $n=3$. In the parent's example, $n=10$.

Why the topic needs it: attention compares every word to every other word. If there are $n$ words, that's an $n\times n$ grid of comparisons. So $n$ controls how big that grid is.

2. A vector — one word as an arrow of numbers

Picture: the list of numbers describing one word. To build intuition we stand a single word's list up as an arrow in space: 2 numbers = an arrow on a flat page, 3 numbers = an arrow in a room.

Why the topic needs it: a word's "meaning" is stored as the direction and length of its arrow. Two words with similar meaning point in similar directions — and measuring "similar direction" is exactly what attention will do (see §6).

See also Query-Key-Value Intuition, which uses these arrows directly.

3. The number $d_{\text{model}}$ — how long each word-vector is

Picture: how many numbers are in one word's list. Longer list = more room to store meaning.

Why the topic needs it: this is the "budget" of numbers the model spends per word. Multi-head attention will split this budget among heads — that is the whole trick of §7.

4. A matrix — a grid of numbers, and what stacking vectors gives you

Picture: figure s01 is a matrix — lay every word down as one row and stack the $n$ rows, giving a grid of shape $[n\times d_{\text{model}}]$ (one row per word, $d_{\text{model}}$ numbers across). In the parent, $Q,K,V\in {\mathbb{R}}^{10\times 512}$ means "a grid with 10 rows — one per word — and 512 columns".

Why the topic needs it: whole sentences are processed as one grid, not word-by-word. This is what makes transformers fast.

5. Matrix multiply & the superscript ${}^T$ — reshaping and mixing

Picture: figure s03 walks one output entry through: slide a row of $A$ across a column of $B$, multiply the aligned numbers, add them up — that single sum is one cell of the answer. Do this for every row–column pair to fill the whole output grid.

Why the topic needs it: The parent writes $Q_i{K}_i^T$. Because each word is a row (our §4 convention), $Q_i$ is $[n\times d_k]$ and $K_i$ is also $[n\times d_k]$ — you cannot multiply them directly (inner numbers $d_k$ and $n$ don't match). Transposing $K_i$ to $[d_k\times n]$ fixes the shapes, and the multiply $Q_i{K}_i^T$ becomes an $n\times n$ grid where cell $(i,j)$ is "row $i$ dotted with row $j$" = how much word $i$ aligns with word $j$. See §6 for why that number means "similarity".

This reshaping-by-multiply is exactly what Linear Projections in Neural Networks is about, and the projection matrices $W_i^Q,W_i^K,W_i^V$ (defined in §7 below) are what get multiplied in.

6. The dot product — the "how alike are these two arrows?" tool

Picture: two arrows and the angle between them.

All cases (never leaves you guessing):

• Same direction → large positive → "very relevant".
• Perpendicular (90°) → zero → "unrelated".
• Opposite direction → negative → "actively pushing apart".

Why the topic needs it: this is the raw score of "how much word $i$'s question aligns with word $j$'s key" in the parent's walkthrough.

7. Q, K, V and the projection matrices — the three roles a word plays

Why the topic needs it: without these three roles there is no "asking" and "answering". The subscripts you saw in the parent — $q_{3,1}$, $k_{j,1}$ — are now fully named: word 3's query in head 1, word $j$'s key in head 1.

8. Splitting the budget: $h$ and $d_k$

Picture: cut the 512-long word-vector into 8 shorter pieces of 64 each.

Why the topic needs it: this is the reason it's called "multi-head". Each head sees the words through its own smaller projection, so one can track grammar while another tracks meaning.

9. Softmax — turning raw scores into attention weights

All cases: one very large score → its weight ≈ 1, others ≈ 0 (near hard-max). All equal scores → equal weights $1/n$ (pure average). Negative scores are fine — $e^s$ is still positive.

Why the topic needs it: the weights ${\text{weight}}_{3,j}$ in the parent tell word 3 how much of each other word's value to keep. See Self-Attention and Attention Visualization to see these weights drawn as a heat grid.

10. The weighted sum of values — how a word gathers its answer

Picture: figure s05 — the same $d_v$-long output slot, filled mostly by the value-vector of the word with weight 0.6, a little by the one with 0.3, barely by the one with 0.1.

Why the topic needs it: this final blend is the answer attention produces — a context-aware version of each word, built from the words it decided were relevant.

11. The scale factor $\sqrt{d_k}$

Why divide by it? A dot product of two 64-long vectors adds up 64 products, so it tends to grow large. Large scores make softmax nearly one-hot and kill the gradient. Dividing by $\sqrt{d_k}$ keeps scores in a sane range so learning stays stable.

${\text{score}}_{3,3}=\frac{40}{\sqrt{64}}=\frac{40}{8}=5.0$

How the pieces feed the topic

Each foundation on the left must be understood before the one it points to. Everything funnels into the topic node at the bottom.

Equipment checklist

Test yourself — reveal only after you have an answer.