Self-attention mechanism in detail: Foundations

This page assumes you have seen none of the notation in the parent note. We build every symbol from a picture, in an order where each one leans only on the ones before it. By the end you can read the formula $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$ one symbol at a time and know what each stroke means.

1. A word becomes a list of numbers: the vector

Computers cannot store the idea "cat". So we hand each word a short list of numbers, like $[1, 0, 1, 0]$ . That list is called an embedding and, mathematically, a list of numbers with a fixed length is a vector.

Figure — Self-attention mechanism in detail

Look at the arrow above. Its two numbers $[x, y]$ tell you how far right and how far up to walk. Why the topic needs this: self-attention never sees the letters "c-a-t"; it only ever sees these arrows. Meaning is now a direction in space, and "similar meaning" will become "similar direction".

The symbol $d_{m o d e l}$ is simply how many numbers are in one word's vector. If every word is a list of 4 numbers, then $d_{m o d e l} = 4$ .

Recall

What does each coordinate of an embedding vector represent physically? ::: A distance to walk along one axis; together the coordinates point the word-arrow in a direction that encodes its meaning.

2. Stacking word-vectors into a grid: the matrix $X$

A sentence is several words, so it is several vectors. Stack them as rows and you get a rectangle of numbers — a matrix.

In the figure, three words become three rows; each row is one arrow from Section 1. Why the topic needs $n$ and $d_{m o d e l}$ : every shape you meet later is built from these two counts, and shape-mismatches are the number-one bug in attention code, so keep them in view.

3. The symbol $R$ and the notation $X \in R^{n \times d_{m o d e l}}$

$R$ is just shorthand for "any real number" — the whole number line, positives, negatives, decimals. The blackboard-bold letter is a set: the bag of all allowed values.

Why the topic needs it: every quantity ( $Q$ , $K$ , $V$ , the weight matrices) will be introduced with an $\in R^{\dots}$ tag so you always know its shape before you use it.

4. Multiplying a matrix by a matrix: how $X W$ works

Self-attention constantly re-expresses the grid $X$ by multiplying it with another grid $W$ . That operation is matrix multiplication, and it is the single mechanical skill you need. (We meet the specific grids $W^{Q}, W^{K}, W^{V}$ and the matrices $Q, K, V$ they produce in Section 5 — here we just learn the machine that combines two grids.)

Follow the highlighted row and column in the figure: three products, then one sum, lands in one output cell. Why the topic needs this: re-expressing words ( $X$ times a weight grid), scoring pairs, and mixing content are all this one operation. Master this and the formula stops being magic.

5. The learned "view-changers": $W^{Q}, W^{K}, W^{V}$

A weight matrix is a grid of numbers the model learns during training. Multiplying $X$ by it produces a new grid — a re-expressed version of the same words.

The results have names we will reuse everywhere: $Q$ (the query matrix, shape $n \times d_{k}$ ), $K$ (the key matrix, shape $n \times d_{k}$ ), and $V$ (the value matrix, shape $n \times d_{v}$ ).

Why three and not one? The same word plays three roles at once (asker, answerer, content-holder). Giving each role its own learned lens lets the model ask one kind of question, advertise a different kind of answer, and hand over a third thing entirely. See 3.2.05-Attention-mechanism-basics for where query/key/value first appear.

The new symbol $d_{k}$ is just the width of the query/key space (how many numbers a query has). $d_{v}$ is the width of the value space. Often $d_{k} = d_{m o d e l} / h$ where $h$ is the number of heads in 4.1.03-Multi-head-attention.

Recall

Why must every $W$ have exactly $d_{m o d e l}$ rows? ::: So the inner shapes match in $X W$ : $X$ has $d_{m o d e l}$ columns, so $W$ needs $d_{m o d e l}$ rows for the multiplication to be legal.

6. Flipping a grid: the transpose $K^{T}$

The little raised $T$ means transpose: turn rows into columns. If $K$ is $(n \times d_{k})$ , then $K^{T}$ is $(d_{k} \times n)$ — the same numbers, reflected across the diagonal.

Why the topic needs it: to compute every query against every key we need shape $(n \times d_{k})$ times $(d_{k} \times n)$ , giving an $(n \times n)$ table — one cell per word-pair. $Q$ and $K$ are both $(n \times d_{k})$ , so $Q K$ is illegal (inner $d_{k} \neq = n$ ); transposing $K$ to $(d_{k} \times n)$ makes the inner numbers ( $d_{k}$ ) match and $Q K^{T}$ works.

7. Measuring "do we point the same way?": the dot product

The core question of attention is "how relevant is word $j$ to word $i$ ?". We answer it with the dot product of their vectors.

The figure shows three cases so you never meet an unshown scenario:

Same direction (top): large positive dot product → high relevance.
Perpendicular (middle): dot product $= 0$ → no relevance.
Opposite (bottom): negative dot product → they actively disagree.

Why the dot product and not, say, subtraction? Because it rewards alignment of direction, which is exactly what "similar meaning" became in Section 1. Now we can name the whole table of these numbers.

Recall

If a query is perpendicular to a key, what is their attention score before softmax? ::: Zero — perpendicular vectors have dot product 0, meaning "no alignment, no relevance".

8. Why divide by $d_{k}$ : the scaling factor

When you add up $d_{k}$ products in a dot product, the total tends to grow as $d_{k}$ grows — more terms, bigger sums. Left alone, scores get huge and the next step (softmax) misbehaves.

Why $d_{k}$ and not $d_{k}$ ? Because the spread (standard deviation), not the count, grows like $d_{k}$ . We cancel exactly what grows. Full variance argument lives in the parent's Step 3; here just remember: scale to keep scores tame so gradients stay healthy (see 5.3.03-Gradient-flow-in-deep-networks).

9. Turning scores into a choice: softmax

We have a row of scores; we want a row of weights that are positive and add to 1 — a share-out, like slicing a pie. That is softmax.

Three properties, three reasons the topic wants exactly this:

Always positive ( $e^{x} > 0$ always) → weights are legal shares.
Sums to 1 (we divide by the total) → each row is a probability distribution over words.
Amplifies gaps → a slightly higher score becomes a clearly bigger share, letting a word "focus".

The output symbol $α_{ij}$ (Greek alpha) is "how much word $i$ attends to word $j$ ". Stack all of them and you get the attention matrix $A$ , shape $n \times n$ , every row summing to 1.

Recall

After softmax, what must each row of the attention matrix add up to, and why? ::: Exactly 1 — softmax divides by the row's total, turning scores into shares of a single pie.

10. Mixing the content: the final weighted sum $A V$

Now use the weights to blend the value vectors.

Word $i$ 's new vector is a weighted average of all value vectors: big-weight words contribute a lot, tiny-weight words almost nothing. This is the "each word rebuilds itself as a blend of others" promise from the top of the page, now made of symbols you have all defined.

The cost of doing this scales as $O (n^{2} d_{k} + n^{2} d_{v})$ : the $Q K^{T}$ scores cost $n^{2} d_{k}$ (an $n \times n$ table, each cell a $d_{k}$ -length dot product) and the $A V$ mix costs $n^{2} d_{v}$ . The recurring $n^{2}$ is the price of scoring every word against every word; details in 4.2.02-Computational-complexity-of-transformers.

Prerequisite map

This map shows the load-bearing order: arrows and grids at the bottom, the final self-attention output at the top. Every foundation on this page feeds a node here — and every node feeds the parent formula.

Equipment checklist

Cover the right side and test yourself.

A vector is ::: an ordered list of numbers, drawable as an arrow whose direction encodes meaning.
The shape $n \times d_{m o d e l}$ means ::: $n$ words (rows), each described by $d_{m o d e l}$ numbers (columns).
$X \in R^{n \times d_{m o d e l}}$ says ::: $X$ is a grid of real numbers with $n$ rows and $d_{m o d e l}$ columns — just a size label.
To multiply matrices legally ::: the inner shape numbers must match; result is (rows of first) $\times$ (columns of second).
$W^{Q}, W^{K}, W^{V}$ are ::: learned lenses turning each word into a query, a key, and a value respectively.
$K^{T}$ (transpose) does ::: swaps rows and columns so $Q K^{T}$ has matching inner dimension and yields an $n \times n$ pair-score table.
The score matrix $S$ is ::: the $n \times n$ grid $S = Q K^{T}$ whose cell $S_{ij}$ is the dot product $q_{i} \cdot k_{j}$ .
The dot product measures ::: how aligned two vectors are — positive for same direction, zero for perpendicular, negative for opposite.
We divide scores by $d_{k}$ because ::: the dot-product spread grows like $d_{k}$ , and we cancel that to keep softmax and gradients stable.
The softmax max-subtraction trick does ::: subtracts the largest score before exponentiating, giving identical weights but preventing overflow/underflow.
Softmax converts scores into ::: positive weights that sum to 1, amplifying larger scores.
The output matrix $Z$ is ::: the $n \times d_{v}$ result $Z = A V$ , each row a word rebuilt as a weighted blend of all value vectors.