Foundations — Query, key, value matrices

This page builds every symbol used in the parent topic starting from a smart-12-year-old's blank slate. We never use a symbol before you can see it. By the end you own the alphabet the whole topic is written in.

1. A number, then a list of numbers (a vector)

Before matrices, before attention, there is a single number — like $0.5$. A number alone tells us almost nothing about a word.

So we bundle numbers into a list, written across in brackets: $x=[0.5,-0.3,0.8,0.2]$

The picture: think of a vector as an arrow starting at the origin. Each component says how far to move along one axis. With 2 components you can literally draw the arrow on paper.

WHY the topic needs it: every word (token) in a transformer is represented by such a list of numbers — that is how a computer "holds" a word. See Embedding Layer for where these lists come from.

2. The dimension $d$ — how long is the list?

We keep asking "how many numbers are in this list?" That count gets its own symbol.

The parent note uses several dimensions with subscripts (little labels below the letter):

The subscript is just a name tag: $d_k$ means "the $d$ that belongs to keys". Nothing more mysterious than that.

3. The set symbol ${\mathbb{R}}^d$ — "a real list of length $d$"

The parent writes things like $x_i\in {\mathbb{R}}^{d_{model}}$. Let us earn every piece.

• $\mathbb{R}$ (the blackboard R) means the real numbers — all ordinary numbers like $0.5$, $-3$, $\pi$.
• ${\mathbb{R}}^d$ means a list of $d$ real numbers. So ${\mathbb{R}}^4$ is "any 4-number list."
• $\in$ is the word belongs to. Read $x\in {\mathbb{R}}^4$ as "$x$ is a 4-number list of real values."

WHY the topic needs it: it is the shorthand that tells you the shape of every object at a glance — before you compute anything, you know how big it is.

4. Subscripts and index $i$, $j$ — labelling the tokens

A sentence has many words. We line them up and give each a position number.

• $n$ ::: the sequence length — how many words (tokens) we have.
• $i,j$ ::: two roaming positions. When we say "token $i$ attends to token $j$", $i$ is the one asking and $j$ is the one being looked at.

The picture: a row of numbered boxes, one per word. $i$ and $j$ are two fingers pointing at boxes — sometimes the same box, sometimes different ones.

5. The matrix — a stack of vectors

If one word is a list (a vector), a whole sentence is a stack of lists: a grid of numbers.

The parent's $X\in {\mathbb{R}}^{n\times d_{model}}$ is exactly this: $n$ rows (one per word) and $d_{model}$ columns (the numbers describing each word).

WHY the topic needs it: stacking lets us process every word at once with one operation instead of a loop — the reason transformers are fast. Built by the Embedding Layer and Positional Encoding.

6. Matrix multiplication — the "transform" operation

The parent writes $Q=X{W}^Q$. That $X{W}^Q$ is a matrix multiplication. This is the single most important operation on the page, so we build it slowly.

Start with the simplest case: a row-vector times a column of a matrix. That is a dot product.

The picture: slide one list on top of the other, multiply where they line up, sum the pile.

Now, matrix multiplication is just many dot products arranged in a grid. To get the entry in row $i$, column $j$ of the result, take the dot product of row $i$ of the left matrix with column $j$ of the right matrix.

WHY the topic needs it: $X{W}^Q$ takes each word's $d_{model}$-list and turns it into a shorter $d_k$-list — this is the query transformation. Same machine, three different weight grids, gives Q, K, and V.

7. The weight matrices $W^Q,W^K,W^V$ — learned lenses

The parent uses three:

• $W^Q\in {\mathbb{R}}^{d_{model}\times d_k}$ — turns a word into its query.
• $W^K\in {\mathbb{R}}^{d_{model}\times d_k}$ — turns a word into its key.
• $W^V\in {\mathbb{R}}^{d_{model}\times d_v}$ — turns a word into its value.

The superscript $Q$, $K$, $V$ is just a name tag saying which job the lens does. Each of these is one linear layer — that is exactly what "a learned $W$ times your input" means.

8. Transpose $(\cdot {)}^T$ — flipping rows and columns

The attention formula contains $Q{K}^T$ and the parent also writes $(W^Q{x}_i{)}^T$.

WHY it appears: to dot-product every query against every key we need shapes to line up. $Q$ is $n\times d_k$ and $K$ is $n\times d_k$; flipping $K$ to $K^T$ makes it $d_k\times n$, so $Q{K}^T$ has shape $n\times n$ — one relevance number for every pair of tokens. That grid of numbers is the whole point.

9. Softmax and the weight symbols ${\alpha }_{i,j}$

The parent's final formula uses $\text{softmax}$ and the output uses ${\alpha }_{i,j}$.

• ${\alpha }_{i,j}$ ::: the attention weight — the share of attention token $i$ gives to token $j$. Because softmax makes them sum to 1, the row ${\alpha }_{i,1},{\alpha }_{i,2},\dots$ is like slicing a pie among all tokens.
• ${\sum }_{j=1}^n$ ::: the sum sign. It means "add up the following as $j$ runs from 1 to $n$." So ${\sum }_{j=1}^n{\alpha }_{i,j}{v}_j$ = "mix all value vectors, each weighted by its attention share."

The scaling $\frac{1}{\sqrt{d_k}}$ ( ​ is the square root) shrinks scores before softmax so they don't blow up when $d_k$ is large. Full mechanics live in Scaled Dot-Product Attention.

10. How it all connects

Every foundation above flows down into the parent topic. If any arrow is unfamiliar, revisit that node before reading the parent's derivation.

Equipment checklist

Test yourself — say the answer out loud, then reveal.