Query, key, value matrices: Visual walkthrough

Before we touch attention, we must earn every symbol. Let's begin from the very first thing a transformer sees.

Step 1 — A token is a row of numbers

WHAT. After the tokenizer splits the text, an embedding layer turns each token (say the token "cat") into a fixed list of numbers, e.g. four of them: $x = [0.5, - 0.3, 0.8, 0.2]$

Here $x$ is called an embedding vector. The little "vector" just means an ordered list of numbers. Each number is one coordinate — think of it as one dial the model can read. Four numbers means four dials, so we say this vector lives in a 4-dimensional space, written $R^{4}$ . The general size is $d_{m o d e l}$ ("model dimension"); here $d_{m o d e l} = 4$ .

WHY. Computers cannot multiply the letters c-a-t. They can only multiply numbers. So the first job is to represent meaning as numbers. Everything downstream is arithmetic on this row.

PICTURE. Below, the four numbers of $x$ are drawn as four colored bars. Tall bar = big number, bar below the line = negative number. This bar-strip is our mental image of "one token."

Step 2 — Multiplying a vector by a matrix = mixing the dials

WHAT. A matrix is just a rectangular grid of numbers. When we write $q = x W$ , we take our row $x$ and a grid $W$ , and produce a new, shorter row $q$ . Each output number is a weighted sum of the input numbers: $q_{col c} = dial 1 x_{1} \cdot W_{1, c} + x_{2} \cdot W_{2, c} + x_{3} \cdot W_{3, c} + x_{4} \cdot W_{4, c}$

Read this term by term: to make output column $c$ , walk down column $c$ of the grid, multiply each grid entry $W_{r, c}$ by the matching input dial $x_{r}$ , and add them all up.

WHY this tool and not another? We need a way to re-mix the dials — to build brand-new features like "is this a verb?" out of the raw meaning dials. A linear transformation (matrix multiply) is the simplest such re-mixer: it can amplify, suppress, flip the sign of, and blend any dials. It is exactly what a linear layer does. We choose it because it is (a) flexible enough to build any weighted combination and (b) simple enough to train with backpropagation.

PICTURE. Each output cell is fed by arrows from every input bar; the arrow's number is the grid entry it passes through. One output cell = one full column of the grid consumed.

Step 3 — The Query: "what am I looking for?"

WHAT. Multiply the token by a special grid $W^{Q}$ (shape $4 \times 2$ ) to get a query vector: $2 numbers q = 4 numbers x 4 \times 2 grid W^{Q}$

The output has only 2 numbers because we chose $d_{k} = 2$ . So $q$ lives in a smaller 2-dimensional "search space" $R^{2}$ — small enough to draw as a real arrow on a page.

Term by term:

$q_{1} = x_{1} (0.1) + x_{2} (0.2) + x_{3} (- 0.3) + x_{4} (0.6) = 0.05 - 0.06 - 0.24 + 0.12 = - 0.13$
$q_{2} = x_{1} (0.4) + x_{2} (- 0.1) + x_{3} (0.5) + x_{4} (0.2) = 0.20 + 0.03 + 0.40 + 0.04 = 0.67$

$q = [- 0.13, 0.67]$

WHY. The raw dials of $x$ are a hopeless mush of meaning, position, and grammar all together. The query grid $W^{Q}$ is a learned lens that pulls out only "what kind of context this token wants." We give the search its own space so it can be optimized just for matching, independent of meaning.

PICTURE. The 4-bar strip on the left funnels through the grid and lands as one 2D arrow $q$ on a plane. That arrow is the token's search intent.

Step 4 — The Key: "what do I advertise?"

WHAT. Take the same token $x$ but a different grid $W^{K}$ to get the key vector $k$ : $k = x W^{K}$ With the numbers from the parent note:

$k_{1} = 0.5 (- 0.2) + (- 0.3) (0.5) + 0.8 (0.4) + 0.2 (- 0.1) = - 0.10 - 0.15 + 0.32 - 0.02 = 0.05$
$k_{2} = 0.5 (0.3) + (- 0.3) (0.1) + 0.8 (- 0.2) + 0.2 (0.6) = 0.15 - 0.03 - 0.16 + 0.12 = 0.08$

$k = [0.05, 0.08]$

WHY a different grid from the query? Because searching and being-found are different jobs. My query says "I want a verb." My key says "I am a noun, subject-ish." If we forced $W^{Q} = W^{K}$ , every token could only ever match tokens like itself — attention could never be asymmetric (cat→sat ≠ sat→cat). Two grids buy us direction.

PICTURE. The same 4-bar strip now flows through a second, different grid and lands as a different arrow $k$ on the same plane. One token → two arrows.

Step 5 — The dot product turns two arrows into one relevance number

WHAT. To score "how relevant is token $j$ to token $i$ " we compute the dot product of the query of $i$ with the key of $j$ : $score = q \cdot k = q_{1} k_{1} + q_{2} k_{2}$ For our single token attending to itself: $score_{1, 1} = (- 0.13) (0.05) + (0.67) (0.08) = - 0.0065 + 0.0536 = 0.0471$

WHY the dot product and not something else? We need a single number saying "how aligned are these two arrows." The dot product is exactly that: it is large-positive when the arrows point the same way, zero when they are perpendicular, and negative when they point opposite.

First, the length symbol $∣ \cdot ∣$ . For any arrow $q = [q_{1}, q_{2}]$ , its length (how far its tip reaches from the origin) is written $∣ q ∣$ and computed by the Pythagorean rule: $∣ q ∣ = q_{1}^{2} + q_{2}^{2}$ The two vertical bars just mean "size of this arrow, ignoring its direction" — always a non-negative number. So $∣ q ∣$ and $∣ k ∣$ below are simply the reaches of the two arrows.

What is $θ$ ? Draw both arrows starting from the same point (the origin). The ==angle $θ$ == is simply the opening between them, measured from $q$ to $k$ — $0°$ if they lie on top of each other, $90°$ if they form a right angle, $180°$ if they point dead opposite. The dot product packages the two lengths together with this angle into one formula: $q \cdot k = ∣ q ∣ ∣ k ∣ cos θ$ Here $cos θ$ (cosine) is the "agreement dial": $cos 0° = 1$ (full agreement), $cos 90° = 0$ (none), $cos 180° = - 1$ (full disagreement). So the more two search-and-offer arrows agree in direction, the higher the relevance. No other simple operation reads "agreement of direction" so cleanly. This is the core of Scaled Dot-Product Attention and drives Self-Attention.

PICTURE. $q$ and $k$ drawn tail-to-tail from the origin; the angle $θ$ between them is shaded in mint, and the resulting score sits on a little "relevance meter" that swings positive/zero/negative.

Step 6 — Why we divide by $d_{k}$ before softmax

WHAT. We don't feed the raw score into the next stage. We scale it: $scaled score = \frac{q \cdot k}{d _{k}} = \frac{0.0471}{2} \approx \frac{0.0471}{1.414} \approx 0.0333$

WHY. A dot product adds up $d_{k}$ product-terms. If we double the number of dimensions, the sum tends to grow (roughly like $d_{k}$ ). Huge scores make the next step — softmax — saturate: it puts almost all weight on one token and nearly zero gradient everywhere else, so learning stalls. Dividing by $d_{k}$ cancels that growth and keeps scores in a gentle range where gradients still flow.

PICTURE. Two bar charts of scores — "unscaled" (tall, spiky) vs "scaled" (calm) — with a softmax curve behind showing the flat, dead zones the tall bars fall into.

Step 7 — Softmax: turning scores into listening-percentages

WHAT. For a query, we now have one scaled score against every token in the sentence: a row $s = [s_{1}, s_{2}, \dots, s_{n}]$ . The softmax function turns this row of raw numbers into a row of weights that are all positive and sum to 1 — a set of percentages saying "how much of my attention goes to each token": $α_{j} = softmax (s)_{j} = \frac{e ^{s_{j}}}{m = 1 \sum n e ^{s_{m}}}$

Term by term: $e^{s_{j}}$ ( $e \approx 2.718$ raised to the score) makes every entry positive and blows up big scores more than small ones; the denominator $\sum_{m} e^{s_{m}}$ adds up all those positives so that dividing by it forces the whole row to sum to 1. Bigger score in → bigger slice out.

WHY this tool? We need to convert "raw relevance" into a proper weighting — non-negative and totalling 100% — so we can take a weighted average of information. Softmax is the smooth, differentiable way to do that (differentiable matters for backprop).

PICTURE. Three scaled scores enter; softmax bends them into three slices of a pie that sum to 1.

Step 8 — The Value: "what information do I actually hand over?"

WHAT. A third grid $W^{V}$ (shape $4 \times 3$ , so $d_{v} = 3$ ) turns the token into a value vector — the payload: $v = x W^{V}$

$v_{1} = 0.5 (0.3) + (- 0.3) (- 0.1) + 0.8 (0.6) + 0.2 (0.2) = 0.15 + 0.03 + 0.48 + 0.04 = 0.70$
$v_{2} = 0.5 (- 0.2) + (- 0.3) (0.4) + 0.8 (0.1) + 0.2 (- 0.5) = - 0.10 - 0.12 + 0.08 - 0.10 = - 0.24$
$v_{3} = 0.5 (0.5) + (- 0.3) (0.2) + 0.8 (- 0.3) + 0.2 (0.4) = 0.25 - 0.06 - 0.24 + 0.08 = 0.03$

$v = [0.70, - 0.24, 0.03]$

WHY a third grid, and why size 3? What made a token relevant (its key) is not the same as what it should contribute (its value). "sat" is relevant to "cat" because of a subject–verb link (a key/query fact), but the information passed on is "cat = animal, subject" (a value fact). Separating them lets the model match on one thing and deliver another. The value space may need a different width, so $d_{v}$ need not equal $d_{k}$ .

PICTURE. The same 4-bar strip forks a third time through $W^{V}$ into a 3-bar payload $v$ , drawn as a little parcel labelled "info to send."

Step 9 — Stacking into matrices (the full sequence)

WHAT. A sentence is many rows stacked: $X \in R^{n \times d_{m o d e l}}$ , one row per token ( $n$ = sentence length). The same three grids act on the whole stack at once: $Q = X W^{Q}, K = X W^{K}, V = X W^{V}$ Row $i$ of $Q$ is token $i$ 's query; row $j$ of $K$ is token $j$ 's key; row $j$ of $V$ is its value.

The transpose symbol $⊤$ . Writing $K^{⊤}$ (read "K transpose") means flip the grid across its diagonal: every row becomes a column and every column becomes a row. So if $K$ has shape $n \times d_{k}$ (tokens down, key-features across), then $K^{⊤}$ has shape $d_{k} \times n$ (key-features down, tokens across). Why do this? To multiply two grids, the across-count of the left grid must equal the down-count of the right grid. $Q$ is $n \times d_{k}$ ; to pair its queries against the keys we need something $d_{k} \times n$ on the right — exactly $K^{⊤}$ . The product $Q K^{⊤}$ is then $n \times n$ , and its entry in row $i$ , column $j$ is precisely the dot product of query $i$ with key $j$ — every query-meets-key score computed in one shot: $Attention (Q, K, V) = softmax_{row} (\frac{Q K ^{⊤}}{d _{k}}) V$

Which axis does softmax run along? The subscript "row" is essential: softmax is applied across each row of the $n \times n$ score grid — i.e. for a fixed query $i$ , over all keys $j$ . That way each query's weights sum to 1 independently. Running it down columns instead would be a classic implementation bug (in code: softmax(dim=-1)).

WHY matrices? Doing every token's projection and every pair's score as one big multiply is exactly what GPUs are fast at. Same recipe, applied to a whole page instead of one token. This packing is what Multi-Head Attention then repeats in parallel, and the identical machinery serves Cross-Attention (queries from one sequence, keys/values from another).

PICTURE. Input grid $X$ ( $n$ rows) fanning into three output grids $Q$ , $K$ , $V$ , each row color-matched back to its source token.

The one-picture summary

Here is the whole derivation compressed into a single diagram. Read it left to right:

Far left, the butter circle labelled $x$ is one token's embedding — the four dials of Steps 1–2.
Three colored arrows fan out from $x$ , one per learned grid. The lavender arrow carries $x$ through $W^{Q}$ into the query $q$ ("what I want", Step 3); the coral arrow through $W^{K}$ into the key $k$ ("what I offer", Step 4); the mint arrow through $W^{V}$ into the value $v$ ("info I'll hand over", Step 8). One token, three roles — the central lesson.
Top middle, "dot → relevance": the $q$ and $k$ circles both feed the dot-product node, producing one relevance number (Step 5), scaled by $\frac{1}{d _{k}}$ (Step 6).
Right of that, "softmax → weights (sum 1)": relevance numbers become attention percentages that add to 1 (Step 7).
Bottom right, "weighted $v$ = OUTPUT": those weights multiply the value $v$ and sum up into the token's new, context-aware vector (the $\sum_{j} α_{j} v_{j}$ of Step 8).

Trace any path and you have re-derived attention: want and offer meet to make relevance, relevance becomes weights, weights pull in values.

Recall Feynman retelling — say it back in plain words

Imagine every token in a sentence is a person in a room (a "token" is a small chunk of text — sometimes a whole word like cat, sometimes just a word-piece like un or ness). Each person makes three cards from their own single ID:

a want-card (query): "I'm looking for a verb."
an offer-card (key): "I'm a noun, I'm the subject."
an info-card (value): the actual gossip they'll share.

To decide who listens to whom, everyone holds up their want-card next to everyone else's offer-card and checks how well the arrows line up — that lining-up is the dot product, and a bigger overlap means "more relevant." We shrink those overlap numbers a bit (divide by $d_{k}$ ) so nobody shouts over everyone else and the learning doesn't freeze. Then softmax turns each person's overlaps into listening percentages that add up to 100%. Finally each person collects everyone's info-cards, weighted by those percentages, and blends them into a new, context-aware understanding of themselves. Three cards from one ID — because what you want, how you're found, and what you say are three different jobs.

Recall Quick checks

What is the difference between a "word" and a "token"? ::: A tokenizer chops text into tokens, which are often sub-word pieces; a common word may be one token, a rare word several. The model embeds and attends over tokens, not raw words. Why three separate matrices and not one? ::: Wanting, being-matched, and contributing information are different jobs; separate learned grids let attention be asymmetric and let the model match on one feature while delivering another. What does $∣ q ∣$ mean? ::: The length (magnitude) of the arrow $q$ , computed as $q_{1}^{2} + q_{2}^{2}$ — a non-negative number ignoring direction. What does the dot product of $q$ and $k$ measure? ::: Directional agreement of the two arrows via $∣ q ∣∣ k ∣ cos θ$ — big positive when aligned, zero when perpendicular, negative when opposed. What does the transpose $K^{⊤}$ do and why is it needed? ::: It flips $K$ so rows become columns ( $n \times d_{k} \to d_{k} \times n$ ), making the shapes line up so $Q K^{⊤}$ gives every query-key dot product at once. Why divide by $d_{k}$ ? ::: Dot products grow with dimension; scaling keeps them in a range where softmax doesn't saturate and gradients keep flowing. Along which axis is softmax applied in $Q K^{⊤}$ ? ::: Along each row (over all keys for a fixed query), so every query's weights sum to 1. If $q = 0$ , what is the score against any key? ::: Zero — a token with no search intent matches nothing.