Worked examples — Query, key, value matrices

This page is a companion to the parent topic. We already know the three recipes $Q=X{W}^Q,K=X{W}^K,V=X{W}^V.$ Here we stress-test them. We will feed the machinery tricky inputs — all-positive tokens, mixed signs, a token of all zeros, two identical tokens, orthogonal projections — so that when the exam (or real code) throws one at you, you have already seen it.

Before anything, plain-word reminders so no symbol is unearned:

In the picture the teal key points nearly the same way as the orange query, so $q\cdot k$ is large and positive (high relevance). The plum key points off to the upper left, away from the query, so its score is small or negative (low relevance). This is the geometric fact every example below relies on.

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The examples below are labelled with the cell they cover.

Cell	Scenario	What could break / what to notice
A	All-positive token, normal $W$	Baseline projection, no sign traps
B	Mixed-sign token & weights	Cancellation — terms fight each other
C	Zero token $x=0$ (degenerate / padding)	Every projection is $0$; score $=0$
D	Two identical tokens	Self-score vs cross-score behaviour
E	Orthogonal query & key	Dot product exactly $0$ — the "right-angle" case
F	Scaling limit: score before vs after $\frac{1}{\sqrt{d_k}}$	Why we divide, saturation of softmax
G	Full softmax over a row (word problem)	Turning raw scores into weights that sum to 1
H	Exam twist: $d_v\ne d_k$, output vector shape	Values live in a different-size space

---

Example 1 — Cell A: the clean baseline

Forecast: guess — will $q$ and $k$ look similar? (They come from the same $x$ but different lenses, so no.)

1. Query, first output slot. Multiply $x$ against column 1 of $W^Q$ and add: $0.5(0.1)+(-0.3)(0.2)+0.8(-0.3)+0.2(0.6)=0.05-0.06-0.24+0.12=-\mathrm{0.13.}$ Why this step? A projection's each output number is a dot product of the token with one column of the weight matrix. Column 1 is the recipe for the first new coordinate.
2. Query, second slot (column 2): $0.5(0.4)+(-0.3)(-0.1)+0.8(0.5)+0.2(0.2)=0.20+0.03+0.40+0.04=\mathrm{0.67.}$ Why this step? The second coordinate uses column 2 as its recipe; we repeat the same token-times-column dot product so that both slots of $q$ are filled. So $q=[-0.13,0.67]$.
3. Key, both slots the same way with $W^K$: $k_1=0.5(-0.2)+(-0.3)(0.5)+0.8(0.4)+0.2(-0.1)=0.05,$ $k_2=0.5(0.3)+(-0.3)(0.1)+0.8(-0.2)+0.2(0.6)=\mathrm{0.08.}$ So $k=[0.05,0.08]$.

Why they differ: same word, two lenses. $q$ says "what I'm hunting for," $k$ says "what I advertise." Different matrices, different arrows.

Verify: $q$ has one big component ($0.67$) driven mostly by the third feature ($0.8\times 0.5$). $k$ is small in both slots because its columns partly cancel. Shapes: $[1\times 4]\times [4\times 2]=[1\times 2]$. ✓

---

Example 2 — Cell B: mixed signs and cancellation

Forecast: with big positive and negative weights, guess whether the result is large or near zero.

1. Slot 1: $0.5(0.4)+(-0.3)(0.5)+0.8(-0.5)+0.2(0.4)=0.20-0.15-0.40+0.08=-\mathrm{0.27.}$ Why this step? Same dot-product recipe. Notice the third term $-0.40$ dominates and drags the sum negative.
2. Slot 2: $0.5(-0.4)+(-0.3)(0.5)+0.8(-0.5)+0.2(0.4)=-0.20-0.15-0.40+0.08=-\mathrm{0.67.}$ So $k=[-0.27,-0.67]$.

Lesson: mixed signs mean features can cancel or reinforce. Here the big positive feature $0.8$ hits a negative weight $-0.5$ and pulls everything down. There is no error — negative key components are normal; they just mean "this arrow points into the negative region."

Verify: flip the sign of the whole third column input intuition — the third feature contributes $0.8\times (-0.5)=-0.40$ to both slots, and both slots are indeed negative. ✓

---

Example 3 — Cell C: the zero (padding) token

Forecast: guess before computing — does a zero token attend to anything?

1. Every projection is zero. Each output slot is $\sum (\text{0})(\text{weight})=0$. So $q=[0,0]$ and $k=[0,0]$ (length $d_k=2$), and $v=[0,0,0]$ (length $d_v=3$). Why this step? Multiplying the zero vector by any matrix gives zero — the arrow has no length, so no lens can stretch it into something.
2. Self-score: $q\cdot k=0(0)+0(0)=0$.

This is exactly why real transformers use a mask: a genuine all-zero score would still get a nonzero softmax weight (softmax of zeros is uniform), so we must additively force padding scores to $-\infty$. The degenerate input reveals why masking exists.

Verify: $[0,0,0,0]\cdot (\text{any column})=0$ for all four columns; product is the zero vector, dot product $0$. ✓

---

Example 4 — Cell D: two identical tokens

Forecast: identical tokens — will the two scores be equal?

1. Query of token 1 (columns of $W^Q$, only slots 1 and 3 survive since $x=[1,0,1,0]$): $q_1=[0.1+(-0.3),0.4+0.5]=[-0.2,0.9].$ Why this step? Zeros in $x$ kill those rows of $W^Q$; only the "1" features contribute.
2. Key of token 2 (same input, $W^K$): $k_2=[-0.2+0.4,0.3+(-0.2)]=[0.2,0.1].$ Since $x_2=x_1$, $k_1=k_2=[0.2,0.1]$.
3. Scores: $q_1\cdot k_1=(-0.2)(0.2)+(0.9)(0.1)=-0.04+0.09=0.05,$ $q_1\cdot k_2=\text{same numbers}=\mathrm{0.05.}$

Lesson: when tokens are identical, self- and cross-scores are equal — the query of one lands on the same key arrow. Attention here cannot distinguish position from content alone; that is precisely the job of Positional Encoding.

Verify: $k_1$ and $k_2$ use identical inputs, so they are the same vector; hence both dot products equal $0.05$. ✓

---

Example 5 — Cell E: orthogonal query and key (score = 0)

Forecast: these arrows look "different." Guess the sign of the score.

1. Dot product: $q\cdot k=3(4)+4(-3)=12-12=0.$ Why this step? Score is always the dot product of the query and key arrows.
2. Geometry: a score of exactly $0$ means the two arrows are at a right angle (perpendicular / orthogonal). Neither aligned nor opposed — pure neutrality.

The figure below draws exactly this pair. Notice the little square at the origin marking the $90^{\circ }$ corner between the orange query $q=[3,4]$ and the teal key $k=[4,-3]$: when the corner is a perfect right angle, the score collapses to $0$.

This is the boundary case between positive (relevant) and negative (anti-relevant) scores. After softmax it still gets a finite weight, but a smaller one than any positive-scoring token in the row.

Verify: $3\cdot 4+4\cdot (-3)=0$; lengths are $|q|=|k|=5$, so the angle satisfies $\cos \theta =0/(5\cdot 5)=0\Rightarrow \theta =90^{\circ }$. ✓

---

Example 6 — Cell F: why we scale by $1/\sqrt{d_k}$

Forecast: which raw score threatens softmax more — the small one or the big one?

1. Scale factor: $\frac{1}{\sqrt{d_k}}$. For $d_k=2$: $\frac{1}{\sqrt{2}}\approx 0.7071$. For $d_k=64$: $\frac{1}{\sqrt{64}}=\frac{1}{8}=0.125$. Why this step? Dot products of $d_k$ random-ish terms grow roughly like $\sqrt{d_k}$ in magnitude; dividing by $\sqrt{d_k}$ cancels that growth so scores stay in a tame range.
2. Scaled scores: $0.047\times 0.7071=0.03324,8.0\times 0.125=\mathrm{1.0.}$
3. Danger without scaling: a raw $8.0$ fed into softmax against neighbours near $0$ gives a weight near $1$ for one token and near $0$ for the rest — the gradient there is almost flat (saturation), so Backpropagation gets tiny updates. Scaling to $1.0$ keeps the softmax responsive.

Verify: $0.047/\sqrt{2}=0.03324$ and $8/\sqrt{64}=1.0$. ✓

---

Example 7 — Cell G: full softmax row (word problem)

Forecast: guess the ranking of the three weights before computing.

1. Scale: $s/2=[1.0,0.0,-0.5]$. Why this step? Same $1/\sqrt{d_k}$ scaling as Example 6, applied to the whole row of raw scores before softmax.
2. Exponentiate: $e^{1.0}=2.7183,e^{0.0}=1.0,e^{-0.5}=\mathrm{0.6065.}$ Why this step? Softmax (defined at the top of this page) first raises $e$ to each scaled score; exponentials are always positive and make bigger scores dominate smoothly.
3. Normalise by the sum $2.7183+1.0+0.6065=4.3248$: $\alpha =[\frac{2.7183}{4.3248},\frac{1.0}{4.3248},\frac{0.6065}{4.3248}]=[0.6286,0.2312,0.1402].$ Why this step? Dividing by the total is the second half of softmax — it forces the weights to sum to $1$ so they read as "fractions of attention."
4. Read it: "sat" attends most (63%) to "the", then "cat", least to itself.

Verify: weights sum to $0.6286+0.2312+0.1402=1.000$ (up to rounding), all positive, ordering matches score ordering. ✓

---

Example 8 — Cell H: exam twist, $d_v\ne d_k$ and output shape

Forecast: what length is the output — $d_k$, $d_v$, or $d_{model}$?

1. Weighted sum of values: $\text{out}=0.6v_1+0.4v_2.$ Why this step? The output of attention is a convex mix of value vectors — weights sum to 1, so it is a weighted average of the information each token offers.
2. Component-wise: ${\text{out}}_1=0.6(0.70)+0.4(0.10)=0.42+0.04=0.46,$ ${\text{out}}_2=0.6(-0.24)+0.4(0.50)=-0.144+0.20=0.056,$ ${\text{out}}_3=0.6(0.03)+0.4(-0.20)=0.018-0.08=-\mathrm{0.062.}$ So $\text{out}=[0.46,0.056,-0.062]$.
3. Shape: the output lives in $d_v=3$ space — not $d_k$, not $d_{model}$. Values, not keys, set the output size.

Twist resolved: the query/key space ($d_k$) only decides how much to blend; the value space ($d_v$) decides what gets blended, and therefore the output length. In Multi-Head Attention these per-head outputs are concatenated and re-projected back to $d_{model}$ by a final linear layer.

Verify: each output component lies between the corresponding min and max of $v_1,v_2$ (a convex average must); e.g. $0.46\in [0.10,0.70]$. Weights $0.6+0.4=1$. ✓

---

Recall Quick self-test

A padding token $x=0$ produces which query vector? ::: The zero vector — every projection of $0$ is $0$, so it must be masked out. An attention score of exactly $0$ means the query and key arrows are... ::: perpendicular (orthogonal, $90^{\circ }$) — neither aligned nor opposed. Why divide raw scores by $\sqrt{d_k}$? ::: dot products grow like $\sqrt{d_k}$; dividing keeps softmax out of its flat saturation region so gradients survive. The attention output vector has length equal to... ::: $d_v$ (the value dimension), because it is a weighted average of value vectors.

Related: Scaled Dot-Product Attention, Self-Attention, Cross-Attention, Embedding Layer, Query, key, value matrices.