Query, key, value matrices: Question bank — AI-ML deep dive

This bank hunts the mental slips around query, key, value matrices — the ideas that feel right but hide a wrong assumption. No heavy arithmetic here (see the worked-examples pages for that). Each line is a question ::: answer reveal: cover the right side, commit to an answer out loud with a reason, then check.

Symbols and shapes you need first

Before any trap, pin down every symbol these questions use. Nothing below is used before it appears here.

The picture below shows one token embedding being fanned out by three different learned matrices into three different roles. Notice the red value vector — it can be a different length ( $d_{v}$ ) than the black query and key ( $d_{k}$ ).

True or false — justify

True or false:

Q

K

, and

V

are three different slices of the input matrix

X

False. They are three separate linear projections

X W^{Q}

X W^{K}

X W^{V}

of the same full

X

— every projection sees all

d_{m o d e l}

dimensions, nothing is sliced off.

True or false: since

Q = X W^{Q}

K = X W^{K}

V = X W^{V}

all come from the same

X

, the three matrices carry identical information.

False. The learned weights

W^{Q}, W^{K}, W^{V}

are different, so each projection reshapes

X

into a different subspace — matching-intent, matching-offer, and content — that emphasise different features.

True or false: the weight matrices

W^{Q}, W^{K}, W^{V}

are fixed by the architecture designer.

False. Only their shapes are fixed. The entries start random and are learned by backpropagation; a translation model tunes them differently than a summariser.

True or false: swapping

W^{Q}

and

W^{K}

leaves the attention output unchanged.

False. The score is

q_{i} \cdot k_{j} = (W^{Q} x_{i})^{T} (W^{K} x_{j})

, which is asymmetric — swapping the roles makes token

i

query with

j

's intent, changing every relevance score.

True or false: for self-attention,

Q

K

, and

V

all come from the same sequence.

True. Self-attention feeds one sequence into all three projections; cross-attention is where

Q

comes from one sequence and

K, V

from another.

True or false:

d_{k}

and

d_{v}

must be equal.

False.

d_{k}

sizes the matching space (queries and keys must share it to take a dot product);

d_{v}

sizes the content space and is free to differ. They are often set equal only for convenience.

True or false: the dot product

q_{i} \cdot k_{j}

requires

q_{i}

and

k_{j}

to live in the same-dimension space.

True. A dot product pairs coordinates one-to-one, so queries and keys must both be

d_{k}

-dimensional — this is exactly why

W^{Q}

and

W^{K}

share output size

d_{k}

True or false: making

d_{k}

larger always improves attention quality.

False. Larger

d_{k}

inflates dot-product magnitudes, pushing softmax toward saturation and killing gradients — the reason scaled dot-product attention divides by

d_{k}

True or false: the value vector

v_{j}

decides whether token

j

is attended to.

False. Whether-to-attend is decided by the query–key score;

v_{j}

only decides what information token

j

contributes once it has been selected.

Spot the error

"Attention is symmetric: if

i

attends strongly to

j

, then

j

attends strongly to

i

." — find the flaw.

The score uses

i

's query with

j

's key, while

j \to i

uses

j

's query with

i

's key — different vectors from different matrices, so scores are generally unequal.

"We can drop

W^{V}

and just use the raw embedding

x_{j}

as the value." — what's wrong?

Then the content space is forced to equal the raw semantic space, removing the model's freedom to separate "what made

j

relevant" from "what information

j

should pass on."

W^{V}

exists precisely to decouple these.

W^{Q} \in R^{d_{m o d e l} \times d_{k}}

, so

Q = W^{Q} X

." — spot the error.

The multiplication order is wrong. With rows of

X

being tokens,

Q = X W^{Q}

; writing

W^{Q} X

has mismatched inner dimensions and would mix tokens instead of projecting each independently.

"Since softmax normalises the scores, scaling by

1/ d_{k}

is redundant." — why is this wrong?

Softmax normalises the sum, not the spread. Large raw scores make softmax nearly one-hot with vanishing gradients; the scaling controls the spread before softmax, which normalisation cannot fix afterward.

"Because the same

X

feeds all three, we only need to learn one matrix and reuse it for Q, K, V." — the flaw?

Reusing one matrix collapses query, key, and value into the same space, destroying the asymmetry and the query-vs-content decoupling that make attention expressive.

"The attention output for token

i

is a weighted sum of the keys

k_{j}

." — correct it.

It is a weighted sum of the values

v_{j}

output_{i} = \sum_{j} α_{i, j} v_{j}

. Keys only produce the weights

α

; values carry the information.

Why questions

Why do we need a separate key projection at all instead of matching queries directly against raw embeddings?

Raw embeddings mix meaning, position and syntax on the same axes; the key projection carves out a subspace where "should I be attended to" is cleanly measurable, independent of raw semantics.

Why is the dot product a sensible measure of relevance rather than, say, elementwise difference?

The dot product measures alignment of direction in the

d_{k}

matching space via

q_{i} \cdot k_{j} = ∣ q_{i} ∣∣ k_{j} ∣ cos θ

— large-positive means the arrows point the same way, which is exactly "this key answers my query."

Why can

d_{k}

be smaller than

d_{m o d e l}

without losing the ability to attend?

Attention only needs a subspace rich enough to compare relevance, not to store all meaning. A smaller

d_{k}

also cuts compute and reduces overfitting, since matching needs fewer degrees of freedom than full semantics.

Why is it useful that "cat" can query for verbs (Q–K space) yet supply noun features (V space)?

Because the feature that makes a token relevant differs from the feature it should deliver. Separate

W^{K}

and

W^{V}

let one token match on grammatical role and contribute semantic content — impossible with a single projection.

Why are the same linear layers applied to every token position identically?

The projection is content-based, not position-based; applying the same

W

to each row keeps the operation permutation-consistent, and any position information must already be injected via positional encoding into

X

Edge cases

What is a token's self-relevance score

score_{i, i}

, and is it always the largest?

It is

q_{i} \cdot k_{i}

using

i

's own query and key. Because

W^{Q} \neq = W^{K}

, this is not guaranteed to be the maximum — a token can attend more strongly to a different token than to itself.

W^{Q} = 0

(all-zero query weights), what happens to attention?

Every

q_{i}

is the zero vector, so all scores are zero, softmax outputs a uniform distribution, and each output becomes the plain average of all values — attention degenerates into mean-pooling.

For a sequence of length

n = 1

(a single token), what does attention compute?

The lone token attends only to itself; softmax over one score is

1

, so the output is exactly its own value vector

v_{1}

, regardless of the score value.

If two tokens have identical embeddings, must their query vectors be identical?

Yes. The projection

q = x W^{Q}

is a deterministic function of

x

, so equal embeddings give equal queries — any later difference in attention must come from differing positions encoded into

X

beforehand.

What happens to the softmax weights if all relevance scores for token

i

are equal?

Softmax returns a uniform distribution

α_{i, j} = 1/ n

, and the output is the unweighted average of all value vectors — attention provides no selectivity in this degenerate case.

In multi-head attention, why can one head learn a

W^{Q}

focused on syntax while another focuses on coreference?

Each head owns independent

W^{Q}, W^{K}, W^{V}

with their own

d_{k}

-slice, so they are optimised separately and can specialise on different relational patterns simultaneously.

Fast self-test — trace each trap to its root

Recall The one assumption behind most traps

Symmetry / sameness illusion ::: Treating $Q, K, V$ as the same object ignores that they are three independent learned projections — see "True or false" items 1–2.

Recall The directionality trap

Why attention is not symmetric ::: $i \to j$ uses $q_{i} \cdot k_{j}$ ; $j \to i$ uses $q_{j} \cdot k_{i}$ — different vectors — see "Spot the error" item 1 and "Edge cases" self-relevance.

Recall The dimension trap

Why $d_{k} \neq = d_{v}$ never breaks ::: $d_{k}$ vanishes inside the dot product, $d_{v}$ only sets output width — see the tiny worked-shapes example.

Recall The softmax trap

What scaling by $d_{k}$ actually fixes ::: The spread of scores before softmax, not the sum — see "Spot the error" item 4.