Self-attention mechanism in detail: Question bank

This bank exists to catch the conceptual misunderstandings that the parent note can accidentally leave behind. Every item is a one-line reveal: read the prompt, answer it in your head with a reason, then check. If your reason was "I just remember the answer", that item still owns you — reread the justification.

Before we start, three words we lean on constantly. We define them once here, in plain language, so no reveal below uses them cold.

True or false — justify

Softmax attention weights in one row always sum to exactly 1

True. Softmax divides each exponentiated score by the sum of all exponentiated scores in that row, so by construction the row is a probability distribution summing to 1.

Self-attention needs an external query from a decoder, like the older attention did

False. The "self" means the sequence generates its own queries from itself (

Q = X W^{Q}

); no outside sequence is required.

Because attention is a weighted average of value vectors, an output can point in a direction no single value points in

True. A convex combination of vectors can land anywhere inside their convex hull, which generally includes directions none of the originals individually has.

The scaling factor

d_{k}

changes which position gets the most attention

False. Dividing every score in a row by the same constant does not change their order; it only rescales them so softmax is less sharply peaked, which affects gradients, not the argmax.

Self-attention is permutation-equivariant: shuffle the input rows and the outputs shuffle the same way

True. Nothing in

softmax (Q K^{T} / d_{k}) V

references position order, which is exactly why 4.1.04-Positional-encoding must be added separately.

Removing softmax and using raw scores as weights would still give a valid attention mechanism

False in practice. Raw scores can be negative and don't sum to 1, so "weights" could be negative or blow up; softmax guarantees positive, normalized gating.

Setting

W^{Q} = W^{K}

(tied query and key matrices) forces the score matrix to be symmetric

True. Then

Q K^{T} = X W^{Q} (W^{Q})^{T} X^{T}

, and

M = W^{Q} (W^{Q})^{T}

is symmetric, so

X M X^{T}

is symmetric — position

i

's score on

j

equals

j

's on

i

A larger dot product between a query and a key always means the two words are semantically similar

False. The dot product is taken in the projected

Q

K

space, not embedding space; the learned projections decide what "high score" means, and it often encodes a syntactic relation (verb→subject), not similarity.

Spot the error

"We use

Q K^{T}

because we want to compare every value against every value."

Error: we compare every query against every key, not values. Values are only summoned afterward, weighted by the query–key scores.

d_{k}

is there to normalize the output vectors to unit length."

Error: it normalizes the variance of the scores before softmax. The output magnitude comes from the value vectors and weights, and is not forced to unit length.

"Softmax makes attention differentiable, so we can also just replace it with argmax for the same training."

Error: argmax is not differentiable, so gradients can't flow through the selection. Softmax's smoothness is precisely what lets backprop reach

W^{Q}

and

W^{K}

"Self-attention avoids vanishing gradients because it has fewer layers than an RNN."

Error: the depth is not the point. It avoids the RNN's long sequential gradient chain — any two positions are one attention hop apart, so gradients don't decay over the sequence length. See 5.3.03-Gradient-flow-in-deep-networks.

"Because

Q

K

V

come from the same

X

, they are identical matrices."

Error: they share the input but pass through different learned projections

W^{Q}, W^{K}, W^{V}

, so they differ. Same source, different lenses.

"Attention complexity is

O (n)

because each word does one lookup."

Error: each word scores against all

n

words, giving an

n \times n

score matrix, so it's

O (n^{2} d)

. See 4.2.02-Computational-complexity-of-transformers.

"The transpose in

Q K^{T}

is just a formatting trick with no meaning."

Error: the transpose aligns the shared dimension

d_{k}

so that

(n \times d_{k}) (d_{k} \times n)

contracts over the feature axis — that contraction is the dot product between each query and each key.

"If two rows of the attention matrix are equal, the two words are interchangeable in the sentence."

Error: equal rows mean the two words attend to others identically, but their value contributions and their roles as keys can still differ, so they are not interchangeable.

Why questions

Why project into

Q

K

V

at all instead of dotting raw embeddings together?

Raw embeddings encode meaning, not "relevance"; the learned projections carve out a subspace where the dot product measures how much position i should listen to j, which is a different question than "are these words similar".

Why does the variance of a dot product grow with

d_{k}

The dot product sums

d_{k}

independent mean-zero product terms; variances of independent terms add, so total variance is

d_{k}

and standard deviation is

d_{k}

— hence dividing by

d_{k}

restores unit variance.

Why is softmax preferred over simply dividing scores by their sum?

Plain division breaks on negative scores and treats a score of 3 and 30 nearly linearly; the exponential is always positive and amplifies gaps, letting the model sharply gate irrelevant positions off.

Why can self-attention be computed in parallel while an RNN cannot?

Every output depends only on the fixed input

X

through matrix multiplies, with no step depending on a previous step's result; an RNN's hidden state at time

t

needs the state at

t - 1

, forcing serial computation.

Why does each word need to be a query, key, and value simultaneously?

To understand itself it must ask (query), to help others it must advertise (key), and once chosen it must contribute (value) — three distinct jobs the single word performs at once, unlike older encoder–decoder attention where these were split across sequences. Compare 3.2.05-Attention-mechanism-basics.

Edge cases

What happens to the output when all scores in a row are equal (e.g. all zero)?

Softmax gives a uniform distribution (

1/ n

each), so the output is the plain average of all value vectors — the position "listens equally" to everyone, learning nothing selective.

For a sequence of length

n = 1

, what does attention compute?

The single row softmaxes to

[1]

, so the output equals that one token's own value vector — self-attention collapses to an identity-like pass on the value.

If one score in a row is far larger than the rest, what does softmax approach?

A near one-hot vector, so the output nearly copies a single value vector; this is the "sharply peaked" regime the

d_{k}

scaling was introduced to soften.

What does the diagonal of the attention matrix represent, and can it be the largest entry?

The diagonal

α_{ii}

is how much a position attends to itself; it can absolutely be the row maximum (a word often finds itself most relevant), and nothing forbids it.

d_{k}

were set to 1, does the scaling term still do anything?

1 = 1

, so division by

d_{k}

is a no-op; with a single feature the dot product is already unit-variance, so scaling is only meaningful once

d_{k} > 1

What if two different words produce identical key vectors?

Any query gives them the same score, so they receive equal attention weight and are indistinguishable as keys — the model can only tell them apart through their (possibly different) value vectors.

Recall Self-check before you leave

The three roles of a single token ::: query (asks), key (advertises), value (contributes content) The one thing $d_{k}$ changes ::: the sharpness/variance of scores before softmax, protecting gradients — never the ranking The reason positional encoding is needed ::: raw self-attention is permutation-equivariant and ignores order

Related: 4.1.01-Transformer-architecture-overview · 4.1.03-Multi-head-attention · 4.1.02 Self-attention mechanism in detail (Hinglish)