Transformers — attention mechanism, self-attention: Question bank

This bank targets the concepts, not the arithmetic — for number-crunching go to the D3/D4 pages. Prerequisites worth revisiting when you stumble: Softmax function, Dot product & cosine similarity, Positional Encoding, Multi-Head Attention, RNNs and LSTMs, and Gradient vanishing/saturation.

True or false — justify

Self-attention processes tokens strictly one after another, like an RNN.

False — every position reads from every other position in a single matrix multiply

Q K^{⊤}

, so there is no left-to-right dependency and it fully parallelises. See RNNs and LSTMs for the sequential contrast.

Softmax turns the raw scores into weights that are all non-negative and sum to 1.

True — that is exactly its purpose:

e^{s}

is always positive and dividing by the row sum forces the weights in each row to add to 1, giving a valid weighted-average.

The attention weights

α_{ij}

are learned parameters updated by gradient descent.

False — they are computed on the fly from the current inputs; the learned parameters are

W_{Q}, W_{K}, W_{V}, W_{O}

. Same weights on different inputs give different

α_{ij}

Shuffling the order of the input tokens leaves each output token's content unchanged (just reordered).

True — plain self-attention is permutation-equivariant: it has no built-in notion of position, which is exactly why Positional Encoding must be added.

Dividing the scores by

d_{k}

changes which token gets the highest attention weight.

False — dividing every score in a row by the same constant does not reorder them; it only rescales, softening the softmax so gradients stay healthy. The argmax token is unchanged.

A query, key and value for the same token are always equal to each other.

False — they are three different projections

W_{Q} x_{i}, W_{K} x_{i}, W_{V} x_{i}

of the same input, so they generally differ; they'd only match if the weight matrices were identical.

In self-attention, Q, K and V all come from the same sequence.

True — that "self" is the whole point; in cross-attention (e.g. decoder reading encoder) the queries come from one sequence and keys/values from another.

The dot product

q_{i} \cdot k_{j}

is large exactly when the two vectors point in similar directions.

True — dot product grows with alignment (and magnitude), which is why it works as a "these two are relevant" score. See Dot product & cosine similarity.

Multi-head attention lets each head learn a completely different set of value contents from the input embeddings.

True — each head has its own

W_{Q}, W_{K}, W_{V}

, so it projects into its own subspace and can specialise (e.g. one head on short-range vibration, another on long-range trend). See Multi-Head Attention.

Because attention connects distant positions directly, information from step 1 is preserved when reaching step 100.

True — the direct connection means no repeated hidden-state overwriting, so there is no long-range dilution like an RNN suffers.

Spot the error

"We apply softmax down the value/feature axis so each feature sums to 1."

Error: softmax is applied across keys/positions (row-wise on

Q K^{⊤}

), so the weights over which tokens to read sum to 1 — never over feature dimensions.

Q K^{⊤}

has shape

n \times d_{k}

, one score per feature."

Error:

Q K^{⊤}

has shape

n \times n

— one score for every (query token, key token) pair. It is a token-by-token relevance matrix, not a per-feature array.

"Larger

d_{k}

is always better since more dimensions means more capacity, so scaling is optional."

Error: without

/ d_{k}

the dot product's variance grows with

d_{k}

, pushing softmax into a near one-hot regime where gradients to other tokens vanish. Scaling is what makes large

d_{k}

usable. See Gradient vanishing/saturation.

"Since self-attention sees the whole sequence at once, it inherently knows timestep order."

Error: seeing all tokens is not the same as knowing their order — attention is permutation-equivariant. Order must be injected via Positional Encoding.

"The output

z_{i}

is a weighted sum of the keys, weighted by query-key match."

Error: the blend is over values

v_{j}

, not keys. Keys only produce the match scores; values carry the content that gets mixed.

"We scale by

d_{k}

(not

d_{k}

) because the score variance grows like

d_{k}

Error: variance grows like

d_{k}

, so the standard deviation grows like

d_{k}

. To renormalise the spread back to

\approx 1

we divide by the standard deviation

d_{k}

, not the variance.

"Each attention head must use the full model dimension

d_{model}

to keep enough capacity."

Error: each head uses the reduced dimension

d_{k} = d_{model} / h

, so

h

heads together cost about the same as one full-size attention — the savings are what make many heads affordable.

Why questions

Why use the dot product to measure relevance instead of, say, Euclidean distance between

q_{i}

and

k_{j}

The dot product directly rewards directional agreement and scales with magnitude, is a single cheap matrix multiply for all pairs at once, and is smoothly differentiable — perfect as an alignment score feeding softmax. See Dot product & cosine similarity.

Why softmax and not just dividing each score by the sum of scores?

Raw scores can be negative, so plain normalisation could give negative or ill-defined weights; the

e^{(\cdot)}

in softmax forces positivity and amplifies the largest score smoothly and differentiably. See Softmax function.

Why does the scaling factor prevent vanishing gradients specifically?

A large-variance score makes softmax nearly one-hot; at a saturated softmax the derivative

α (1 - α) \to 0

, so gradients to the non-selected tokens die. Keeping variance

\approx 1

keeps softmax in its responsive region. See Gradient vanishing/saturation.

Why split attention into multiple heads rather than one big attention?

One softmax can only form a single relevance pattern per query; multiple heads let the model attend to several kinds of relationships simultaneously (different subspaces), then concatenate them for a richer representation. See Multi-Head Attention.

Why is attention better than an RNN for a 200-step aerospace telemetry sequence?

Any timestep can attend directly to any earlier one (e.g.

t = 180

reading the pressure drop at

t = 120

) with no information loss through a chain of hidden states, and all positions compute in parallel. See Time-series anomaly detection (aerospace telemetry).

Why do we need positional encodings if the tokens already appear in order in the input matrix?

Attention treats the rows as an unordered set — the ordering in the matrix is invisible to the math — so position must be encoded into the vectors themselves to be usable. See Positional Encoding.

Why are Q, K, V three separate projections instead of using the raw input for all three?

Separate learned matrices let a token advertise (key), search (query), and deliver (value) using different representations, so what it looks for need not match what it contributes — far more expressive than reusing one vector.

Edge cases

If all keys are identical, what does each row of the attention weights look like?

All scores in a row become equal, so softmax gives a uniform distribution — every token contributes equally and the output is just the plain average of the values.

What happens to

z_{i}

if one score is enormously larger than all others?

Softmax collapses toward one-hot, so

z_{i}

becomes essentially that single value

v_{j}

— a hard lookup instead of a soft blend, and gradients to the other tokens nearly vanish.

For a sequence of length 1, what does self-attention output?

The single token attends only to itself: the one score softmaxes to weight 1, so the output is exactly its own value vector

v_{1}

(a no-op blend).

If two tokens are identical inputs but sit at different positions, will they get identical outputs?

Without positional encoding, yes — identical inputs give identical Q/K/V and identical attention, so their outputs match. Adding position information is what breaks this tie. See Positional Encoding.

What is the effect of setting all scores to zero (e.g. orthogonal q and k everywhere)?

Every score is 0, softmax of equal values is uniform, so the output is the unweighted mean of all values — attention degenerates into a plain averaging layer.

In a decoder using causal masking, why do future positions get score

- \infty

before softmax?

e^{- \infty} = 0

, so those positions receive exactly zero weight, preventing a token from attending to tokens that come after it — essential for left-to-right generation.

d_{k} \to

very large without scaling, what limiting behaviour do the attention weights approach?

The score variance blows up, so softmax approaches one-hot for almost every row — attention becomes a near-deterministic hard lookup with dead gradients, which is exactly the failure the

d_{k}

scaling prevents.

Recall Quick self-check

Cover every answer above and rate yourself: any item where your justification was a bare yes/no is a gap. The recurring themes are: softmax is over positions, weights are computed not learned, attention is order-blind, values (not keys) get blended, and scaling exists to tame variance. Nail those five and most traps disappear.