Question bank — Scaled dot-product attention

False. Scaling multiplies every logit in a row by the same constant $\frac{1}{\sqrt{d_k}}$, so their order is preserved — the largest logit stays largest. It changes how sharp the distribution is, not who wins.

True by construction: dividing each $e^{z_i}$ by the row sum ${\sum }_j{e}^{z_j}$ forces the row to add to 1. That is why each row is read as a distribution over keys.

True — identical keys give identical logits against any query, and softmax maps equal logits to equal weights. The value vectors behind them can still differ, so the outputs they contribute differ.

False. $\frac{1}{\sqrt{d_k}}$ is a fixed constant chosen from the variance argument; nothing about it is learned. Only $W_Q,W_K,W_V$ are learned.

False. Larger $d_k$ adds parameters and widens the variance of dot products, inviting overfitting and more extreme outliers. Transformers prefer many heads with small $d_k$ over one head with huge $d_k$.

True. The output is $AV$ where $A$ is $n\times m$ and $V$ is $m\times d_v$, giving $n\times d_v$ — the key/query dimension $d_k$ has already been "summed away" inside $Q{K}^T$.

False. High precision stops overflow, but the real damage is gradient quality: large logits still push softmax into saturated flat regions where learning stalls. See Gradient Descent.

False. They all come from the same input $X$, but through different weight matrices: $Q=X{W}_Q$, $K=X{W}_K$, $V=X{W}_V$. So a token can ask a different question than it answers.

False. Zero means the vectors are orthogonal (perpendicular), a neutral "unrelated" score. Opposite directions give a large negative dot product.

True. $e^{z_i+c}=e^c{e}^{z_i}$, and the $e^c$ cancels between numerator and row-sum. This is why we can subtract the row max for numerical stability without changing the result.

The variance is $d_k$ but the standard deviation is $\sqrt{d_k}$, and we scale by standard deviation to bring the spread back to 1. Dividing by $d_k$ over-shrinks the logits, flattening them toward a uniform distribution.

Wrong shape. $K^T$ is $d_k\times m$, so $Q{K}^T$ is $n\times m$ — a score for every query–key pair. The $d_k$ dimension is contracted away by the dot product.

Softmax is applied row-wise: each query gets its own distribution over the $m$ keys. A single global softmax would let unrelated queries compete against each other.

$Q{K}^T$ and $K{Q}^T$ are transposes of each other, not equal, and they use different learned weights $W_Q\ne W_K$. The query/key roles are not interchangeable even in Self-Attention.

It sets forbidden logits to $-\infty$ before softmax, so $e^{-\infty }=0$ and the row still renormalises to sum to 1. Zeroing after softmax would break the distribution (rows would no longer sum to 1).

They are linked: multi-head attention splits the model dimension into smaller per-head $d_k$, and each head still scales by its own $\sqrt{d_k}$. See Multi-Head Attention.

The dot product is cheap ($O(d_k)$), differentiable, and directly measures directional alignment in embedding space — large when vectors point the same way. It plugs straight into fast matrix multiplication ($Q{K}^T$).

A saturated softmax output $\approx$ one-hot has gradient ${\sigma }_i(1-{\sigma }_i)\approx 0$, so backprop sends almost no signal to $Q$ and $K$. The model cannot tell that one key is slightly better than another.

It is a sum of $d_k$ independent zero-mean products, each contributing variance 1, and independent variances add. More terms means more accumulated spread, hence variance $=d_k$.

GPUs/TPUs are optimised for dense matrix multiplication, so batching every query into $Q{K}^T$ is dramatically faster than one-at-a-time loops — same maths, hardware-friendly form.

Keys are the "address labels" used only to score relevance; values are the actual content to retrieve. The output is a soft-blended answer, so we mix the values weighted by how relevant each key was.

Only the sources of $Q$, $K$, $V$ change: Self-Attention draws them from one sequence, Cross-Attention draws $Q$ from the decoder and $K,V$ from the encoder. The scaled-dot-product computation is identical.

Attention weights depend only on content compatibility, so permuting the tokens permutes the outputs identically — order is invisible. Positional Encoding injects order so "dog bites man" differs from "man bites dog".

Equal logits give a uniform distribution: each of the $m$ keys gets weight $1/m$. The output is then a plain average of all value vectors.

Every logit becomes $-\infty$, so softmax is $0/0$ — undefined. In practice such rows must never occur (a query always sees at least one key, e.g. itself in causal masking).

Softmax of a single logit is always $1$, so the output equals that one value vector regardless of the score. There is no competition when there is nothing to compete against.

The logit spread grows without bound, so softmax collapses toward a one-hot vector on the single largest score — near-total saturation. Scaling by $\sqrt{d_k}$ is precisely what prevents this collapse.

Its dot product with every key is 0, so all logits are equal and softmax gives a uniform average of the values — the query expresses no preference. This is the degenerate "no question asked" case.

Identical queries produce identical logit rows, identical softmax weights, and therefore identical output rows. Two tokens asking the same question retrieve the same blended answer.

$e^{\text{large negative}}\to 0$, so that key receives essentially zero attention weight and its value barely enters the output. This is exactly how masking drives weights to 0.