Question bank — Masked attention for autoregression

True. Position $i$ may attend to $j\le i$, so the allowed entries form the lower triangle including the diagonal; the masked (blocked) entries $j>i$ are the strict upper triangle.

False. We fill them with $==-\infty ==$ so that $\text{softmax}$ maps them to exactly $0$; filling with $0$ would leave $e^0=1$ of weight on future tokens.

False. Masking still computes all rows in parallel in one matrix operation; it only forbids certain columns. RNNs are forced to step sequentially — masking keeps the transformer's parallelism.

True. The diagonal $M_{ii}=0$ is left open — self-attention gives position $i$ its own content and positional signal, which it needs to predict the next token.

False, and that's the danger. Loss would look better because the model copies the visible answer — but the skill is useless at inference where the answer doesn't exist yet.

False. Every token still exists and still has a $Q$, $K$, $V$. The mask only zeroes certain attention weights, not the tokens themselves.

True. The causal pattern depends only on positions ($j>i$), not on content, so one $T\times T$ triangular mask is reused. (A separate padding mask handles variable real lengths.)

False. Post-softmax zeroing breaks the sum-to-one property; the surviving weights no longer form a probability distribution unless you renormalize — which is exactly what the pre-softmax $-\infty$ does for free.

Error: every row still sums to $==1==$. Softmax is computed only over the unmasked entries, so those $i$ weights renormalize among themselves — the masked columns contribute $0$, not a shrunken total.

Error: we predict $y_{i+1}$ from position $i$, so position $i$'s own representation is legitimately available. Only strictly-future positions $j>i$ leak the answer. Masking the diagonal would blind a token to itself.

Error: diagonal=0 includes the main diagonal, so it would mask a token from attending to itself. You need diagonal=1 (strictly-upper) to block only $j>i$.

Error: with KV Caching you still batch a growing key/value set, and any parallel-scored context must respect causality. More importantly the mask must exist during training so the model learns the constraint it faces at inference.

Half-error: $-10^9$ works numerically and is common, but it is an approximation. In edge cases (all-masked rows, low precision) it can produce tiny nonzero leakage or NaNs, so it must be handled carefully.

The ordering is right (mask after scaling) but the reason is wrong: $-\infty /\sqrt{d_k}$ is still $-\infty$. We add the mask after scaling simply so the finite scores are already stabilized before the softmax sees them.

Error: an encoder uses no causal mask at all — every position sees every other. Causal masking is the decoder's defining restriction, not a doubled version of the encoder.

The future contains the very tokens we must predict — seeing them is leakage. The past is legitimate context that is genuinely available at inference, so attending backward mirrors the real generation setting.

We want exactly zero attention on future tokens, independent of scores. Only $-\infty$ guarantees $e^{s+M}=0$ for every possible score; a learned value could accidentally let some future weight survive.

The mask is a single additive matrix applied to a full $T\times T$ score matrix computed at once. There is no sequential dependency in the computation — only in what information each row is allowed to use. RNNs must physically pass a hidden state step by step.

Because position $i$'s output depends only on keys/values at positions $\le i$, past $K,V$ vectors never change when new tokens arrive — so they can be stored once and reused, saving recomputation.

If training let the model peek ahead, it would learn a copy shortcut absent at inference, producing a train–test mismatch that collapses generation quality. The mask forces training to face the same information limits as generation.

Positional encoding only labels order; it doesn't restrict the attention weights. Without the mask, a token could still attend with full weight to a later position it can identify as "future."

Only column $1$ is unmasked, so $\text{softmax}$ over a single value gives weight $1.0$ on itself — the first token can attend to nothing but itself, which is correct since it has no past.

All columns are unmasked; the final position attends over the entire sequence up to and including itself, giving it the richest context — appropriate since it predicts the token after the whole sequence.

$\text{softmax}$ over all-$-\infty$ is undefined (NaN). Implementations guard against this by never fully masking a valid query, or by clamping, since such rows carry no meaningful output anyway.

A $1\times 1$ matrix with a single $0$ (allowed). There is no future to hide, so masking is a no-op and the token attends only to itself.

The causal mask blocks future positions (content-independent, triangular); the padding mask blocks meaningless filler positions (varies per sequence). They are combined by taking the union of blocked entries — both are added as $-\infty$ before softmax.

Teacher forcing supplies the correct past; the mask still forbids the future. Without it, teacher forcing would hand the model the whole ground-truth sequence at once, letting it read the answer it's supposed to predict.