Encoder vs decoder vs encoder-decoder: Question bank

This is a misconception gym for the parent note 4.1.10 Encoder vs decoder vs encoder-decoder. Each line is a question ::: answer reveal. Cover the question with your hand, commit to an answer and a reason, then check. If your reason is wrong even when your yes/no is right, that counts as a miss.

Before we start, three plain-word anchors so nothing below uses an undefined term:

True or false — justify

Bidirectional attention makes a model better at generating text than causal attention.

False — bidirectionality lets a token cheat by seeing the answer it is supposed to predict, so it cannot generate autoregressively at all; generation needs the causal restriction.

BERT can be used to write a story one word at a time.

False — BERT is encoder-only with bidirectional attention and no causal mask, so it has no notion of "next token given only the past"; it fills blanks, it does not continue text.

In a decoder, the causal mask is applied after the softmax to zero out future tokens.

False — the mask adds

- \infty

to future scores before softmax; masking after softmax would leave the surviving weights un-renormalized and no longer summing to 1.

A decoder-only model literally never looks at future tokens during training.

True in effect, but the mechanism matters — all positions are processed in parallel with future tokens present in memory, and the causal mask is what forbids each position from attending to them.

Cross-attention and self-attention use the same three inputs.

False — self-attention draws Q, K, V from one sequence; cross-attention draws Q from the decoder but K and V from the encoder output, which is exactly how information crosses the source–target boundary.

Encoder-only and decoder-only models differ only in the attention mask.

Largely true architecturally — the block structure is nearly identical, and the mask (all-ones vs lower-triangular) plus the training objective is what separates a BERT from a GPT.

The [CLS] token's usefulness comes from causal attention.

False — [CLS] works precisely because attention is bidirectional, letting it aggregate information from the entire sentence including tokens to its right.

Adding cross-attention to a GPT-style model would turn it into an encoder-decoder.

Partly — you also need a separate encoder stack producing K and V; cross-attention layers are useless without an encoded source to attend to.

Spot the error

"To make BERT causal, just multiply the attention weights by a lower-triangular matrix of 1s and 0s."

The error is renormalization — zeroing weights post-softmax breaks the probability sum; the fix is to add

- \infty

to the scores before softmax so the survivors re-normalize correctly.

"An encoder-decoder's decoder skips self-attention because cross-attention already handles context."

Wrong — the decoder keeps causal self-attention over the target it has generated so far, then does cross-attention to the source; both are needed, they answer different questions.

"In GPT the last layer's representation of the first token already knows the whole sentence."

False — the causal mask means token 1 only ever attends to itself, so its representation is context-free no matter how many layers you stack.

"During translation, the decoder attends to the encoder using the encoder's queries."

The roles are swapped — cross-attention takes the query from the decoder ("what am I asking about?") and keys/values from the encoder ("what does the source know?").

"Softmax over attention scores can output a negative weight if the score is negative."

No — softmax exponentiates every score, so

e^{s} > 0

always; all attention weights are strictly positive and sum to 1, regardless of raw score sign.

"Encoder-only models are pointless for question answering because they can't generate."

Wrong when the answer is a span inside the input — the model predicts start/end positions over existing tokens, no generation required, which is exactly extractive QA's design.

"The chain-rule product

P (x_{1}, \dots, x_{n}) = \prod_{t} P (x_{t} ∣ x_{< t})

only holds for decoder models."

The factorization is a universal identity of probability; decoders are simply built to compute each conditional directly, but the chain rule itself is not architecture-specific.

Why questions

Why does causal masking use

- \infty

rather than a large negative number like

- 100

Because

e^{- \infty} = 0

gives an exact zero weight; a finite

- 100

leaves a tiny leak of future information, which over many layers and long sequences can still corrupt the autoregressive property.

Why can the encoder process all its tokens in one parallel pass while the decoder generates one at a time?

The encoder's whole input is known up front, so all positions compute at once; the decoder's later tokens don't exist yet at inference, forcing a sequential step-by-step loop.

Why does scaling scores by

d_{k}

appear in every one of these architectures?

With large key dimension

d_{k}

the dot products grow large in magnitude, pushing softmax into a saturated near-one-hot regime with tiny gradients; dividing by

d_{k}

keeps scores in a healthy range so learning stays stable — see Self-Attention Mechanism.

Why does an encoder-decoder outperform a decoder-only model on translation, in principle?

The encoder builds a bidirectional representation of the source (every source word sees full context) before any target word is produced, whereas a decoder-only model must read and write in one causal pass, seeing source context only left-to-right.

Why do all three architectures still need positional information?

Attention is permutation-invariant — it treats a sequence as an unordered set — so without Positional Encoding "dog bites man" and "man bites dog" would be indistinguishable to the model.

Why is the [CLS] representation, not the first real word's, used for classification in BERT?

[CLS] carries no word meaning of its own, so it is a neutral slot free to aggregate the whole sentence's signal via bidirectional attention, rather than being biased toward one input word.

Why does cross-attention take K and V from the encoder but Q from the decoder?

The decoder asks the question (Q = "what target word do I need next?") while the encoder holds the searchable source content (K = index, V = payload), so retrieval flows source → target exactly as translation requires.

Edge cases

What does causal attention compute for the very first token of a sequence?

It attends only to itself — a single-element softmax that outputs weight 1 on that token — so the first representation is purely its own embedding plus position, with no context.

If a sequence has length 1, do encoder and decoder attention behave identically?

Yes — with one token there is no future to mask and no other token to attend to, so bidirectional and causal masks collapse to the same trivial self-attention.

At the last decoding step of an encoder-decoder, can the decoder self-attention see the whole target?

Yes — the final token's causal window covers all previously generated target tokens, so only at this last step does the decoder have full left-context of its own output.

What happens to cross-attention if the encoder output is empty (source length 0)?

The softmax denominator sums over zero keys and is undefined (0/0) — a degenerate case that must be forbidden; every encoder-decoder task requires at least one source token to attend to.

When two attention scores are exactly equal, how does softmax distribute weight?

It splits weight equally between them — softmax is symmetric in equal inputs — so tied relevance yields tied attention, never an arbitrary tie-break.

If every attention score in a row is identical, what does the output vector equal?

A uniform softmax makes all weights

1/ n

, so the output is the plain average of the value vectors — the "I have no preference" degenerate case, useful as a sanity check.

Can a decoder-only model like GPT do the bidirectional understanding tasks BERT does?

Partially — with clever prompting it can, but its causal mask means early tokens never see later context, so it is structurally weaker at tasks needing full bidirectional reads unless reformulated as generation.

Recall Fast self-test

Which mask is lower-triangular? ::: The causal (decoder) mask — 1 on and below the diagonal, 0 (or $- \infty$ in scores) above it. Which architecture has cross-attention layers? ::: Only the encoder-decoder — T5-style Sequenceto-Sequence Models use them to connect source and target. Where does the query come from in cross-attention? ::: From the decoder's current target position, never from the encoder.