Level 1 — RecognitionTransformer Architecture

Transformer Architecture

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Chapter: 4.1 Transformer Architecture Level: 1 — Recognition (MCQ, Matching, True/False with justification) Time Limit: 20 minutes Total Marks: 30

Section A — Multiple Choice (1 mark each) — 10 marks

Choose the single best answer.

Q1. The primary limitation of RNNs that transformers overcome is:

(a) They cannot process text
(b) Sequential computation prevents parallelization over the sequence
(c) They have no weights
(d) They cannot use GPUs at all

Q2. In scaled dot-product attention, the scores are scaled by:

(a) $\sqrt{d_k}$ in the denominator
(b) $d_k$ in the numerator
(c) $\frac{1}{d_k^2}$
(d) the sequence length $n$

Q3. The self-attention output for a single head is computed as:

(a) $\text{softmax}(QK^\top)V$
(b) $\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$
(c) $\text{softmax}(Q+K)V$
(d) $QK^\top V$

Q4. In multi-head attention with model dimension $d_{model}=512$ and $h=8$ heads, the per-head dimension $d_k$ is typically:

(a) 512
(b) 8
(c) 64
(d) 4096

Q5. Sinusoidal positional encodings are added because:

(a) self-attention is otherwise permutation-invariant (order-agnostic)
(b) they reduce the number of parameters to zero
(c) they replace the feed-forward network
(d) they normalize the gradients

Q6. Masked (causal) attention in the decoder prevents a position from:

(a) attending to itself
(b) attending to future positions
(c) attending to any position
(d) using positional encodings

Q7. The computational and memory complexity of standard self-attention in sequence length $n$ is:

(a) $O(n)$
(b) $O(n \log n)$
(c) $O(n^2)$
(d) $O(n^3)$

Q8. The feed-forward sublayer in the original Transformer is:

(a) a single linear layer
(b) two linear layers with a nonlinearity (e.g. ReLU) between them
(c) a recurrent layer
(d) a convolution over the whole sequence

Q9. Flash attention improves efficiency mainly by:

(a) changing the attention math to be approximate always
(b) avoiding materialization of the full $n\times n$ attention matrix via tiling/IO-awareness
(c) removing the softmax
(d) using RNNs internally

Q10. Rotary Positional Embeddings (RoPE) encode position by:

(a) adding a learned bias vector
(b) rotating query/key vectors by position-dependent angles
(c) concatenating one-hot position vectors
(d) scaling values by position index

Section B — Matching (1 mark each) — 6 marks

Match each term in Column X to its correct description in Column Y.

Column X	Column Y
Q11. Query matrix $Q$	A. Broadcasts values weighted by attention
Q12. Key matrix $K$	B. Represents what each token is "looking for"
Q13. Value matrix $V$	C. Content carried forward once weights are known
Q14. Encoder	D. Compared against queries to produce scores
Q15. Decoder	E. Bidirectional context, no causal mask
Q16. Residual connection	F. Autoregressive, uses masked self-attention
	G. Adds sublayer input to its output to ease gradient flow

Section C — True/False WITH Justification (2 marks each: 1 T/F + 1 justification) — 14 marks

Q17. "In the original 'Attention is All You Need' paper, LayerNorm is applied after adding the residual (Post-LN)." True / False — justify.

Q18. "Multi-head attention lets the model attend to information from different representation subspaces simultaneously." True / False — justify.

Q19. "Self-attention without positional information can distinguish the sentence 'dog bites man' from 'man bites dog'." True / False — justify.

Q20. "The $\sqrt{d_k}$ scaling is used to prevent dot products from growing large and pushing softmax into regions of tiny gradients." True / False — justify.

Q21. "An encoder-decoder Transformer uses cross-attention where the decoder's queries attend to the encoder's keys and values." True / False — justify.

Q22. "Flash attention changes the mathematical result of attention to an approximation." True / False — justify.

Q23. "Increasing sequence length from $n$ to $2n$ roughly quadruples the attention compute cost." True / False — justify.

Answer keyMark scheme & solutions

Section A — MCQ (1 mark each)

Q1 — (b). RNNs compute hidden states step-by-step; each depends on the previous, so the sequence cannot be parallelized in time. Transformers process all positions at once. (1)

Q2 — (a). Scores are $\frac{QK^\top}{\sqrt{d_k}}$ ; scaling by $\sqrt{d_k}$ keeps variance controlled. (1)

Q3 — (b). Full scaled dot-product attention formula. (a) omits scaling; (d) omits softmax. (1)

Q4 — (c). $d_k = d_{model}/h = 512/8 = 64$ . (1)

Q5 — (a). Self-attention is permutation-equivariant; without positional signal it cannot use order. (1)

Q6 — (b). Causal masking sets future scores to $-\infty$ so predictions depend only on past/current tokens. (1)

Q7 — (c). The $QK^\top$ matrix is $n\times n$ → $O(n^2)$ time and memory. (1)

Q8 — (b). FFN = $\text{Linear} \to \text{ReLU} \to \text{Linear}$ , applied position-wise. (1)

Q9 — (b). Flash attention is IO-aware and tiles the computation, computing exact softmax without storing the full matrix. (1)

Q10 — (b). RoPE applies a rotation to Q and K depending on absolute position, encoding relative position via inner products. (1)

Section B — Matching (1 mark each)

Q	Answer	Reason
Q11	B	Query = what a token seeks.
Q12	D	Keys are compared to queries → scores.
Q13	C	Values carry content that gets aggregated.
Q14	E	Encoder is bidirectional, unmasked.
Q15	F	Decoder is autoregressive with masked self-attention.
Q16	G	Residual adds input to output; eases gradient flow.

(A is a distractor describing the aggregation step generally.)

Section C — True/False with Justification (1 + 1)

Q17 — True. (T/F 1) The original paper uses Post-LN: sublayer output is added to the input, then LayerNorm is applied: $\text{LayerNorm}(x + \text{Sublayer}(x))$ . (justification 1)

Q18 — True. Each head has its own $W_Q, W_K, W_V$ projections into a subspace, so different heads capture different relations (syntax, position, coreference), and outputs are concatenated. (2)

Q19 — False. Both sentences contain the same token set; permutation-invariant self-attention gives identical representations without positional encoding, so it cannot distinguish them. (2)

Q20 — True. For $d_k$ -dim vectors with unit-variance entries, dot products have variance $\sim d_k$ ; dividing by $\sqrt{d_k}$ normalizes variance to $\sim1$ , keeping softmax out of saturated, low-gradient regions. (2)

Q21 — True. In cross-attention the decoder supplies $Q$ , and $K,V$ come from the encoder output, letting the decoder condition on the source sequence. (2)

Q22 — False. Flash attention computes the exact same softmax attention; it only reorders memory access (tiling, online softmax) to be faster and more memory-efficient. (2)

Q23 — True. Cost is $O(n^2)$ ; doubling $n$ multiplies cost by $2^2 = 4$ . (2)

[
  {"claim":"Per-head dim d_k = d_model/h = 512/8 = 64 (Q4)","code":"d_model=512; h=8; d_k=d_model/h; result = (d_k==64)"},
  {"claim":"Dot-product variance for d_k iid unit-variance dims scales with d_k, so scaling normalizes it (Q20)","code":"d_k=64; var_before=d_k; var_after=var_before/(sqrt(d_k)**2); result = (simplify(var_after)==1)"},
  {"claim":"Doubling sequence length quadruples O(n^2) attention cost (Q23)","code":"n=symbols('n',positive=True); ratio=(2*n)**2/n**2; result = (simplify(ratio)==4)"},
  {"claim":"Attention score matrix QK^T has n x n entries giving O(n^2) memory (Q7)","code":"n=symbols('n',positive=True); entries=n*n; result = (simplify(entries)==n**2)"}
]