Scaling & Efficient Architectures

Q1. (4 marks)

• Finding: For a fixed compute budget, model size $N$ and training data $D$ should be scaled in equal proportion; previous large models (e.g. GPT-3, Gopher) were undertrained — too large for their data. (2)
• Rule: Compute-optimal scaling requires roughly $D \approx 20 \times N$ tokens per parameter. (2)

Q2. (3 marks)

• $E$: irreducible loss — entropy of the data / Bayes error, unattainable floor. (1)
• $A/N^{\alpha}$: error due to finite model size (parameters). (1)
• $B/D^{\beta}$: error due to finite training data (tokens). (1)

Q3. (4 marks)

• Rearranging $C = 6ND$: $D = \dfrac{C}{6N}$. (1)
• $D = \dfrac{6\times10^{21}}{6 \times 1\times10^{10}}$. (1)
• $= \dfrac{6\times10^{21}}{6\times10^{10}} = 1\times10^{11}$ tokens. (1)
• (Sanity: $D/N = 10^{11}/10^{10} = 10$, near the compute-optimal $\sim 20$ regime.) (1)

Q4. (4 marks)

• Emergent ability: a capability that is absent (near-random) in smaller models but appears abruptly once model scale (params/data/compute) crosses a threshold; not predictable by smooth extrapolation from small-model performance. (2)
• Criticism (any one): emergence may be an artifact of discontinuous / nonlinear metrics (e.g. exact-match accuracy); using smooth continuous metrics often shows gradual, predictable improvement. (2)

Q5. (6 marks) (a)

• Gating network outputs a score/logit per expert, converted (softmax) to weights; top-$k$ selects the $k$ highest-scoring experts. (2)
• Only those $k$ experts process the token; outputs combined as a weighted sum by the gate weights. (1)

(b)

• Total params scale with number of experts $E$ (all experts stored). (1.5)
• Per-token compute scales with $k$ (only $k \ll E$ experts activated per token), so FLOPs stay roughly constant as $E$ grows. (1.5)

Q6. (4 marks)

• Problem: routing can collapse so a few experts receive most tokens while others are starved, causing under-utilization and instability; some experts overflow capacity. (2)
• Auxiliary loss: penalizes imbalance (e.g. product of fraction of tokens routed to each expert and mean gate probability), encouraging uniform expert utilization. (2)

Q7. (5 marks)

• Tensor parallelism: splits individual layer's weight matrices/tensors across devices so a single operation is computed jointly. (2)
• Pipeline parallelism: splits the layers/stages of the model across devices, passing activations forward between stages. (2)
• Drawback of pipeline parallelism: pipeline bubbles (idle time) when stages wait for micro-batches. (1)

Q8. (3 marks)

• Stage 1: optimizer states sharded. (1)
• Stage 2: optimizer states + gradients sharded. (1)
• Stage 3: optimizer states + gradients + parameters sharded. (1)

Q9. (4 marks)

• KV-cache stores keys and values for every layer and every position: values $= 2 \cdot L \cdot s \cdot d$ (factor 2 for K and V). (3)
• Reduction technique (any one): Multi-Query Attention (MQA) / Grouped-Query Attention (GQA), quantization of cache, sliding-window / local attention, or eviction/paged KV. (1)

Q10. (3 marks) (a) Draft model cheaply generates several candidate tokens ahead; the target model verifies them in one parallel forward pass, accepting the longest valid prefix and continuing. (2) (b) The output distribution is identical to sampling from the target model (lossless / distribution-preserving); it only speeds up decoding. (1)

[
  {"claim":"Q3: D = C/(6N) = 6e21/(6*1e10) = 1e11 tokens", "code":"C=Integer(6)*10**21; N=Integer(1)*10**10; D=C/(6*N); result = (D == 10**11)"},
  {"claim":"Q3 sanity: D/N = 10", "code":"D=Integer(10)**11; N=Integer(10)**10; result = (D/N == 10)"},
  {"claim":"Q9: KV-cache size = 2*L*s*d for L=32,s=2048,d=4096 gives 536870912", "code":"L,s,d=32,2048,4096; result = (2*L*s*d == 536870912)"},
  {"claim":"Chinchilla approx: for N=1e10 compute-optimal tokens ~20N = 2e11", "code":"N=Integer(10)**10; result = (20*N == 2*10**11)"}
]