Pretraining & Fine-Tuning LLMs

Question 1

(a) (4 marks) Forward: $h = W_0 x + \frac{\alpha}{r} B (A x)$. Compute $Ax$ first ($r$-dim), then $B(\cdot)$ — cheaper than forming $BA$. (2) Initialization: $A \sim \mathcal{N}(0,\sigma^2)$ (or small random), $B = 0$. Then $BA = 0 \Rightarrow \Delta W = 0$ at step 0. (1) Why: the model starts exactly as the pretrained model, so early training doesn't inject random noise into a well-tuned function; gradients flow gradually. If both were random, initial output would be corrupted. (1)

(b) (4 marks) LoRA params $= r\cdot k + d\cdot r = 16(4096) + 4096(16) = 65536 + 65536 = 131072$. (2) Full FT params for layer $= d\cdot k = 4096^2 = 16{,}777{,}216$. (1) Percentage $= 131072/16777216 = 0.78125\%$. (1)

(c) (3 marks) Per layer: 2 matrices → $2 \times 131072 = 262144$. (1) Total: $32 \times 262144 = 8{,}388{,}608 \approx 8.39$M trainable params. (2)

(d) (4 marks) (i) fp16: $7\times10^9 \times 2$ bytes $= 1.4\times10^{10}$ B $= 14$ GB. (1.5) (ii) NF4: $7\times10^9 \times 0.5$ bytes $= 3.5\times10^{9}$ B $= 3.5$ GB. (1.5) Compression ratio $= 14/3.5 = 4\times$. (1)

(e) (4 marks) ($\Leftarrow$) If $\operatorname{rank}(M)\le r$, by SVD $M = U\Sigma V^\top$ with $\le r$ nonzero singular values. Set $B = \frac{r}{\alpha} U_{:, :r}\Sigma_{:r,:r}$ and $A = V_{:,:r}^\top$; then $\frac{\alpha}{r}BA = U\Sigma V^\top = M$. (2) ($\Rightarrow$) $\operatorname{rank}(BA)\le \min(\operatorname{rank}(B),\operatorname{rank}(A)) \le r$ since $A$ has $r$ rows and $B$ has $r$ columns. Scaling by $\alpha/r$ doesn't change rank. So if $\frac{\alpha}{r}BA = M$ then $\operatorname{rank}(M)\le r$. (2)

(f) (3 marks) Double quantization quantizes the quantization constants (the per-block scale/absmax factors) themselves — a second-level quantization of the metadata. It saves additional memory (~0.4 bits/param) at negligible accuracy cost.

Question 2

(a) (6 marks) (i) Causal LM: $\mathcal{L}_{CLM} = -\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t})$. Every token contributes; strictly left-context. (3) (ii) Masked LM: let $\mathcal{M}$ be masked positions. $\mathcal{L}_{MLM} = -\sum_{t\in\mathcal{M}}\log p_\theta(x_t \mid x_{\setminus \mathcal{M}})$. Only masked tokens contribute; bidirectional context. (3)

(b) (4 marks) Input: the <X> sat <Y> the mat Target: <X> cat <Y> on the <Z> (Sentinels replace corrupted spans in the input; the target lists sentinel+original-span, terminated by final sentinel <Z>.) (4; 2 input, 2 target)

(c) (4 marks) At inference there is no [MASK] token, but if 100% of masked positions were [MASK] during training the model would only learn representations conditioned on seeing [MASK], creating a pretrain/finetune mismatch. (2) Keeping 10% unchanged forces the model to build good representations for real tokens too; 10% random tokens force it not to blindly trust the observed token, encouraging robust context use. (2)

(d) (6 marks) $C = 6PN = 6 \times 7\times10^9 \times 10^{12} = 4.2\times10^{22}$ FLOPs. (3) Compute-optimal tokens $\approx 20P = 20\times7\times10^9 = 1.4\times10^{11}$. (1) Actual tokens $N = 10^{12} \gg 1.4\times10^{11}$ (ratio $N/P = 142.9 \gg 20$). (1) Therefore the model is over-trained relative to Chinchilla compute-optimality (trained on far more tokens than compute-optimal — common for inference-efficient deployment). (1)

Question 3

(a) (4 marks) Feature extraction: backbone frozen (no gradients to pretrained weights); only a new head is trained. Full fine-tuning: all parameters receive gradients and update. (2) Preferable when: very small target dataset (avoids overfitting), limited compute, or when preserving the general representation / avoiding forgetting is important. (2)

(b) (6 marks) $\mathcal{L}_{KD} = \tau^2 \, \mathrm{KL}\!\left(\mathrm{softmax}(z^t/\tau) \,\|\, \mathrm{softmax}(z^s/\tau)\right)$, $\mathcal{L}_{CE} = -\sum_i y_i\log\mathrm{softmax}(z^s)_i$, combined $\mathcal{L}=\lambda\mathcal{L}_{KD}+(1-\lambda)\mathcal{L}_{CE}$. (3) Definitions: $z^t,z^s$ teacher/student logits, $\tau$ temperature softening the distributions, $y$ hard labels. (1) $\tau^2$ scaling: the gradient of the softened KL term is proportional to $1/\tau^2$ (softmax with $z/\tau$ shrinks logit gradients by $\tau$; the KL derivative brings another $1/\tau$). Multiplying by $\tau^2$ keeps the KD gradient magnitude comparable to the hard-label CE gradient so their relative weighting stays $\approx\lambda$ across temperatures. (2)

(c) (5 marks) Formal: forgetting $= \mathrm{Perf}_A(\theta_{\text{init}}) - \mathrm{Perf}_A(\theta_{\text{after B}})$; positive means degradation on A after training on B. (1) (i) LoRA: freezes original weights $W_0$; task B only edits low-rank adapters. Base capability on A stored in $W_0$ is untouched, so removing/swapping adapters recovers A. (2) (ii) Replay: interleave a small set of task-A examples during B training; gradients keep A's loss low, constraining the update to a region good for both tasks. (2)

(d) (3 marks) Scale $s = |w|_{\max}/127 = 0.5/127 = 0.0039370\ldots$ (1) Quantize: $q = \mathrm{round}(0.31/s) = \mathrm{round}(0.31/0.0039370) = \mathrm{round}(78.74) = 79$. (1) Dequantize: $\hat w = 79 \times s = 79 \times 0.0039370 = 0.311024$; error $= |0.311024 - 0.31| = 0.001024$. (1)

[
  {"claim":"LoRA per-layer params r*k+d*r = 131072", "code":"r,d,k=16,4096,4096\nresult=(r*k+d*r==131072)"},
  {"claim":"LoRA fraction of full FT is 0.78125%", "code":"r,d,k=16,4096,4096\nresult=(Rational(r*k+d*r, d*k)*100==Rational(78125,100000))"},
  {"claim":"Total QV LoRA params over 32 layers = 8388608", "code":"r,d,k=16,4096,4096\nper=2*(r*k+d*r)\nresult=(32*per==8388608)"},
  {"claim":"NF4 compression vs fp16 is 4x and C=4.2e22 FLOPs", "code":"fp16=7e9*2\nnf4=7e9*0.5\nC=6*7e9*1e12\nresult=(abs(fp16/nf4-4)<1e-9 and abs(C-4.2e22)<1e15)"},
  {"claim":"INT8 dequant of 0.31 gives error ~0.001024", "code":"s=Rational(1,2)/127\nq=round(float(Rational(31,100)/s))\ndeq=q*s\nerr=abs(float(deq)-0.31)\nresult=(q==79 and abs(err-0.001024)<1e-5)"}
]