Training Deep Networks

Question 1 (12)

(a) (4 marks: 1 each for $m_t$, $v_t$, bias-correction pair, final update) $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$ $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$ $\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^t}$ $\theta_t = \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

(b) (4 marks) Unrolling with $m_0=0$: $m_t = (1-\beta_1)\sum_{i=1}^{t}\beta_1^{t-i} g_i.$ Taking expectation with $\mathbb{E}[g_i]=\mathbb{E}[g]$: $\mathbb{E}[m_t] = (1-\beta_1)\,\mathbb{E}[g]\sum_{i=1}^{t}\beta_1^{t-i} = (1-\beta_1)\mathbb{E}[g]\cdot\frac{1-\beta_1^{t}}{1-\beta_1} = (1-\beta_1^{t})\mathbb{E}[g].$ Since $m_t$ is biased toward 0 (factor $1-\beta_1^t <1$), dividing by $(1-\beta_1^t)$ corrects it. (2 for unroll+geometric sum, 2 for result & reasoning)

(c) (4 marks) In Adam-with-L2, the penalty enters the gradient ($g_t \leftarrow g_t + \lambda\theta$), so decay is scaled by the adaptive $1/\sqrt{\hat v}$ — parameters with large historical gradients get less effective decay. AdamW decouples decay: it applies $\theta \leftarrow \theta - \alpha\lambda\theta$ directly to the weights, outside the adaptive step: $\theta_t = \theta_{t-1} - \alpha\Big(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} + \lambda\theta_{t-1}\Big).$ This makes decay uniform/independent of gradient magnitudes → better generalization.

Question 2 (10)

(a) (4) Momentum: $v_t = \mu v_{t-1} + g_t,\qquad \theta_t = \theta_{t-1} - \eta v_t.$ Nesterov: $v_t = \mu v_{t-1} + \nabla_\theta f(\theta_{t-1} - \eta\mu v_{t-1}),\qquad \theta_t=\theta_{t-1}-\eta v_t.$ (gradient evaluated at the look-ahead point).

(b) (4) Steady state $v_\infty = \mu v_\infty + g \Rightarrow v_\infty = \dfrac{g}{1-\mu}$. Effective step is $\eta v_\infty = \dfrac{\eta g}{1-\mu}$, i.e. a multiplier of $\dfrac{1}{1-\mu}$ vs plain SGD ($\eta g$). For $\mu=0.9$ → $10\times$.

(c) (2) Momentum uses the gradient at the current point; Nesterov evaluates the gradient at the anticipated future position, giving a correction that reduces overshoot.

Question 3 (14)

(a) (4) $\mu_B=\frac1m\sum_i x_i,\quad \sigma_B^2=\frac1m\sum_i(x_i-\mu_B)^2,$ $\hat x_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},\quad y_i=\gamma\hat x_i+\beta.$

(b) (6) With $\sigma^2\equiv\sigma_B^2+\epsilon$, standard result: $\frac{\partial\mathcal L}{\partial x_i}=\frac{1}{m\sqrt{\sigma^2}}\left(m\frac{\partial\mathcal L}{\partial\hat x_i}-\sum_{j}\frac{\partial\mathcal L}{\partial\hat x_j}-\hat x_i\sum_j\frac{\partial\mathcal L}{\partial\hat x_j}\hat x_j\right).$ (3 for identifying the three paths $x_i\to\hat x_i,\ \to\mu,\ \to\sigma^2$; 3 for correct assembled form.)

(c) (4) (i) BN normalizes across the batch dimension (per feature over the batch); LN normalizes across the feature dimension per single example. (ii) BN's statistics depend on batch → uses running averages at inference and is unstable for small batches; LN is batch-independent, identical at train/inference (ideal for RNNs/Transformers).

Question 4 (10)

(a) (3) $\eta_t=\eta_{min}+\tfrac12(\eta_{max}-\eta_{min})\Big(1+\cos\big(\tfrac{t}{T}\pi\big)\Big).$

(b) (4) $\eta_t=\begin{cases}\dfrac{t}{w}\,\eta_{max}, & t\le w\\[2mm]\eta_{min}+\tfrac12(\eta_{max}-\eta_{min})\Big(1+\cos\big(\tfrac{t-w}{T-w}\pi\big)\Big), & t>w.\end{cases}$

(c) (3) Early in training, adaptive optimizers have unreliable variance estimates ($\hat v$) and large-batch gradients can cause huge steps; warmup keeps steps small until statistics stabilize, preventing divergence.

Question 5 (8)

(a) (3) Inverted dropout: at train time keep each unit with prob $p$, and divide surviving activations by $p$ (scale factor $1/p$), so expected activation is preserved; at test time no scaling/masking is applied.

(b) (3) Loss $L'=L+\frac{\lambda}{2}\|\theta\|^2$, so $\nabla_\theta L'=g+\lambda\theta$. Update: $\theta\leftarrow\theta-\eta(g+\lambda\theta)=(1-\eta\lambda)\theta-\eta g.$

(c) (2) Monitor validation loss (or metric); stop when it fails to improve for patience consecutive evaluations, restoring the best checkpoint.

Question 6 (6)

(6: 2 for norm computation, 2 for conditional scale, 2 for correct rescale)

total_norm = np.sqrt(sum(np.sum(g**2) for g in grads))
if total_norm > clip_norm:
    scale = clip_norm / (total_norm + 1e-6)
    grads = [g * scale for g in grads]

Trigger: only when global norm exceeds clip_norm; rescale all grads by clip_norm/total_norm so the clipped global norm equals clip_norm, preserving direction.

[
{"claim":"E[m_t] = (1-beta1**t)*E[g] via geometric sum", "code":"b1=symbols('b1'); t=4; s=summation(b1**(t-i),(i,1,t)); result = simplify((1-b1)*s - (1-b1**t))==0"},
{"claim":"Momentum steady-state effective multiplier is 1/(1-mu); mu=0.9 -> 10", "code":"mu=Rational(9,10); result = simplify(1/(1-mu))==10"},
{"claim":"L2 SGD update gives (1-eta*lambda) coefficient on theta", "code":"eta,lam,th,g=symbols('eta lam th g'); upd=th-eta*(g+lam*th); result = simplify(upd-((1-eta*lam)*th-eta*g))==0"},
{"claim":"Cosine schedule at t=0 gives eta_max and t=T gives eta_min", "code":"emax,emin=symbols('emax emin'); f=lambda t,T: emin+Rational(1,2)*(emax-emin)*(1+cos(t/T*pi)); result = (simplify(f(0,1)-emax)==0) and (simplify(f(1,1)-emin)==0)"}
]