Training Deep Networks

Question 1 (14)

(a) Step 1, $g_1 = 2$.

• $m_1 = 0.9(0) + 0.1(2) = 0.2$; $v_1 = 0.999(0) + 0.001(4) = 0.004$. (1)
• Bias correction: $\hat m_1 = m_1/(1-\beta_1^1) = 0.2/0.1 = 2.0$. (1)
• $\hat v_1 = v_1/(1-\beta_2^1) = 0.004/0.001 = 4.0$. (1)
• Update: $w_1 = w_0 - \alpha\,\hat m_1/(\sqrt{\hat v_1}+\epsilon) = 0 - 0.1\cdot 2/( 2+10^{-8}) \approx -0.1$. (2)

Why: bias correction removes the initialization-toward-zero bias; step 1 the corrected moments equal the raw gradient/gradient², so the effective step is $-\alpha\cdot\text{sign}(g)$.

(b) Step 2, $g_2 = -1$.

• $m_2 = 0.9(0.2) + 0.1(-1) = 0.08$. (1)
• $v_2 = 0.999(0.004) + 0.001(1) = 0.003996 + 0.001 = 0.004996$. (1)
• $\hat m_2 = 0.08/(1-0.9^2) = 0.08/0.19 = 0.421053$. (1)
• $\hat v_2 = 0.004996/(1-0.999^2) = 0.004996/0.001999 = 2.499250$. (1)
• $w_2 = w_1 - 0.1\cdot 0.421053/(\sqrt{2.499250}+10^{-8}) = -0.1 - 0.1\cdot0.421053/1.580901 \approx -0.1 - 0.026634 = -0.126634$. (1)

(c) AdamW: adaptive step plus decoupled decay: $w_2 = w_1 - \alpha\Big(\tfrac{\hat m_2}{\sqrt{\hat v_2}+\epsilon} + \lambda w_1\Big)\;\text{or}\;= w_1 - \alpha\tfrac{\hat m_2}{\sqrt{\hat v_2}+\epsilon} - \alpha\lambda w_1.$ With $w_1=-0.1$: decay term $-\alpha\lambda w_1 = -0.1\cdot0.01\cdot(-0.1) = +10^{-4}$, so $w_2 \approx -0.126534$. (2) Why (1 sentence, 2): L2 added to the loss enters the gradient and is then divided by $\sqrt{\hat v}$ (so decay is scaled per-parameter and coupled to gradient magnitude), whereas AdamW applies decay directly to the weight independent of the adaptive denominator, giving uniform, predictable shrinkage.

Question 2 (12)

(a)

• Warmup ($t\le W$): $\eta(t) = \eta_{\max}\cdot t/W$. (2)
• Cosine ($W<t\le T$): $\eta(t) = \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})\big(1+\cos(\pi\,\tfrac{t-W}{T-W})\big)$. (2)

(b)

• $\eta(250) = 5\times10^{-4}\cdot 250/500 = 2.5\times10^{-4}$. (2)
• $\eta(5250)$: progress $= (5250-500)/(10000-500) = 4750/9500 = 0.5$. $\cos(\pi\cdot0.5)=0$, so $\eta = \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})(1+0) = 5\times10^{-5} + 0.5(4.5\times10^{-4}) = 5\times10^{-5}+2.25\times10^{-4} = 2.75\times10^{-4}$. (2)

(c) Early in training the second-moment estimate $\hat v$ is built from few, high-variance gradient samples, so it can be small/unreliable → the adaptive step $\hat m/\sqrt{\hat v}$ can be huge and unstable, worse with large batches (fewer steps, sharper early curvature); warmup keeps $\eta$ small until $\hat v$ stabilizes, preventing early divergence. (4)

Question 3 (12)

(a)

• Mean $\mu = (4+8+6+2)/4 = 5$. (1)
• Deviations: $-1,3,1,-3$; squares $1,9,1,9$; sum $=20$; biased var $= 20/4 = 5$; $\sigma = \sqrt5 \approx 2.23607$. (2)
• Normalized $\hat x_i = (x_i-5)/\sqrt5$: $[-0.44721, 1.34164, 0.44721, -1.34164]$. (1)
• $y_i = \gamma\hat x_i + \beta = 2\hat x_i + 1$: $[0.10557, 3.68328, 1.89443, -1.68328]$. (2)

(b)

• $\mu_{\text{new}} = 0.9(5.5)+0.1(5) = 4.95+0.5 = 5.45$. (1.5)
• $\sigma^2_{\text{new}} = 0.9(3.0)+0.1(5) = 2.7+0.5 = 3.2$. (1.5)

(c) Failure mode: batch norm's statistics depend on batch composition, so with very small batch sizes (or batch size 1) estimates are noisy/unreliable, and it couples examples in a batch — problematic for sequence models / variable-length data. Layer norm normalizes over features per-example, independent of batch → preferred in Transformers / RNNs. (3)

Question 4 (12)

(a) Inverted dropout: kept units scaled by $1/p = 1/0.8 = 1.25$; dropped units set to 0. Keep $\{1,3,4\}$ → $[1.0\cdot1.25,\;0,\;2.0\cdot1.25,\;-1.0\cdot1.25,\;0] = [1.25,\;0,\;2.5,\;-1.25,\;0]$. (4)

(b) Rescaling by $1/p$ at train time keeps the expected activation equal to the no-dropout value (since a unit survives with prob $p$, $\mathbb E[\text{output}] = p\cdot(a/p) = a$). This guarantees that at inference dropout can simply be turned off (identity) with no rescaling needed, keeping train/test activation magnitudes consistent. (3)

(c) Val loss rose for 6 epochs > patience 5 → early stopping triggers, training halts; the deployed weights are those from epoch 12 (best validation loss), restored via checkpointing. (2)

(d) L1 ($\lambda\|w\|_1$) drives many weights exactly to zero → sparse solutions; L2 ($\lambda\|w\|_2^2$) shrinks all weights smoothly toward zero without exact zeros. For an interpretable sparse feature selector prefer L1. (3)

Question 5 (10)

(a) $\|g\|_2 = \sqrt{36+64+0} = \sqrt{100} = 10 > 5$, so clip: $g' = g\cdot(c/\|g\|) = g\cdot0.5 = [3,4,0]$. (4)

(b) $\Delta\theta = -\eta g' = -0.1\cdot[3,4,0] = [-0.3,-0.4,0]$. (2)

(c) Loss spikes/NaNs arise when exploding gradients (common in RNNs due to repeated Jacobian products) produce huge update steps that overshoot to unstable regions/overflow. Clipping caps the update norm, guaranteeing bounded steps regardless of a single anomalous gradient, preserving direction. Interaction: a lower clipping threshold has a similar stabilizing effect to a lower learning rate, so threshold and $\eta$ must be tuned together — too tight a clip with a large $\eta$ still allows the scaled step to be large per component only up to $c$, but an overly small $c$ slows convergence like an over-shrunk lr. (4)

[
  {"claim":"Adam w1 = -0.1 (step 1)","code":"a=Rational(1,10); m1=0.1*2; v1=0.001*4; mh=m1/0.1; vh=v1/0.001; w1=0 - float(a)*mh/(sqrt(vh)+1e-8); result = abs(w1 - (-0.1)) < 1e-6"},
  {"claim":"Adam w2 approx -0.126634","code":"m1=0.2; v1=0.004; m2=0.9*m1+0.1*(-1); v2=0.999*v1+0.001*1; mh=m2/(1-0.9**2); vh=v2/(1-0.999**2); w1=-0.1; w2=w1-0.1*mh/(sqrt(vh)+1e-8); result = abs(float(w2)-(-0.126634))<1e-4"},
  {"claim":"cosine schedule eta(5250)=2.75e-4","code":"emin=5e-5; emax=5e-4; prog=(5250-500)/(10000-500); eta=emin+0.5*(emax-emin)*(1+cos(pi*prog)); result = abs(float(eta)-2.75e-4)<1e-9"},
  {"claim":"BN running var update = 3.2","code":"v=0.9*3.0+0.1*5; result = abs(v-3.2)<1e-12"},
  {"claim":"gradient clip [6,8,0] to [3,4,0]","code":"import sympy as sp; g=sp.Matrix([6,8,0]); n=sp.sqrt(g.dot(g)); gc=g*(5/n); result = gc == sp.Matrix([3,4,0])"}
]