Level 2 — RecallTraining Deep Networks

Training Deep Networks

⏱ 30 minutes40 marksprintable — key stays hidden on paper

Difficulty: Level 2 (Recall: definitions, standard textbook problems, short derivations) Time Limit: 30 minutes Total Marks: 40

Q1. Define stochastic gradient descent (SGD) and state how it differs from full-batch gradient descent in terms of the data used per update. (3 marks)

Q2. A dataset has $N = 50{,}000$ training examples. Using a mini-batch size of $250$ , how many parameter updates (iterations) occur in one epoch? How many updates occur after $8$ epochs? (4 marks)

Q3. Write the update equations for classical (heavy-ball) momentum, defining every symbol. State one practical benefit of momentum over plain SGD. (5 marks)

Q4. For AdaGrad, the per-parameter update is $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}}\, g_t,$ where $G_t = \sum_{\tau=1}^{t} g_\tau^2$ . Explain in one or two sentences why AdaGrad's learning rate decreases over time, and state how RMSprop fixes this problem. (5 marks)

Q5. Adam maintains biased first and second moment estimates $m_t$ and $v_t$ . Given $\beta_1 = 0.9$ , a gradient sequence with $m_1 = 0.2$ (after the first update, $m_0=0$ ), compute the bias-corrected first moment $\hat{m}_1$ . Show the formula and the numeric value. (4 marks)

Q6. State the formula for batch normalization of an activation $x_i$ over a mini-batch (including the learnable scale and shift). Name one key difference between batch normalization and layer normalization. (5 marks)

Q7. Explain dropout as a regularization technique. If dropout keeps a unit with probability $p = 0.8$ , what scaling is applied to the retained activations at training time under inverted dropout? (4 marks)

Q8. Define L2 weight decay. Write the modified loss function with regularization strength $\lambda$ , and give the resulting gradient contribution added to $\frac{\partial L}{\partial w}$ . (4 marks)

Q9. Describe gradient clipping by norm. If the gradient vector $g$ has L2 norm $\|g\| = 10$ and the clipping threshold is $c = 4$ , write the rescaled gradient. (3 marks)

Q10. State what early stopping is and name the quantity typically monitored to decide when to stop. (3 marks)

End of paper

Answer keyMark scheme & solutions

Q1. (3 marks)

SGD updates parameters using the gradient computed from a single training example (or one sample) per step rather than the whole dataset. (1)
Update: $\theta \leftarrow \theta - \eta \nabla_\theta L(\theta; x^{(i)}, y^{(i)})$ . (1)
Difference: full-batch uses all $N$ examples per update (one update per epoch, smooth but costly); SGD uses one example, giving noisy but frequent/cheaper updates. (1)

Q2. (4 marks)

Updates per epoch $= N / \text{batch size} = 50000 / 250 = 200$ . (2)
After 8 epochs $= 200 \times 8 = 1600$ updates. (2) Why: one epoch = one full pass; number of mini-batches = examples ÷ batch size.

Q3. (5 marks)

Velocity update: $v_{t} = \gamma v_{t-1} - \eta g_t$ (or $v_t=\gamma v_{t-1}+\eta g_t$ with subtraction below). (2)
Parameter update: $\theta_{t} = \theta_{t-1} + v_{t}$ (or $\theta_t = \theta_{t-1} - v_t$ in the alternate sign convention). (1)
Symbols: $\gamma$ = momentum coefficient (typically 0.9), $\eta$ = learning rate, $g_t$ = gradient at step $t$ , $v_t$ = velocity/accumulated update. (1)
Benefit: accelerates in consistent-gradient directions and dampens oscillations in high-curvature directions → faster convergence. (1)

Q4. (5 marks)

$G_t$ is a running sum of squared gradients, so it is non-decreasing; the denominator $\sqrt{G_t+\epsilon}$ grows over time. (2)
Hence the effective learning rate $\eta/\sqrt{G_t+\epsilon}$ shrinks monotonically, eventually becoming too small to learn. (1)
RMSprop replaces the cumulative sum with an exponentially decaying moving average: $E[g^2]_t = \rho E[g^2]_{t-1} + (1-\rho)g_t^2$ . (1)
This prevents unbounded growth, keeping the effective learning rate from vanishing. (1)

Q5. (4 marks)

Bias-correction formula: $\hat{m}_t = \dfrac{m_t}{1-\beta_1^t}$ . (2)
At $t=1$ : $\hat{m}_1 = \dfrac{0.2}{1 - 0.9^1} = \dfrac{0.2}{0.1} = 2.0$ . (2)

Q6. (5 marks)

Batch statistics: $\mu_B = \frac{1}{m}\sum_i x_i$ , $\sigma_B^2 = \frac{1}{m}\sum_i (x_i-\mu_B)^2$ . (1)
Normalize: $\hat{x}_i = \dfrac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$ . (2)
Scale & shift: $y_i = \gamma \hat{x}_i + \beta$ , with learnable $\gamma,\beta$ . (1)
Difference: BatchNorm normalizes across the batch dimension (per feature), while LayerNorm normalizes across features within a single example (batch-independent). (1)

Q7. (4 marks)

Dropout randomly deactivates (zeros) units with some probability during training, forcing the network not to rely on specific neurons → reduces co-adaptation/overfitting. (2)
With keep probability $p=0.8$ , inverted dropout scales retained activations by $1/p = 1/0.8 = 1.25$ so expected activation is preserved and no scaling is needed at test time. (2)

Q8. (4 marks)

L2 weight decay adds a penalty proportional to the squared magnitude of weights. (1)
Loss: $\tilde{L} = L + \frac{\lambda}{2}\sum_w w^2$ (or $\lambda \|w\|_2^2$ ). (2)
Gradient contribution: $\frac{\partial \tilde L}{\partial w} = \frac{\partial L}{\partial w} + \lambda w$ . (1)

Q9. (3 marks)

Gradient clipping by norm rescales $g$ if $\|g\| > c$ : $g \leftarrow c \cdot \frac{g}{\|g\|}$ (leave unchanged otherwise). (1)
Here $\|g\|=10 > 4$ , so rescaled $g = \frac{4}{10} g = 0.4\, g$ . (2)

Q10. (3 marks)

Early stopping halts training when performance on a validation set stops improving (before overfitting sets in), typically restoring best-so-far weights. (2)
Monitored quantity: validation loss (or validation error/accuracy). (1)

[
  {"claim": "One epoch has 200 updates and 8 epochs have 1600 updates", "code": "N=50000; b=250; per_epoch=N//b; total=per_epoch*8; result = (per_epoch==200 and total==1600)"},
  {"claim": "Adam bias-corrected first moment at t=1 with beta1=0.9, m1=0.2 equals 2.0", "code": "beta1=Rational(9,10); m1=Rational(1,5); mhat=m1/(1-beta1**1); result = (mhat==2)"},
  {"claim": "Inverted dropout scaling with p=0.8 is 1.25", "code": "p=Rational(4,5); scale=1/p; result = (scale==Rational(5,4))"},
  {"claim": "Gradient clip: norm 10 with threshold 4 gives factor 0.4", "code": "g_norm=10; c=4; factor=Rational(c,g_norm); result = (factor==Rational(2,5))"}
]