Level 1 — RecognitionTraining Deep Networks

Training Deep Networks

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Chapter: 3.2 Training Deep Networks Level: 1 (Recognition) Time limit: 20 minutes Total marks: 30

Section A — Multiple Choice (1 mark each)

Choose the single best answer.

Q1. In pure Stochastic Gradient Descent (SGD), the gradient at each update step is computed using:

(a) the entire training dataset
(b) a single training example
(c) a mini-batch of 32 examples
(d) the validation set

Q2. Compared to full-batch gradient descent, mini-batch gradient descent primarily offers:

(a) exact gradients with no noise
(b) a balance between computational efficiency and gradient noise
(c) guaranteed convergence to the global minimum
(d) no need for a learning rate

Q3. The momentum update introduces a velocity term $v_t = \gamma v_{t-1} + \eta \nabla_\theta J$ . Increasing $\gamma$ generally:

(a) reduces the influence of past gradients
(b) accumulates more past gradients, smoothing the trajectory
(c) has no effect on convergence
(d) sets the learning rate to zero

Q4. The key idea distinguishing Nesterov momentum from classical momentum is that the gradient is evaluated:

(a) at the current position
(b) at a "lookahead" position after applying the velocity
(c) using the Hessian matrix
(d) after normalizing the weights

Q5. AdaGrad adapts the learning rate per parameter by dividing by the square root of:

(a) the current gradient
(b) the accumulated sum of squared past gradients
(c) the number of epochs
(d) the batch size

Q6. A known limitation of AdaGrad that RMSprop addresses is:

(a) it uses too much memory
(b) its accumulated denominator grows monotonically, shrinking the learning rate too much
(c) it cannot handle sparse gradients
(d) it requires labeled data

Q7. Adam combines the ideas of:

(a) dropout and batch norm
(b) momentum (first moment) and RMSprop (second moment)
(c) L1 and L2 regularization
(d) early stopping and warmup

Q8. The main difference between Adam and AdamW is that AdamW:

(a) removes the second moment estimate
(b) decouples weight decay from the gradient-based update
(c) uses a fixed learning rate
(d) does not use bias correction

Q9. Learning rate warmup is typically used to:

(a) increase the batch size gradually
(b) start with a small learning rate and increase it early in training for stability
(c) freeze the first layers
(d) apply dropout only at the start

Q10. During inference (test time), batch normalization uses:

(a) the current batch's mean and variance
(b) running (moving average) estimates of mean and variance from training
(c) zeros for mean and ones for variance
(d) the validation set statistics

Q11. Layer normalization differs from batch normalization mainly because it normalizes across:

(a) the batch dimension
(b) the features of a single sample (independent of batch size)
(c) the epochs
(d) the learning rate

Q12. Dropout with rate $p=0.5$ during training:

(a) removes 50% of the training data
(b) randomly zeros approximately 50% of the units in a layer
(c) halves the learning rate
(d) doubles the number of layers

Q13. Early stopping monitors a metric on the validation set and halts training when:

(a) training loss reaches zero
(b) the validation metric stops improving (begins to worsen)
(c) all weights become zero
(d) the learning rate is warmed up

Q14. L2 weight decay adds a penalty proportional to:

(a) the sum of absolute values of weights
(b) the sum of squared weights
(c) the number of layers
(d) the batch size

Q15. Gradient clipping is most commonly applied to prevent:

(a) vanishing gradients
(b) exploding gradients
(c) overfitting
(d) slow data loading

Section B — Matching (5 marks: ½ mark each)

Match each optimizer/technique (Column X) to its defining feature (Column Y). Write pairs, e.g. A–3.

Column X	Column Y
A. AdaGrad	1. Randomly deactivates neurons during training
B. Momentum	2. Per-parameter LR from full accumulation of squared gradients
C. Dropout	3. Uses velocity term to accelerate along consistent directions
D. Batch Norm	4. Normalizes activations across the batch dimension
E. Data Augmentation	5. Generates transformed copies of training data (flips, crops)

(Column X items B–E correspondingly.)

Section C — True/False WITH Justification (2 marks each: 1 for T/F, 1 for justification)

Q17. "L1 weight decay tends to produce sparse weight vectors, whereas L2 tends to shrink all weights smoothly toward zero." — True or False? Justify.

Q18. "A cosine annealing learning rate schedule keeps the learning rate constant throughout training." — True or False? Justify.

Q19. "Increasing the mini-batch size generally reduces the variance (noise) of the gradient estimate." — True or False? Justify.

Q20. "Grid search always finds better hyperparameters than random search for the same compute budget in high-dimensional spaces." — True or False? Justify.

Answer keyMark scheme & solutions

Section A (1 mark each, 15 marks)

Q1 — (b). Pure SGD uses one example per update; this is what makes updates noisy but cheap. (1)

Q2 — (b). Mini-batch trades exact but expensive full-batch gradients against cheap but noisy single-sample gradients — a practical balance. (1)

Q3 — (b). Higher $\gamma$ (e.g., 0.9→0.99) weights the accumulated past velocity more, smoothing and accelerating the trajectory. (1)

Q4 — (b). Nesterov computes the gradient at the lookahead point $\theta - \gamma v_{t-1}$ , giving a "correction" before committing. (1)

Q5 — (b). AdaGrad scales LR by $1/\sqrt{G_t+\epsilon}$ where $G_t=\sum_i g_i^2$ (accumulated squared gradients). (1)

Q6 — (b). AdaGrad's $G_t$ grows without bound so effective LR decays toward zero; RMSprop uses an exponential moving average instead. (1)

Q7 — (b). Adam maintains 1st moment $m_t$ (momentum) and 2nd moment $v_t$ (RMSprop-style), both bias-corrected. (1)

Q8 — (b). AdamW decouples weight decay: it applies $\theta \leftarrow \theta - \eta\lambda\theta$ separately rather than folding it into the gradient/adaptive denominator. (1)

Q9 — (b). Warmup ramps LR up from a small value to avoid unstable large early updates (esp. with adaptive optimizers / transformers). (1)

Q10 — (b). At inference BN uses stored running mean/variance so outputs are deterministic and batch-independent. (1)

Q11 — (b). LayerNorm normalizes over the feature dimension per sample, so it is independent of batch size (good for RNNs/transformers). (1)

Q12 — (b). Dropout randomly zeros units with probability $p$ during training only. (1)

Q13 — (b). Early stopping halts when validation performance stops improving (patience exceeded), preventing overfitting. (1)

Q14 — (b). L2 penalty is $\frac{\lambda}{2}\sum w^2$ . (1)

Q15 — (b). Clipping caps gradient norm/value to prevent exploding gradients (common in RNNs). (1)

Section B — Matching (5 marks)

A–2 (AdaGrad → full accumulation of squared gradients) (½)
B–3 (Momentum → velocity term) (½)
C–1 (Dropout → deactivates neurons) (½)
D–4 (Batch Norm → across batch dimension) (½)
E–5 (Data Augmentation → transformed copies) (½)

(Remaining ½ marks distributed: each correct pair = 1 mark, total 5. Award 1 per correct pair.)

Section C — True/False + Justification (2 marks each, 8 marks)

Q17 — TRUE. (1) L1 penalty $\lambda\sum|w|$ has a constant-magnitude subgradient that drives small weights exactly to zero (sparsity); L2 penalty $\frac{\lambda}{2}\sum w^2$ has gradient proportional to $w$ , shrinking weights proportionally but rarely to exactly zero. (1)

Q18 — FALSE. (1) Cosine annealing decreases the LR following a half-cosine curve from $\eta_{max}$ toward $\eta_{min}$ over training; it is not constant. (1)

Q19 — TRUE. (1) The mini-batch gradient is an average of $B$ per-sample gradients; variance of the mean scales as $\sigma^2/B$ , so larger $B$ reduces gradient noise. (1)

Q20 — FALSE. (1) In high-dimensional spaces random search often outperforms grid search per unit compute because grid wastes trials on unimportant dimensions, while random search samples important dimensions more densely (Bergstra & Bengio). (1)

[
  {"claim": "Variance of mini-batch gradient mean of B iid samples with per-sample var sigma^2 equals sigma^2/B; larger B reduces it",
   "code": "sigma, B = symbols('sigma B', positive=True); var_mean = sigma**2 / B; from sympy import diff; decreasing = diff(var_mean, B) < 0; result = bool(decreasing.subs({sigma:1, B:2}))"},
  {"claim": "L2 penalty gradient is proportional to w (equals lambda*w for penalty lambda/2*w^2)",
   "code": "w, lam = symbols('w lambda'); pen = lam/2 * w**2; g = diff(pen, w); result = (g == lam*w)"},
  {"claim": "AdaGrad accumulated denominator G_t is nondecreasing since it sums squares",
   "code": "g1, g2, g3 = symbols('g1 g2 g3', real=True); G3 = g1**2 + g2**2 + g3**2; G2 = g1**2 + g2**2; result = bool(simplify(G3 - G2) == g3**2) and True"},
  {"claim": "Cosine annealing eta(t) = eta_min + 0.5*(eta_max-eta_min)*(1+cos(pi*t/T)) decreases from eta_max at t=0 to eta_min at t=T",
   "code": "t, T, emax, emin = symbols('t T emax emin', positive=True); eta = emin + Rational(1,2)*(emax-emin)*(1+cos(pi*t/T)); start = eta.subs(t,0); end = eta.subs(t,T); result = bool(simplify(start-emax)==0 and simplify(end-emin)==0)"}
]