Interleaved — Phase 3

AI-ML interleaved practice

printable — key stays hidden on paper

Instructions: Solve all problems. Show your work. Use $...$ notation for any math. Each problem tests a different subtopic — read carefully and choose the appropriate method. No calculators needed for conceptual parts; keep arithmetic exact where possible.

1. (6 marks) A single neuron receives inputs $x = [2, -1, 3]$ with weights $w = [0.5, -1.0, 0.25]$ and bias $b = -0.5$ . Compute the pre-activation $z$ , then the output under (a) sigmoid and (b) ReLU. Give $z$ exactly and the ReLU output exactly.

2. (5 marks) You are training a 30-layer plain feedforward network with sigmoid activations and notice the early layers barely update. Name the phenomenon, explain mechanistically why depth + sigmoid causes it, and give two concrete architectural fixes.

3. (7 marks) For the Adam optimizer with $\beta_1 = 0.9$ , $\beta_2 = 0.999$ , $\epsilon = 10^{-8}$ , learning rate $\alpha = 0.1$ : at step $t=1$ the gradient is $g_1 = 0.2$ . Compute the bias-corrected $\hat{m}_1$ , $\hat{v}_1$ , and the resulting parameter update $\Delta\theta$ .

4. (4 marks) A 3-class classifier outputs logits $z = [2.0, 1.0, 0.1]$ . Compute the softmax probabilities (leave in terms of $e$ , then give decimal to 3 dp). Which class is predicted?

5. (6 marks) Given the true one-hot label $y = [0, 1, 0]$ and the predicted distribution from Problem 4, compute the cross-entropy loss. Then state what MSE would (incorrectly) give as its gradient character here, and explain in one sentence why cross-entropy is preferred for classification.

6. (6 marks) Consider the computational graph $L = (a\cdot b + c)^2$ with $a=2, b=3, c=1$ . Using backpropagation on the graph, compute $\partial L/\partial a$ , $\partial L/\partial b$ , and $\partial L/\partial c$ .

7. (5 marks) You initialize a layer with $n_{in} = 400$ input units feeding into a ReLU. State the He initialization variance and Xavier initialization variance, compute both numerically, and say which you should use here and why.

8. (6 marks) A mini-batch of pre-activations for one feature is $[4, 8, 6, 2]$ . Apply batch normalization (with $\gamma=1$ , $\beta=0$ , $\epsilon=0$ ). Compute the mean, variance, and the normalized outputs. Then state the one key difference between batch norm and layer norm regarding what axis the statistics are computed over.

9. (5 marks) Compare SGD-with-momentum ( $\mu=0.9$ ) update to plain SGD. Given plain gradient sequence $g_1 = 1, g_2 = 1, g_3 = 1$ (constant) and learning rate $\eta = 0.1$ , compute the momentum velocity $v_3$ and the resulting step size, versus plain SGD's step. What does this illustrate?

10. (4 marks) State the Universal Approximation Theorem precisely (what class of functions, what network, what conditions), and explain in one sentence why it does not guarantee that gradient descent will find such a network.

11. (4 marks) During dropout with keep-probability $p = 0.8$ , a layer's activations at train time are scaled by inverted dropout. If a surviving neuron has raw activation $a = 5$ , what value is used in the forward pass? What happens at test time?

Answer keyMark scheme & solutions

1. (Tests 3.1.3 Forward propagation + 3.1.4/3.1.5 activations) $z = (2)(0.5) + (-1)(-1.0) + (3)(0.25) + (-0.5) = 1 + 1 + 0.75 - 0.5 = 2.25$ . (a) Sigmoid: $\sigma(2.25) = 1/(1+e^{-2.25}) \approx 0.9046$ . (b) ReLU: $\max(0, 2.25) = 2.25$ . Why: Must recognize forward-prop dot product before applying activation — the choice of activation changes only the final map, not $z$ .

2. (Tests 3.1.11 Vanishing gradients) Phenomenon: vanishing gradients. Sigmoid derivative $\sigma'(z) = \sigma(z)(1-\sigma(z)) \le 0.25$ . Backprop multiplies these across 30 layers; $0.25^{30}\approx 10^{-18}$ , so early-layer gradients underflow to ~0. Fixes: (a) use ReLU/GELU (derivative 1 in positive region), (b) residual/skip connections, (c) batch/layer norm, (d) careful (He) init. (Any two.) Why: The symptom (early layers frozen) points to gradient magnitude decay, not exploding — must distinguish from the exploding case.

3. (Tests 3.2.5 Adam) $m_1 = 0.9(0) + 0.1(0.2) = 0.02$ ; $v_1 = 0.999(0) + 0.001(0.04) = 0.00004$ . Bias correction: $\hat{m}_1 = 0.02/(1-0.9) = 0.2$ ; $\hat{v}_1 = 0.00004/(1-0.999) = 0.04$ . $\Delta\theta = -\alpha \hat{m}_1/(\sqrt{\hat{v}_1}+\epsilon) = -0.1(0.2)/(0.2 + 10^{-8}) \approx -0.1$ . Why: At $t=1$ bias correction is essential; the near-cancellation gives update $\approx -\alpha\cdot\text{sign}(g)$ , showing Adam's normalized early steps.

4. (Tests 3.1.6 Softmax) $e^{2.0}=7.389$ , $e^{1.0}=2.718$ , $e^{0.1}=1.105$ ; sum $=11.212$ . Probs: $[0.659, 0.242, 0.099]$ . Predicted class = class 0. Why: Softmax normalizes exponentiated logits; pick the argmax.

5. (Tests 3.1.8 Cross-entropy) CE $= -\log(p_{\text{true class}}) = -\log(0.242) \approx 1.419$ . MSE would produce gradients that vanish when the sigmoid/softmax saturates (small even when very wrong). Cross-entropy is preferred because its gradient w.r.t. logits is simply $(p - y)$ , giving strong, non-saturating learning signal. Why: Requires selecting the true-class probability (index 1) — a common trap is using the predicted class.

6. (Tests 3.1.9/3.1.10 Backprop on a computational graph) Let $u = ab = 6$ , $s = u + c = 7$ , $L = s^2 = 49$ . $\partial L/\partial s = 2s = 14$ . $\partial s/\partial u = 1$ , $\partial s/\partial c = 1$ . $\partial L/\partial a = 14 \cdot 1 \cdot b = 14\cdot 3 = 42$ . $\partial L/\partial b = 14 \cdot 1 \cdot a = 14\cdot 2 = 28$ . $\partial L/\partial c = 14 \cdot 1 = 14$ . Why: Chain rule through the graph — must track the product node's local grads.

7. (Tests 3.1.12 Weight init) He: $\text{Var} = 2/n_{in} = 2/400 = 0.005$ . Xavier: $\text{Var} = 1/n_{in} = 1/400 = 0.0025$ (or $2/(n_{in}+n_{out})$ variant). Use He because activation is ReLU (accounts for the halved variance from zeroing negatives). Why: Init choice is dictated by activation family — Xavier for tanh/sigmoid, He for ReLU.

8. (Tests 3.2.8 Batch norm) Mean $= (4+8+6+2)/4 = 5$ . Deviations $[-1,3,1,-3]$ , squared $[1,9,1,9]$ , variance $= 20/4 = 5$ . $\sigma = \sqrt5 \approx 2.236$ . Normalized: $[-0.447, 1.342, 0.447, -1.342]$ . Difference: batch norm computes statistics across the batch dimension (per feature); layer norm computes across the feature dimension (per sample). Why: Must normalize over the batch axis here, contrasting with layer norm's per-example axis.

9. (Tests 3.2.3 Momentum) $v_1 = 0.9(0)+1 = 1$ ; $v_2 = 0.9(1)+1 = 1.9$ ; $v_3 = 0.9(1.9)+1 = 2.71$ . Momentum step $= \eta v_3 = 0.1(2.71) = 0.271$ , vs plain SGD step $= \eta g = 0.1$ . Illustrates: momentum accumulates along consistent gradient directions, accelerating (geometric approach to $\eta/(1-\mu)=1$ ). Why: Distinguish accumulating velocity from single-step gradient.

10. (Tests 3.1.7 UAT) UAT: A feedforward network with a single hidden layer containing finitely many neurons and a non-constant, bounded (or non-polynomial) activation can approximate any continuous function on a compact subset of $\mathbb{R}^n$ to arbitrary accuracy. It is an existence theorem — it says weights exist, not that SGD (a local optimizer on a non-convex loss) will find them, nor how many neurons are needed. Why: Precision about "existence vs learnability."

11. (Tests 3.2.10 Dropout) Inverted dropout scales surviving activations by $1/p$ : $5 \times 1/0.8 = 6.25$ . At test time no dropout is applied and no scaling is needed (expectations already match). Why: Inverted dropout puts the correction at train time so test forward pass is unchanged.

[
  {"claim":"Adam step 1 update magnitude ≈ -0.1","code":"m=0.1*0.2; v=0.001*0.04; mhat=m/(1-0.9); vhat=v/(1-0.999); upd=-0.1*mhat/(vhat**0.5+1e-8); result = abs(upd + 0.1) < 1e-3"},
  {"claim":"Backprop grads are dL/da=42, dL/db=28, dL/dc=14","code":"a,b,c=2,3,1; s=a*b+c; dLds=2*s; da=dLds*b; db=dLds*a; dc=dLds; result = (da,db,dc)==(42,28,14)"},
  {"claim":"BatchNorm variance of [4,8,6,2] is 5","code":"xs=[4,8,6,2]; mu=sum(xs)/len(xs); var=sum((x-mu)**2 for x in xs)/len(xs); result = var==5"}
]