Exercises — Training loops from scratch

1. Forward pass — compute the prediction $\hat{y}=f(x;\theta )$. Why: we need to know what the current model says before we can judge it.
2. Loss computation — compute the scalar error $\mathcal{L}(\hat{y},y)$. Why: we need a single number that says "how wrong", so we have something to minimize.
3. Backward pass — compute ${\nabla }_{\theta }\mathcal{L}$. Why: we need to know which direction each parameter should move.
4. Parameter update — $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\mathcal{L}$. Why: actually apply the improvement.
5. Iteration — repeat over all batches and epochs. Why: one nudge is tiny; convergence needs many.

See 3.1.02-Backpropagation-algorithm for step 3's machinery.

• optimizer.zero_grad() ::: resets stored gradients to zero — the from-scratch equivalent is simply not accumulating: each batch you compute fresh dW1, dW2, ... from scratch, overwriting the old ones.
• loss.backward() ::: the manual block computing dloss_dy, dW2, db2, da1, dz1, dW1, db1 — the chain-rule pass.
• optimizer.step() ::: the lines W2 -= lr*dW2, b2 -= lr*db2, etc. — the update rule.

See 3.3.01-PyTorch-fundamentals.

${\theta }_{t+1}=5.0-0.1\times 4.0=5.0-0.4=\boxed{4.6}$ The gradient is positive (increasing $\theta$ raises loss), so we step down. Loss should drop.

First average (the $\frac{1}{|{\mathcal{B}}_t|}$ in our formula): $g_t=\frac{2+(-1)+3+0}{4}=\frac{4}{4}=1.0$ Then update: ${\theta }_{t+1}=10-0.5\times 1.0=\boxed{9.5}$ Why average, not sum? Summing would make the step scale with batch size, so changing $B$ would secretly change your effective learning rate. Averaging keeps step size batch-independent — see 3.5.02-Data-loaders-and-batching.

(a) $\mathcal{L}=(7-4{)}^2=3^2=\boxed{9}$. (b) Differentiate: $\frac{d\mathcal{L}}{d\hat{y}}=2(\hat{y}-y)=2(7-4)=\boxed{6}$. Why MSE and its derivative matter: the derivative $2(\hat{y}-y)$ is the seed of backprop — the very first number the backward pass produces, from which every parameter gradient is chained. Positive means "we overshot, pull prediction down".

With accumulation, .grad at step $t$ holds the running sum of all gradients so far: $\text{grad at step }t=g_1+g_2+\cdots +g_t={\sum }_{k=1}^{t}1=t$

• Step 3 uses gradient $=3$.
• Step $t$ uses gradient $=t$.

So the effective step grows linearly: by step 200 you're stepping 200× harder than intended — a runaway (see the figure below). This is why omitting zero_grad() typically explodes the loss.

Reading the symbols: $X$ is the batch input, $d{a}_1$ is "how the loss changes with the hidden activation", $d{z}_1$ is that same signal after passing back through ReLU, and $\odot$ means element-wise multiply. Because the ReLU mask $(z_1>0)$ is $0$ whenever $z_1=-2\le 0$: $d{z}_1=d{a}_1\odot (z_1>0)=d{a}_1\odot 0=0$ $d{W}_1=X^{\top }d{z}_1=X^{\top }0=0$ The incoming weights get zero gradient, so they never update. The unit is stuck outputting $0$ forever — a dead ReLU. It contributes nothing and cannot recover on its own. Why it happens: ReLU flatlines for negatives; a flat region has zero slope, so no learning signal passes back. Small initialization or 3.4.01-Batch-normalization helps keep pre-activations near the "alive" region.

$\eta =1.5$ (too large). Look at the bowl-shaped loss surface below: the update ${\theta }_{t+1}={\theta }_t-\eta g_t$ takes a step proportional to $\eta$. If $\eta$ is too big, each step overshoots the bottom, landing on the opposite wall higher up — so the loss zig-zags and can even diverge. Too small ($0.001$) creeps down slowly but smoothly; well-chosen ($0.1$) descends steadily.

See 3.4.03-Learning-rate-schedules and 4.2.01-Gradient-descent-variants.

shuffle the N examples          # break batch-order correlation
for start = 0, 32, 64, ..., up to N:
    batch = examples[start : start+32]
    y_pred = forward(batch)      # 1. forward
    loss   = MSE(y_pred, y_batch)# 2. loss
    grads  = backward(loss)      # 3. backward (with 1/B baked in)
    params = params - lr * grads # 4. update

Number of updates $=\lceil \frac{1000}{32}\rceil =\lceil 31.25\rceil =\boxed{32}$ updates (the last batch has only $1000-31\times 32=8$ examples). One epoch ≠ one update — it's many.

We want $\frac{\partial \mathcal{L}}{\partial W_2}$. Break it by the chain rule (each link is "how the next thing changes when this changes"): $\frac{\partial \mathcal{L}}{\partial W_2}=\underset{\text{loss→pred}}{\underbrace{\frac{\partial \mathcal{L}}{\partial \hat{y}}}}\cdot \underset{=1}{\underbrace{\frac{\partial \hat{y}}{\partial z_2}}}\cdot \underset{=a_1}{\underbrace{\frac{\partial z_2}{\partial W_2}}}$

• $\frac{\partial \mathcal{L}}{\partial \hat{y}}=\frac{2}{B}(\hat{y}-y)$ (MSE derivative with the batch average). Call this quantity the loss-to-prediction signal, written $g_{\hat{y}}$.
• $\frac{\partial \hat{y}}{\partial z_2}=1$ (output is linear).
• $\frac{\partial z_2}{\partial W_2}=a_1$ (the hidden activations feeding this layer).

In matrix form (summing correctly over the batch): $d{W}_2=a_1^{\top }{g}_{\hat{y}}$, where $a_1^{\top }$ is the transpose of the hidden activations and $g_{\hat{y}}$ is the loss-to-prediction signal from the first bullet. This is 3.1.02-Backpropagation-algorithm in miniature.

$\frac{d\mathcal{L}}{d\hat{y}}=\mathrm{sign}(\hat{y}-y)=\{\begin{array}{ll}+1& \hat{y}>y\\ -1& \hat{y}<y\\ \text{undefined}& \hat{y}=y\end{array}$ Misbehaves at $\hat{y}=y$ (the kink) — the derivative jumps and isn't defined exactly at zero. In practice frameworks set it to $0$ there (a subgradient). Note MAE's gradient magnitude is constant ($\pm 1$), unlike MSE's $2(\hat{y}-y)$ which shrinks near the target — so MAE keeps pushing at full strength even when close. See 3.3.06-Custom-loss-functions.

Two backward calls accumulate: stored grad $=2+2=4$. Then: ${\theta }_{t+1}=3-0.1\times 4=\boxed{2.6}$ Bug or feature? This is the intended mechanism behind gradient accumulation — deliberately summing gradients over several mini-batches to simulate a larger effective batch on limited memory. It's a feature if you divide appropriately (here you'd want to average by dividing by 2 to keep the effective learning rate honest); a bug if you did it by accident. Same math, different intent — which is exactly why zero_grad() is explicit rather than automatic.

Update rule ${\theta }_{t+1}={\theta }_t-\eta {\mathcal{L}}^{\prime }({\theta }_t)={\theta }_t-0.25\cdot 2({\theta }_t-2)={\theta }_t-0.5({\theta }_t-2)$.

• ${\theta }_1=6-0.5(6-2)=6-2=\boxed{4}$.
• ${\theta }_2=4-0.5(4-2)=4-1=\boxed{3}$.

Distance to the minimum ($\theta =2$) halves each step: $4\to 2\to 1\dots$ Yes, it converges smoothly — this is the well-behaved regime of Exercise 3.3. (If instead $\eta =1.5$, the factor becomes $1-3=-2$, and $\theta$ would fling past 2 and diverge.)

Phenomenon: overfitting. Mechanism (one sentence): the model has enough capacity and epochs to memorize the training set's noise, which keeps driving training loss down while destroying generalization, so validation loss — measured on unseen data — turns back up. One fix: early stopping — halt training at the validation minimum (epoch 20) and keep those weights; alternatives are adding regularization or gathering more data. See 5.1.01-Overfitting-and-underfitting.

The essential update rule ::: ${\theta }_{t+1}={\theta }_t-\eta g_t$ with $g_t$ the batch-averaged gradient. Updates in one epoch ::: $\lceil N/B\rceil$. Why divide the batch gradient by $B$ ::: keeps effective learning rate independent of batch size. Symptom of missing zero_grad ::: gradients accumulate, effective step grows each iteration, loss explodes. Symptom of a dead ReLU ::: pre-activation always $\le 0$, mask zeroes the gradient, weights frozen. Training loss down but validation up ::: overfitting → early stopping / regularization.