Training loops from scratch: Worked examples — AI-ML deep dive

Intuition Why a whole page of worked cases?

The parent note built the machinery: forward pass, loss, backward pass, update. But machinery only becomes intuition when you have watched it behave in every situation — when the gradient is huge, when it is exactly zero, when a neuron dies, when the learning rate is chosen badly. This page walks one concrete number through each of those situations so that no case surprises you later.

Everything below uses one tiny network so you can compute every gradient by hand. Keep this picture in your head throughout:

A single input number $x$ , one weight $w$ , one bias $b$ , giving a prediction $\overset{y}{^} = w x + b$ . We compare $\overset{y}{^}$ to the true target $y$ using squared error $L = (\overset{y}{^} - y)^{2}$ . That is the whole world for most of this page — small enough to check by hand, rich enough to show every scenario.

Before any symbol is used, here is the plain-words dictionary.

Definition The seven symbols we compute with

$x$ — the input, one number fed into the network (the horizontal axis in the picture).
$w$ — the weight, the number we are trying to learn; it scales $x$ .
$b$ — the bias, a number added on; it shifts the prediction up or down.
$\overset{y}{^}$ (say "y-hat") — the prediction, $\overset{y}{^} = w x + b$ . The hat means "our guess", to distinguish it from the truth.
$y$ — the target, the correct answer we wish $\overset{y}{^}$ would equal.
$L$ — the loss, a single number saying how wrong we are. We use $L = (\overset{y}{^} - y)^{2}$ : square the miss so that positive and negative misses both count as "bad", and big misses count a lot more than small ones.
$θ$ (say "theta", Greek letter) — a stand-in name for "any parameter we are learning": it means " $w$ , or $b$ , or both at once". We use it when a rule applies to every parameter equally, so we do not have to write $w$ and $b$ twice.

Why $2 (\overset{y}{^} - y)$ and not something else? Because we chose squared loss. The derivative of $(\overset{y}{^} - y)^{2}$ with respect to $\overset{y}{^}$ is $2 (\overset{y}{^} - y)$ by the power rule. The quantity $(\overset{y}{^} - y)$ is the signed error — positive if we overshot, negative if we undershot. Its sign is the whole story of which direction to move, so we track it carefully in every case below.

The scenario matrix

Every training-loop situation you can meet is one of these cells. The examples afterward each fill one (or more) cell(s).

#	Cell class	What is special	Example
A	Overshoot ( $\overset{y}{^} > y$ , error $+$ )	gradient positive → weight decreases	Ex 1
B	Undershoot ( $\overset{y}{^} < y$ , error $-$ )	gradient negative → weight increases	Ex 2
C	Zero-gradient / already correct	update does nothing	Ex 3
D	Degenerate input $x = 0$	$w$ frozen, only $b$ moves	Ex 4
E	Dead ReLU neuron	gradient blocked to zero	Ex 5
F	Learning rate too large (divergence)	loss explodes	Ex 6
G	Mini-batch averaging over mixed signs	opposing errors partly cancel	Ex 7
H	Real-world word problem (multi-epoch)	loop drives loss down over time	Ex 8
I	Exam twist: forgotten `zero_grad`	gradients accumulate	Ex 9

Worked examples

Worked example Example 1 — Cell A: overshoot, positive error

Model $\overset{y}{^} = w x + b$ with $w = 2$ , $b = 0$ . Input $x = 3$ , target $y = 5$ , learning rate $η = 0.1$ . Do one update on $w$ and $b$ .

Forecast: We predict $\overset{y}{^} = 6$ , which is bigger than $5$ . Guess: does $w$ go up or down?

Forward pass. $\overset{y}{^} = w x + b = 2 (3) + 0 = 6$ . Why this step? We cannot know our error until we have a prediction.
Signed error. $e = \overset{y}{^} - y = 6 - 5 = + 1$ . Why this step? The sign $+ 1$ says "we overshot", so we must move down.
Gradients. $\frac{\partial L}{\partial w} = 2 e x = 2 (1) (3) = 6$ ; $\frac{\partial L}{\partial b} = 2 e = 2$ . Why this step? These say the loss rises if we increase $w$ or $b$ — so we go the other way.
Update. $w \leftarrow 2 - 0.1 (6) = 1.4$ ; $b \leftarrow 0 - 0.1 (2) = - 0.2$ . Why this step? Both parameters shrink, pulling the next prediction toward $5$ .

Verify: New $\overset{y}{^} = 1.4 (3) + (- 0.2) = 4.0$ . New error $= 4.0 - 5 = - 1$ . Old loss $= 1^{2} = 1$ , new loss $= (- 1)^{2} = 1$ — equal here because $η$ overshot slightly past the minimum, but note we did move in the correct direction (from $+ 1$ toward and past $5$ ). Sanity check on sign: overshoot ⇒ weight decreased ✓.

Worked example Example 2 — Cell B: undershoot, negative error

Same model start $w = 2$ , $b = 0$ , $η = 0.1$ . Now $x = 3$ , target $y = 8$ .

Forecast: $\overset{y}{^} = 6 < 8$ , we undershot. Weight should go up this time.

Forward. $\overset{y}{^} = 2 (3) + 0 = 6$ . Why this step? We need the current prediction before we can measure how far off we are.
Signed error. $e = 6 - 8 = - 2$ . Why? Negative means undershoot → push up.
Gradients. $\partial_{w} L = 2 (- 2) (3) = - 12$ ; $\partial_{b} L = 2 (- 2) = - 4$ . Why negative? A negative gradient means "increasing $w$ reduces loss", so the update rule (subtract a negative) will add.
Update. $w \leftarrow 2 - 0.1 (- 12) = 3.2$ ; $b \leftarrow 0 - 0.1 (- 4) = 0.4$ .

Verify: New $\overset{y}{^} = 3.2 (3) + 0.4 = 10.0$ , new error $= 10 - 8 = + 2$ . Loss unchanged in size ( $4 \to 4$ ) but sign flipped — again $η$ stepped a touch far, yet direction is right: undershoot ⇒ weight increased ✓. Compare with Ex 1: sign of error flips the sign of the update. That is the entire quadrant story.

Worked example Example 4 — Cell D: degenerate input

$x = 0$ $w = 2$ , $b = 0$ , $x = 0$ , target $y = 4$ , $η = 0.1$ .

Forecast: The input is zero. Which parameter can possibly learn anything?

Forward. $\overset{y}{^} = 2 (0) + 0 = 0$ . Why this step? Even with a zero input we still need a prediction to define the error.
Signed error. $e = 0 - 4 = - 4$ .
Gradients. $\partial_{w} L = 2 (- 4) (0) = 0$ — killed by the factor $x = 0$ ! And $\partial_{b} L = 2 (- 4) = - 8$ . Why this step matters: when $x = 0$ , the weight's gradient always vanishes because $\partial \overset{y}{^} / \partial w = x = 0$ . The weight is frozen; only the bias can respond.
Update. $w \leftarrow 2 - 0.1 (0) = 2$ (unchanged); $b \leftarrow 0 - 0.1 (- 8) = 0.8$ .

Verify: New $\overset{y}{^} = 2 (0) + 0.8 = 0.8$ , closer to $4$ than $0$ was. Only $b$ moved ✓. Lesson: features that are constantly zero teach their weight nothing — a reason to normalise inputs, see 3.4.01-Batch-normalization.

Worked example Example 5 — Cell E: the dead ReLU neuron

A hidden unit computes $a = ReLU (z) = max (0, z)$ , where $ReLU$ passes positives through and clamps negatives to $0$ . Its pre-activation is $z = - 3$ . During backprop an incoming error signal arrives at this neuron's output — call it $δ$ . Here $δ = 2 (\overset{y}{^} - y) W_{2}$ , where $W_{2}$ is the weight on the wire leaving this neuron toward the output. Suppose the output error is $2 (\overset{y}{^} - y) = - 2$ and that outgoing weight is $W_{2} = 0.4$ , so $δ = (- 2) (0.4) = - 0.8$ . What gradient reaches this neuron's weights?

Forecast: The neuron output is $0$ . Does an error signal still flow back through it?

Forward through ReLU. $a = max (0, - 3) = 0$ . Why? Negative input to ReLU is clamped.
ReLU derivative. $\frac{\partial a}{\partial z} = {10 z > 0 z \leq 0$ . Here $z = - 3 \leq 0$ , so the derivative is $0$ . Why this step: backprop multiplies the incoming error $δ$ by this local derivative.
Gradient into the neuron. $\frac{\partial L}{\partial z} = δ \times \frac{\partial a}{\partial z} = (- 0.8) \times 0 = 0$ . Why: the closed ReLU gate blocks the whole signal.
Consequence. Any weight feeding this neuron gets gradient $0$ and never updates — a dead ReLU.

Verify: Multiply chain: $- 0.8 \times 0 = 0$ ✓. Contrast: if instead $z = + 3$ , the derivative is $1$ and the gradient is $- 0.8 \times 1 = - 0.8$ , flowing normally. This is why very negative biases or too-large learning rates can permanently kill neurons — connected to 3.1.02-Backpropagation-algorithm.

Worked example Example 6 — Cell F: learning rate too large → divergence

$w = 2$ , $b = 0$ , $x = 3$ , target $y = 5$ . Take a big $η = 0.3$ and do two updates on $w$ (freeze $b$ for clarity).

Forecast: A big step feels faster. Will the loss go down, or blow up?

Step 1 forward. $\overset{y}{^} = 2 (3) = 6$ , $e = + 1$ , $\partial_{w} L = 2 (1) (3) = 6$ . Why this step? We need the prediction and its error before we can form the gradient.
Step 1 update. $w \leftarrow 2 - 0.3 (6) = 0.2$ . Loss was $1$ . Why watch this: we jumped clear past the good value.
Step 2 forward. $\overset{y}{^} = 0.2 (3) = 0.6$ , $e = 0.6 - 5 = - 4.4$ , $\partial_{w} L = 2 (- 4.4) (3) = - 26.4$ .
Step 2 update. $w \leftarrow 0.2 - 0.3 (- 26.4) = 8.12$ . New $\overset{y}{^} = 8.12 (3) = 24.36$ , error $\approx 19.36$ .

Verify: Loss sequence $1 \to (0.6 - 5)^{2} = 19.36 \to (24.36 - 5)^{2} \approx 374.8$ — growing, not shrinking. This is divergence: the step size beats the curvature and bounces up the parabola walls. Fix: smaller $η$ . With $η = 0.05$ the same first step gives $w = 2 - 0.05 (6) = 1.7$ , $\overset{y}{^} = 5.1$ , loss $0.01$ — converging. See 4.2.01-Gradient-descent-variants.

Worked example Example 7 — Cell G: mini-batch averaging with mixed-sign errors

Two examples in one batch, model $w = 2$ , $b = 0$ :

Example P: $x = 1$ , $y = 1$ → overshoot.
Example Q: $x = 1$ , $y = 5$ → undershoot. Learning rate $η = 0.1$ . Compute the averaged batch gradient for $w$ and update once. We write $g_{P}$ and $g_{Q}$ for the per-example gradient of the loss with respect to $w$ — one gradient number computed from example P, another from example Q.

Forecast: One error is positive, one negative. Will they cancel, and where does the update land?

Forward, both. $\overset{y}{^}_{P} = 2 (1) = 2$ , $\overset{y}{^}_{Q} = 2 (1) = 2$ . Why this step? Each example needs its own prediction before its own error and gradient can be found.
Signed errors. $e_{P} = 2 - 1 = + 1$ ; $e_{Q} = 2 - 5 = - 3$ .
Per-example grads. $g_{P} = 2 (1) (1) = 2$ ; $g_{Q} = 2 (- 3) (1) = - 6$ (each is $2 e_{i} x_{i}$ ). Why average? The parent note's batch rule averages gradients to cut noise: $\overset{g}{ˉ} = \frac{1}{B} \sum g_{i}$ , where $B$ is the batch size — the number of examples in the batch (here $B = 2$ ).
Batch gradient. $\overset{g}{ˉ} = \frac{1}{2} (2 + (- 6)) = - 2$ .
Update. $w \leftarrow 2 - 0.1 (- 2) = 2.2$ .

Verify: The averaged gradient $- 2$ is negative, so $w$ rises — the batch "leans" toward the bigger, undershooting example Q, which had the larger error magnitude ( $3 > 1$ ). New predictions: $\overset{y}{^}_{P} = 2.2$ , $\overset{y}{^}_{Q} = 2.2$ ; combined loss $(2.2 - 1)^{2} + (2.2 - 5)^{2} = 1.44 + 7.84 = 9.28$ vs old $(1)^{2} + (3)^{2} = 10$ → decreased ✓. See 3.5.02-Data-loaders-and-batching.

Worked example Example 8 — Cell H: real-world word problem, multi-step convergence

A shop models daily sales as $\overset{y}{^} = w \cdot (ads spend)$ , no bias. One data point: ads $= 2$ (₹thousand), true sales $y = 6$ (₹thousand). Start $w = 1$ , $η = 0.05$ . Run three update steps and watch the loss fall.

Forecast: Each step should shrink the loss toward $0$ . Guess whether $w$ heads toward $3$ .

Step 1. $\overset{y}{^} = 1 (2) = 2$ , $e = - 4$ , $g = 2 (- 4) (2) = - 16$ , $w \leftarrow 1 - 0.05 (- 16) = 1.8$ . Loss $= 16$ . Why start with the forward value? The prediction $\overset{y}{^} = 2$ is what makes the error $- 4$ concrete and drives the whole step.
Step 2. $\overset{y}{^} = 1.8 (2) = 3.6$ , $e = - 2.4$ , $g = 2 (- 2.4) (2) = - 9.6$ , $w \leftarrow 1.8 - 0.05 (- 9.6) = 2.28$ . Loss $= 5.76$ .
Step 3. $\overset{y}{^} = 2.28 (2) = 4.56$ , $e = - 1.44$ , $g = 2 (- 1.44) (2) = - 5.76$ , $w \leftarrow 2.28 - 0.05 (- 5.76) = 2.568$ . Loss $= 2.0736$ . Why this step-by-step: it shows the loop's heartbeat — each iteration nudges $w$ toward the target-fitting value $w_{best} = y / x = 3$ .

Verify: Loss sequence $16 \to 5.76 \to 2.0736$ , each about $0.36 \times$ the previous — a smooth geometric decay, no divergence. $w$ climbs $1 \to 1.8 \to 2.28 \to 2.568$ , monotonically toward $3$ ✓. This is exactly the healthy convergence the parent note's loop produces.

Worked example Example 9 — Cell I: exam twist, the forgotten

zero_grad A student runs two batches but forgets optimizer.zero_grad(), so PyTorch adds the new gradient onto the old one. Model $w = 2$ , $b = 0$ , $η = 0.1$ .

Batch 1 gradient for $w$ : $g_{1} = + 4$ .
Batch 2 gradient for $w$ : $g_{2} = + 3$ . What update does batch 2 apply, correctly vs. with the bug?

Forecast: With the bug, does batch 2 use $3$ or $3 + 4$ ?

Correct behaviour. After calling zero_grad, the stored gradient is cleared, so batch 2 sees only $g_{2} = 3$ ; update $w \leftarrow 2 - 0.1 (3) = 1.7$ . Why this step? A clean slate means each update reflects only the batch that just ran.
Buggy behaviour. Without zero_grad, the stored gradient is $g_{1} + g_{2} = 4 + 3 = 7$ . Why? PyTorch accumulates gradients by default (handy for gradient accumulation, harmful when unintended).
Buggy update. $w \leftarrow 2 - 0.1 (7) = 1.3$ . Why watch this: the update over-shoots because it carries stale gradient from batch 1.

Verify: Correct $w = 1.7$ , buggy $w = 1.3$ — the bug over-steps by $0.1 \times g_{1} = 0.4$ exactly ✓. Over many batches these accumulations snowball into exploding gradients. Fix: call optimizer.zero_grad() every step, as the parent note's mistake box warns.

Recall Self-check (reveal after guessing)

Overshoot ( $\overset{y}{^} > y$ ) makes the weight go which way? ::: Down — positive error gives a positive gradient, so subtracting it shrinks $w$ . When the input $x = 0$ , which parameter cannot learn? ::: The weight $w$ , because its gradient $2 (\overset{y}{^} - y) x$ carries the factor $x = 0$ . A ReLU neuron with pre-activation $z = - 3$ passes what fraction of the backward signal? ::: Zero — the ReLU derivative is $0$ for $z \leq 0$ (a dead neuron). Why does too-large $η$ diverge? ::: Each step overshoots the parabola's minimum and lands higher on the opposite wall, so loss grows. Forgetting zero_grad does what to gradients? ::: It accumulates (adds) them across batches, over-stepping the update.

Connections: Training loops from scratch · 3.1.02-Backpropagation-algorithm · 3.3.01-PyTorch-fundamentals · 3.3.06-Custom-loss-functions · 3.4.03-Learning-rate-schedules · 3.4.01-Batch-normalization · 3.5.02-Data-loaders-and-batching · 4.2.01-Gradient-descent-variants · 5.1.01-Overfitting-and-underfitting