Training loops from scratch: Visual walkthrough

Everything below builds toward this single sentence: we repeatedly nudge the knobs of a machine so its output error rolls downhill. Let us earn every word of that.

Step 1 — What is a "loss landscape"? (the hill we roll down)

WHAT. Imagine our whole neural network has just one knob we can turn. Call the knob's setting $θ$ (theta) — it is simply a number we are allowed to choose. For every choice of $θ$ , the network makes some predictions, and those predictions are wrong by some amount. We collect that "wrongness" into a single number called the loss, written $L$ .

If we plot loss (up) against the knob setting $θ$ (sideways), we get a curve — a landscape of hills and valleys.

WHY a picture like this. We cannot "see" a million-knob network, but the logic is identical to one knob: training means finding the bottom of a valley. The picture makes the goal concrete — the lowest point is the best knob setting.

Reading the equation. The loss depends on the knob: $L (θ)$

$L$ — the height of the curve = how wrong we are (0 = perfect).
$(θ)$ — "as a function of the knob" = slide the knob, the height changes.

Step 2 — Which way is downhill? (the derivative = the slope)

WHAT. Standing on the hill at some $θ$ , we ask: if I nudge the knob a tiny bit to the right, does the loss go up or down? The answer is the slope of the curve at that point — its steepness and its direction. This slope is called the derivative, written $\frac{d L}{d θ}$ .

WHY the derivative and not something else. We need a tool that answers exactly one question: "which direction lowers the loss, and how fast?" The derivative is the only tool that gives the local direction of steepest change. Read the picture:

The orange tangent line touches the curve at our point. Its tilt IS the derivative.
Tilt going up to the right (positive slope) → the valley is to the left.
Tilt going down to the right (negative slope) → the valley is to the right.

Reading the equation. $\frac{d L}{d θ} = slope of the loss curve at θ$

$d L$ — a tiny change in loss.
$d θ$ — a tiny change in knob.
their ratio — "how much loss changes per unit of knob change" = steepness.

The rule for finding this slope for a whole network is backpropagation — we borrow it here as a machine that hands us the slope.

Step 3 — The downhill step (why we subtract the gradient)

WHAT. The gradient arrow points uphill (toward more loss). We want less loss, so we walk the opposite way — we subtract it. We take a small step, look at the new slope, subtract again, repeat.

WHY subtract, and why small. The gradient only tells us the direction right here; walk too far and you overshoot the valley and climb the far wall. So we scale the step by a small number $η$ (eta), the learning rate. In the picture, the blue dots march down the hill, each step a scaled hop opposite the arrow.

Reading the equation (the heart of training). $θ_{t + 1} = θ_{t} - η \nabla_{θ} L (θ_{t})$

$θ_{t}$ — knob setting right now (step number $t$ ).
$θ_{t + 1}$ — knob setting after this step.
$-$ — the minus that turns "uphill" into "downhill".
$η$ — step size; too big = overshoot, too small = crawl (tuned via learning-rate schedules).
$\nabla_{θ} L$ — the uphill arrow we flip.

Step 4 — Why not use the whole dataset every step? (mini-batches)

WHAT. The true loss is the average over all $N$ training examples. Computing its exact gradient means looking at every example before taking one step — slow. Instead we grab a small random handful, a mini-batch of size $B$ , and use its average gradient as a cheap estimate.

WHY a handful. Look at the picture: the green path (full-dataset gradient) is smooth but each step is expensive. The orange path (mini-batch) is jittery — a noisy guess of the true direction — but each step is cheap, so we take far more of them and still reach the valley. This is stochastic gradient descent. Single examples (one point) are too jittery; the whole set is too slow; the batch is the sweet spot (see data loaders).

Reading the equation. $\nabla_{θ} L \approx \frac{1}{B} \sum_{i \in B} \nabla_{θ} L_{i}$

$B$ — the current random handful of examples.
$B$ — how many are in it.
$\sum_{i \in B}$ — add up each example's uphill arrow.
$\frac{1}{B}$ — average them, so the step size doesn't depend on batch size.
$\approx$ — "roughly equals": a noisy but unbiased guess of the true gradient.

Step 5 — The full loop as a cycle (four beats, forever)

WHAT. Put steps 1–4 into a repeating cycle. Each turn of the loop: predict → measure → find slopes → nudge knobs. Repeat for every batch, and repeat the whole pass over the data (each pass is one epoch).

WHY a cycle. One step barely moves us; the repetition is what turns random weights into a trained model. The picture is a clock with four ticks — follow the arrows around and around.

Reading the loop, tick by tick. $forward \overset{y}{^} = f (x; θ) \to loss L (\overset{y}{^}, y) \to backward \nabla_{θ} L \to update θ \leftarrow θ - η \nabla_{θ} L$

$\overset{y}{^} = f (x; θ)$ — push input $x$ through the network to get prediction $\overset{y}{^}$ .
$L (\overset{y}{^}, y)$ — compare prediction $\overset{y}{^}$ to truth $y$ ; get one number.
$\nabla_{θ} L$ — ask backprop which knobs caused the error.
$θ \leftarrow θ - η \nabla_{θ} L$ — nudge every knob downhill.

Where these live in code: forward + loss come from your model and loss function, the backward beat is loss.backward(), and the update is optimizer.step() (see PyTorch fundamentals).

Step 6 — Edge case A: the flat spot (zero gradient)

WHAT. What happens when the slope is exactly zero? The gradient arrow has no length, so the update $θ - η \cdot 0 = θ$ changes nothing. We are stuck.

WHY it matters — three flavours (all in the figure).

Green point — bottom of a valley (a minimum): stuck here is good, we've arrived.
Orange point — top of a bump (a maximum): stuck here is bad but unstable; batch noise (Step 4) usually kicks us off.
Gray region — a plateau, a long flat stretch: gradient near zero for many steps, learning crawls. This is one real reason training "stalls."

Reading the degenerate update. $θ_{t + 1} = θ_{t} - η \cdot 0 = θ_{t}$

the gradient is $0$ → the correction vanishes → knob frozen.

Step 7 — Edge case B: the dead ReLU (a sign-dependent zero)

WHAT. Many networks use the ReLU gate: $ReLU (z) = max (0, z)$ — it passes positive inputs unchanged and clamps negatives to zero. Its slope depends on the sign of $z$ :

WHY both signs matter. During backprop we multiply by the ReLU's slope. Read the picture:

$z > 0$ (blue region): slope $= 1$ , gradient flows through untouched.
$z < 0$ (red region): slope $= 0$ , gradient is killed — that neuron sends no learning signal.
$z = 0$ (the corner): slope is undefined; by convention we use $0$ .

If a neuron gets stuck with $z < 0$ for every input, its incoming knobs never receive a gradient — a dead ReLU. This is the sign-based cousin of Step 6's zero, and it's why we initialise weights small and sometimes use batch normalization to keep $z$ near the active region.

Reading the piecewise slope. $\frac{d}{d z} ReLU (z) = {10 z > 0 z < 0$

top line — positive input, gate open, gradient survives.
bottom line — negative input, gate shut, gradient blocked. In code this is the term (z1 > 0) multiplying the backward signal.

Step 8 — Edge case C: the exploding sum (why we zero gradients)

WHAT. Frameworks like PyTorch add each new batch's gradient onto whatever is already stored. If we forget to reset, batch 2's gradient stacks on batch 1's, batch 3 on both, and so on.

WHY it explodes. The stored gradient grows like a running total. The picture shows the red bars ballooning when we forget to reset, versus the blue bars staying correct when we reset each step. A huge gradient means a huge step — we leap out of the valley entirely (this looks identical to the "loss became NaN" disaster and worsens training instability).

Reading the two behaviours. $forgot reset: g_{stored} = g_{1} + g_{2} + \dots + g_{t} vs. with reset: g_{stored} = g_{t}$

left — old gradients pile up → step too big → divergence.
right — only the current batch's gradient → correct step.

The one-picture summary

This final figure compresses the whole walkthrough: a ball starts high on the loss hill (Step 1), reads the slope (Step 2), hops downhill by $- η \nabla$ (Step 3) using jittery mini-batch estimates (Step 4), repeats the four-beat cycle (Step 5), and must survive the three traps — flat spots (Step 6), dead ReLUs (Step 7), and exploding un-reset gradients (Step 8).

Recall Feynman retelling — say it like a story

Picture a ball sitting on a bumpy hill. The height is how wrong our machine is (the loss). We want the ball at the lowest valley — that's the best setting of the knobs.

At each moment we feel the ground under the ball: is it tilted? That tilt is the gradient. The ball rolls opposite to the uphill tilt, taking a small hop whose size is the learning rate. Too big a hop and it flies over the valley; too small and it barely moves.

Feeling the exact tilt using all our data every hop is slow, so we feel it using a small random handful — the direction is a bit shaky but we hop far more often and still get down.

That's the loop: guess (forward), see how wrong (loss), feel the tilt (backward), hop downhill (update) — over and over.

Three things can trap the ball: a flat spot where the ground is level so it stops (a valley = good, a plateau = stuck); a dead gate (a ReLU whose input stayed negative, so its neuron feels nothing); and exploding hops if we forget to wipe the old tilt before feeling the new one, making each hop bigger than the last until the ball flies off the map.

Recall Quick self-check

What direction do we move the weights, relative to the gradient? ::: Opposite — we subtract $η \nabla_{θ} L$ because the gradient points uphill. Why average the batch gradient instead of summing it? ::: So the step size doesn't depend on batch size; summing makes bigger batches push proportionally harder. What is the slope of ReLU for a negative input? ::: Zero — the gate is shut and no gradient flows, which can cause a dead neuron. Why must we call zero_grad each step? ::: Because gradients accumulate; without a reset old batch gradients pile up and the step explodes. A plateau gives near-zero gradient — is the loss curve at a minimum? ::: Not necessarily; a flat plateau also has near-zero slope but is not the bottom, so learning crawls without being finished.