Stochastic gradient descent (SGD): Worked examples

Before anything, recall the one rule everything below uses. In plain words: take your current guess, feel the downhill direction, step a little that way.

We will need two more symbols throughout. $N$ = the total number of training examples in the dataset (its size). $m$ = the mini-batch size, i.e. how many of those $N$ examples we sample on one step. So $m$ ranges from $1$ up to $N$ .

The scenario matrix

Every SGD situation is one of these cells. The examples below are each tagged with the cell(s) they cover.

Cell	Scenario	What could break / surprise
A	Gradient negative ( $g < 0$ )	must move right, sign flip in update
B	Gradient positive ( $g > 0$ )	must move left
C	Gradient zero (at/over the minimum)	step does nothing — degenerate
D	Mini-batch $m = 1$ (pure SGD, max noise)	zig-zag path, chases single points
E	Mini-batch $1 < m < N$ vs full batch ( $N$ = dataset size)	noise reduction $σ^{2} / m$ , cost trade-off
F	Fixed $η$ → bounces forever	never settles exactly
G	Decaying $η$ (Robbins–Monro) → settles	wobbles die out
H	$η$ too large → divergence / overshoot	loss explodes
I	Real-world word problem	translate story → loss → update
J	Exam twist (linear scaling rule / epochs count)	conceptual, not just arithmetic

Prerequisites worth a glance: Gradient Descent (the deterministic parent), Learning Rate Schedules (cell G/H), Batch Size and Generalization (cell E/J), Backpropagation (how real $\nabla ℓ_{i}$ is computed).

Worked Examples

For examples 1–5 we reuse one tiny, fully transparent world so you can see every number:

Figure — Stochastic gradient descent (SGD)

Figure s01 — The two dashed parabolas are each single-point losses (cyan $x_{1} = 2$ , amber $x_{2} = 4$ ); the solid white curve is their average, the full loss. Its lowest point sits at the mean $θ^{*} = 3$ . Reading it: SGD samples one dashed curve at a time, but on average it descends the white one.

Worked example Example 4 — Pure SGD zig-zag vs one clean full-batch step (Cells D, E)

$η = 0.5$ , start $θ_{0} = 0$ . Sequence of draws: $x_{1} = 2$ then $x_{2} = 4$ . Compare to full-batch.

Forecast: will two noisy single-point steps land near where one full-batch step lands ( $= 1.5$ , from the parent note)?

Step on $x_{1} = 2$ : $g = 0 - 2 = - 2$ , $θ_{1} = 0 - 0.5 (- 2) = 1$ .
Step on $x_{2} = 4$ : $g = 1 - 4 = - 3$ , $θ_{2} = 1 - 0.5 (- 3) = 2.5$ . Why these steps? Each chases a single point, so the path zig-zags — but both points sit either side of the true mean $3$ , so the walk drifts toward $3$ .
Full-batch reference: $g = \frac{1}{2} [(0 - 2) + (0 - 4)] = - 3$ , one step $θ = 0 - 0.5 (- 3) = 1.5$ . Why this step? The full gradient averages both points before moving — one smooth stride, no wobble.

Figure s02 — Cyan: two pure-SGD steps ( $0 \to 1 \to 2.5$ ), each arrow chasing a single point, so the path bends (zig-zag). Amber: one full-batch step ( $0 \to 1.5$ ) going straight. Both aim at the white line $θ^{*} = 3$ ; SGD gets closer here only because it took two steps.

Verify: after two SGD steps $θ_{2} = 2.5$ , closer to $θ^{*} = 3$ than the single full-batch $1.5$ — because SGD took two steps. Both are heading to $3$ . ✓

Worked example Example 5 — Variance vs batch size, the diminishing return (Cell E)

Per-example gradient variance $σ^{2} = 8$ . Compare noise (variance and std-dev) for $m \in {1, 4, 16, 64}$ .

Forecast: going from $m = 16$ to $m = 64$ is 4× the compute. Does it cut the noise 4×?

Variance $= σ^{2} / m$ : $8, 2, 0.5, 0.125$ . Why this step? Averaging $m$ independent noisy slopes divides variance by $m$ (parent note derivation).
Std-dev $= σ / m$ : $2.828, 1.414, 0.707, 0.354$ . Why this step? What we actually feel as jitter is the std-dev, and it only falls as $m$ .
From $m = 16 \to 64$ : variance $0.5 \to 0.125$ (4× better), but std-dev $0.707 \to 0.354$ — only 2× better for 4× the cost. Why this step? $4 = 2$ . This $\cdot$ is why huge batches give diminishing returns.

Verify: $8/16 = 0.5$ , $0.5 \approx 0.7071$ ; $8/64 = 0.125$ , $0.125 \approx 0.3536$ ; ratio $0.7071/0.3536 = 2$ . ✓

Worked example Example 6 — Fixed

$η$ bounces, decaying $η$ settles (Cells F, G) Full-batch slope is $g (θ) = θ - 3$ . But pretend each measurement carries a fixed noise of $\pm 1$ (a stand-in for mini-batch noise). Let the step counter $t$ start at $t = 0$ on the first update and increase by $1$ each update ( $t = 0, 1, 2, \dots$ ). Compare $η = 0.5$ constant vs the decaying schedule $η_{t} = 0.5/ (1 + t)$ , which therefore gives $η_{0} = 0.5, η_{1} = 0.25, η_{2} = 0.1 \overline{6}, \dots$

Forecast: near $θ = 3$ the true slope is tiny, but the noise is not. Which schedule stops moving?

Constant $η = 0.5$ , sitting at $θ = 3$ . True slope $0$ , but a noisy draw gives $g = + 1$ . Update: $θ = 3 - 0.5 (1) = 2.5$ . Next noisy draw $g = - 1$ : $θ = 2.5 - 0.5 (- 1) = 3$ . It oscillates in a ball of radius $\approx η \cdot (noise) = 0.5$ . Why this step? The noise never shrinks and $η$ never shrinks, so the product — the wobble — never dies. Cell F.
Decaying $η_{t} = 0.5/ (1 + t)$ . The same $\pm 1$ noise gets multiplied by an ever-smaller step: at $t = 0, 1, 2, \dots$ the wobble is $0.5, 0.25, 0.167, \dots \to 0$ . Why this step? Robbins–Monro: $\sum_{t} η_{t} = \infty$ (the steps still add up to an infinite total distance, so we can reach the minimum no matter how far) yet $\sum_{t} η_{t}^{2} < \infty$ (the squared steps add to a finite total). The noise enters the parameter's variance as a sum of $η_{t}^{2} σ^{2}$ terms; because $\sum_{t} η_{t}^{2}$ is finite, that accumulated noise-variance is bounded and its tail $\to 0$ , so late-stage wobble dies out and $θ$ settles. Cell G.

Figure s03 — Amber trace: constant $η = 0.5$ keeps jittering in a band of half-width $\approx 0.5$ around the white $θ^{*} = 3$ line forever. Cyan trace: the decaying schedule $η_{t} = 0.5/ (1 + t)$ shrinks each wobble ( $0.5, 0.25, 0.167, \dots$ ) until the curve hugs the line. Same noise sequence for both — only the step size differs.

Verify: constant- $η$ wobble radius $= η \cdot ∣ noise ∣ = 0.5 \cdot 1 = 0.5$ , non-zero forever. Decaying wobble at $t = 2$ is $0.5/ (1 + 2) \cdot 1 = 0.1 \overline{6} < 0.5$ and $\to 0$ . ✓

Worked example Example 7 — Learning rate too large, divergence (Cell H)

Sandbox full-batch slope $g (θ) = θ - 3$ . Start $θ_{0} = 0$ . Try $η = 2.5$ .

Forecast: a big step is faster… unless it overshoots so hard the loss grows. Watch $θ_{0} = 0 \to ?$

Step: $g = 0 - 3 = - 3$ , $θ_{1} = 0 - 2.5 (- 3) = 7.5$ . Why this step? We aimed at $3$ but $η = 2.5$ launched us past it to $7.5$ (distance from min: $0 \to 3$ , now $3 \to 4.5$ — grew).
Next: $g = 7.5 - 3 = 4.5$ , $θ_{2} = 7.5 - 2.5 (4.5) = 7.5 - 11.25 = - 3.75$ . Why this step? Now we overshoot the other way, further out (distance $4.5 \to 6.75$ ). The magnitude of the error is multiplied by $∣1 - η ∣ = ∣1 - 2.5∣ = 1.5 > 1$ each step — it explodes.

Verify: error at min $= θ - 3$ . Sequence: $- 3, 4.5, - 6.75$ ; ratios $4.5/ - 3 = - 1.5$ , $- 6.75/4.5 = - 1.5$ . Magnitude grows by $1.5$ each step ⇒ divergence.

A general safety rule lives behind this. Define $L$ = the gradient's steepness, i.e. how fast the slope $g$ itself changes per unit move in $θ$ (formally the Lipschitz constant of the gradient — the largest rate at which $g$ can change). For our sandbox $g (θ) = θ - 3$ , moving $θ$ by $1$ changes $g$ by exactly $1$ , so $L = 1$ . The stability condition is $η < 2/ L$ ; here that means $η < 2$ , and our $η = 2.5$ breaks it — matching the explosion we just saw. ✓

Worked example Example 8 — Real-world word problem (Cell I)

A weather app predicts tomorrow's temperature as a single number $θ$ . Over three days the true temps were $x = {20, 22, 24}$ (°C), each with squared-error loss $ℓ = \frac{1}{2} (θ - x)^{2}$ . Start $θ_{0} = 18$ , $η = 0.3$ , pure SGD ( $m = 1$ ), draws in order $20, 22, 24$ .

Forecast: the best single guess is the average $22$ . Does one epoch of SGD march $θ$ from $18$ toward $22$ ?

Draw $20$ : $g = 18 - 20 = - 2$ , $θ = 18 - 0.3 (- 2) = 18.6$ . Why this step? Prediction $18$ is too cold vs $20$ ⇒ nudge warmer. Units: °C, step $= η \cdot g = 0.3 \cdot 2 = 0.6$ °C. ✓ units consistent.
Draw $22$ : $g = 18.6 - 22 = - 3.4$ , $θ = 18.6 - 0.3 (- 3.4) = 19.62$ .
Draw $24$ : $g = 19.62 - 24 = - 4.38$ , $θ = 19.62 - 0.3 (- 4.38) = 20.934$ . Why these steps? Every day the guess is still below the true temp, so each step warms it further, climbing toward the mean $22$ .

Verify: after one epoch $θ = 20.934$ , up from $18$ , heading to $22$ but not there yet — this epoch happens to march monotonically upward because $θ$ stayed below all three targets, so no bouncing appears here (that asymptotic oscillation is cell F's story, seen in Example 6). One epoch of these steps approaches the mean without reaching it. ✓

Worked example Example 9 — Exam twist: epochs, updates, and the linear scaling rule (Cell J)

Dataset $N = 50, 000$ . (a) With batch size $m = 250$ , how many parameter updates are in $5$ epochs? (b) A recipe tuned at $m = 250$ used $η = 0.1$ . You switch to $m = 1000$ . The linear scaling rule says $η \propto m$ — what new $η$ ?

Forecast: updates per epoch $= N / m$ ; the rule scales $η$ with the batch-size ratio.

Updates per epoch $= N / m = 50000/250 = 200$ . Why this step? One epoch is one sweep; each sweep is $N / m$ mini-batches, each mini-batch = one update.
Total updates $= 200 \times 5 = 1000$ . Why this step? Epochs multiply the per-epoch count.
Batch ratio $= 1000/250 = 4$ , so $η_{new} = 4 \times 0.1 = 0.4$ . Why this step (why linear, not $\cdot$ )? The $m$ appears only for noise magnitude. The linear rule is about total progress, not noise: a $4 \times$ bigger batch means you take $4 \times$ fewer updates per epoch. To cover the same ground per epoch, each update must move $4 \times$ as far — and the distance moved is $η \cdot g$ , so $η$ must grow linearly by the same factor $4$ . That is why $η \propto m$ , not $η \propto m$ . (See Batch Size and Generalization.)

Verify: $50000/250 = 200$ ; $200 \cdot 5 = 1000$ ; $0.1 \cdot (1000/250) = 0.4$ . ✓

Recall Which cell was hardest for you?

The zero-gradient degenerate case (C) and the divergence case (H) are the two that trip people up. In cell C, does the parameter move? ::: No — the step is $η \cdot 0 = 0$ ; only the next mini-batch (with nonzero slope) can move it. In cell H, what quantity multiplies the error each step for our sandbox? ::: $∣1 - η ∣$ ; if that exceeds $1$ (here $η > 2$ ), the error grows. In general the stability bound is $η < 2/ L$ , where $L$ is the gradient's steepness (Lipschitz constant of $\nabla L$ ); our toy has $L = 1$ , giving $η < 2$ .

Related deep dives: Momentum and Nesterov (smoothing the zig-zag of Example 4), Adam and Adaptive Optimizers (per-parameter $η$ ), Bias-Variance Tradeoff (why flat minima from noise generalize).