Mini-batch gradient descent: Visual walkthrough

We are going to earn every symbol before we use it. Let's start with the most basic object of all: a single training example's "which way to nudge" arrow.

Step 1 — What a "gradient" even is, as an arrow

WHAT. Take one training photo $x_{i}$ with its correct answer $y_{i}$ . The network makes a guess; the guess is a bit wrong; the loss $ℓ_{i}$ measures how wrong. The gradient $g_{i}$ is an arrow that points in the direction of parameters that would make the loss bigger — so we step opposite to it.

WHY an arrow. Every network parameter is one axis you can slide. If there are two parameters, "parameter space" is a flat sheet, and the gradient is a genuine arrow on that sheet. We draw the 2-parameter cartoon because your eye can hold two dimensions; the real network has millions of axes, but the logic is identical per axis.

PICTURE. Each dot is one example's gradient arrow $g_{i}$ . They scatter — some point up-right, some down-left. No single arrow is "the truth"; they disagree.

Step 2 — The true gradient is the center of the cloud

WHAT. The quantity we actually want is the average of all $N$ arrows: $true gradient μ = \frac{1}{N} \sum_{i = 1}^{N} one example’s arrow g_{i}, how many examples N$

WHY an average. The whole-dataset loss $J (θ)$ is itself an average of the per-example losses. Slope-of-an-average = average-of-slopes, so the true gradient $μ = \nabla_{θ} J$ is literally the arithmetic mean arrow — the balance point of the whole cloud.

PICTURE. The amber arrow is $μ$ : drop a pin so the cloud balances on it. It sits in the middle even though not one single dot lands exactly there.

Step 3 — A mini-batch is a random handful; its mean is our guess

WHAT. Pick $m$ arrows at random (call this set $B$ , the batch) and average just those: $our guess g_{B} = \frac{1}{m} \sum_{i \in B} a chosen arrow g_{i}, batch size m ≪ N$

WHY random. If we always grabbed the same handful, our guess would be permanently tilted toward that handful's quirks. Grabbing uniformly at random makes every arrow equally likely to be picked — that fairness is what makes the guess honest (Step 4).

PICTURE. Highlight $m = 4$ dots (cyan ring), average only those (cyan arrow). Notice it lands near the amber true arrow but not on it. That gap is the whole story of the rest of the page.

Step 4 — On average, the guess is exactly right (unbiased)

WHAT. Repeat "grab a random handful, take its mean" many times and average those means. You get $μ$ back exactly: $E [g_{B}] = μ = \nabla_{θ} J$

WHY it holds. $E [\cdot]$ (say "expected value") means "the long-run average over all the ways you could have drawn the batch." Each single arrow, drawn uniformly, has long-run average $μ$ (every arrow gets picked equally often). Averaging $m$ things that each average to $μ$ still averages to $μ$ . So there is no systematic tilt — the guess is unbiased.

PICTURE. Many faint cyan batch-mean arrows fan out around the amber true arrow. They lean left and right, but their own center-of-mass sits dead on $μ$ .

Step 5 — How wide is the wobble? Averaging shrinks spread

WHAT. The spread of the individual arrows is the variance $σ^{2}$ (its square root $σ$ is the typical distance of one arrow from the center). The spread of the batch mean is smaller: $Var (g_{B}) = Var (\frac{1}{m} \sum_{i = 1}^{m} g_{i}) = \frac{1}{m ^{2}} \sum_{i = 1}^{m} Var (g_{i}) = \frac{1}{m ^{2}} \cdot m σ^{2} = \frac{σ ^{2}}{m}$

WHY each move.

$\frac{1}{m}$ pulls out of the variance as $\frac{1}{m ^{2}}$ — because variance measures squared spread, and scaling an arrow by $\frac{1}{m}$ scales its spread-squared by $\frac{1}{m ^{2}}$ .
The sum splits into a plain sum of the pieces' variances — this step requires independence (Step 7 is the warning about when it fails). Independent wobbles partly cancel instead of piling up.
There are $m$ identical pieces each of size $σ^{2}$ , giving $m σ^{2}$ . Multiply by $\frac{1}{m ^{2}}$ → $σ^{2} / m$ .

PICTURE. Two clouds of batch-mean arrows: a fat one for small $m$ , a tight one for large $m$ . Same center, different fatness.

Step 6 — Why $1/ m$ means diminishing returns

WHAT. Take the square root of the variance to get the wobble in the same units as the arrows: $σ / m$ . This falls with $m$ , but slowly.

WHY square-root and not $1/ m$ . People instinctively think "double the batch, halve the noise." Not so: the variance falls as $1/ m$ , but the thing you feel — the standard deviation (typical distance) — falls only as $1/ m$ . To halve the wobble you must quadruple the batch (and its compute cost).

PICTURE. Plot wobble $σ / m$ against $m$ . It's a steep drop at first then a long flat tail — the tail is where extra compute buys you almost nothing.

Step 7 — The degenerate cases (never hit a scenario we didn't show)

WHAT & WHY & PICTURE, all four corners of the spectrum:

$m = N$ (full batch). The "handful" is everyone, so $g_{B} = μ$ exactly, wobble $= σ / N \approx 0$ . Smooth path, but one arrow costs a whole dataset. This is plain Gradient Descent.
$m = 1$ (one example). Wobble $= σ / 1 = σ$ — maximal. Very jittery, but each step is dirt cheap and the jitter can bounce you out of bad minima. This is Stochastic Gradient Descent.
Not independent (sorted / unshuffled data). Step 5's split of the sum assumed independence. If every batch is all-cats-then-all-dogs, the arrows inside a batch are correlated, the cancellation fails, and $g_{B}$ is biased toward whatever class you're on. Fix: shuffle every epoch.
Last leftover batch. With $N = 50000, m = 128$ you get $⌈ 50000/128 ⌉ = 391$ steps; the final batch holds only $50000 - 390 \times 128 = 80$ examples — a smaller handful, hence noisier, but still unbiased. Keep it or drop it.

Recall Quick self-check on the edge cases

Wobble at $m = N$ ? ::: essentially zero ( $σ / N$ ).
Why does shuffling matter? ::: it restores the independence that Step 5 needs, keeping the estimate unbiased and low-variance.
Steps per epoch, $N = 50000, m = 128$ ? ::: $391$ (last batch = 80 examples).

The one-picture summary

Everything above in a single diagram: a scattered cloud of per-example arrows ( $g_{i}$ ), their balance point ( $μ$ , amber), one random handful and its mean ( $g_{B}$ , cyan), the fan of possible batch-means centered on $μ$ (unbiased, Step 4), and a shrink-arrow showing that fan tightening as $m$ grows by $1/ m$ (Step 5–6).

Recall Feynman retelling of the whole walkthrough

You want to know which way the whole class is leaning on a question, so you could ask all 5000 kids — but that's slow (Step 2, the true gradient $μ$ ). Instead you ask a random handful of 30 and average their opinions (Step 3, $g_{B}$ ). Any one handful might lean a bit wrong, but if you grabbed handfuls fairly at random, over many handfuls their answers center exactly on the true class opinion — no built-in tilt (Step 4, unbiased). And the more kids per handful, the less each handful wobbles — but sneakily, to cut the wobble in half you need four times as many kids, not twice (Steps 5–6, the $1/ m$ law). This all breaks if you keep asking the same clique or ask kids grouped by neighbourhood, because their answers then move together instead of cancelling — so you reshuffle who's in each handful every round (Step 7). That is mini-batch gradient descent, seen entirely as a cloud of arrows and their balance point.

Connections

Parent: Mini-batch gradient descent — the note this walkthrough visualises.
Stochastic Gradient Descent — the $m = 1$ corner of Step 7.
Gradient Descent — the $m = N$ corner of Step 7.
Bias-Variance Tradeoff — Steps 4–5 are literally "unbiased estimator, variance $σ^{2} / m$ ".
Learning Rate Scheduling — the linear scaling rule that answers Step 6's diminishing returns.
Momentum and Adam — smooth out the Step 5 wobble across steps.
Generalization and Flat Minima — why the leftover wobble (Step 7 case 2) can help.