Mini-batch gradient descent: Worked examples — AI-ML deep dive

The scenario matrix

Every question this topic can throw at you falls into one of these case classes. Each row names the class; the last column says which worked example covers it.

#	Case class	What makes it tricky	Covered by
A	Steps-per-epoch, clean divide ( $N / m$ whole)	none — warm-up	Ex 1
B	Steps-per-epoch, remainder batch ( $N / m$ not whole)	ceiling vs floor, `drop_last`	Ex 2
C	Noise scaling across two batch sizes	the $1/ m$ law, diminishing returns	Ex 3
D	Degenerate: $m = 1$ (pure SGD limit)	max noise, cheapest step	Ex 4
E	Degenerate: $m = N$ (full-batch limit)	zero mini-batch noise, one update/epoch	Ex 4
F	Unbiasedness by enumeration (all batches averaged)	each batch wrong, ensemble right	Ex 5
G	Linear scaling rule (change $m$ , change $η$ )	keep effective step comparable	Ex 6
H	Word problem (real training budget)	translate wall-clock into updates	Ex 7
I	Exam twist: how big must $m$ be for target noise?	invert the $1/ m$ law	Ex 8

Worked Examples

Ex 1 — Clean divide (cell A)

Worked example Statement

Dataset $N = 60, 000$ , batch size $m = 100$ . How many gradient updates per epoch? How many in 5 epochs?

Forecast: guess before reading — will it be more or fewer than 100 updates per epoch?

Step 1. Steps per epoch $= ⌈ N / m ⌉ = ⌈ 60000/100 ⌉ = ⌈ 600 ⌉ = 600$ . Why this step? One step = one batch processed = one call to $θ_{t + 1} = θ_{t} - η g_{B}$ . An epoch is one full sweep through all $N$ examples, so it contains exactly (number of batches) steps.

Step 2. Here $60000/100 = 600$ exactly, so no remainder — the ceiling $⌈ \cdot ⌉$ changes nothing. Why this step? We check for a leftover batch before trusting the division; clean divides are the easy case (cell A) precisely because there is no leftover.

Step 3. In 5 epochs: $5 \times 600 = 3000$ updates. Why this step? Epochs just repeat the sweep, so updates multiply.

Verify: $600 \times 100 = 60000 = N$ ✓ (every example used exactly once, no leftover). Units: (examples ÷ examples/step) = steps ✓.

Ex 2 — Remainder batch (cell B)

Worked example Statement

$N = 50, 000$ , $m = 128$ . Steps per epoch if we keep the last short batch? And if drop_last=True?

Forecast: will keeping vs dropping differ by one step, or by many?

Step 1. Full batches $= ⌊ 50000/128 ⌋ = ⌊ 390.625 ⌋ = 390$ . Why this step? The floor $⌊ \cdot ⌋$ counts how many complete batches of 128 we can carve out.

Step 2. Leftover $= 50000 - 390 \times 128 = 50000 - 49920 = 80$ examples. Why this step? Whatever the 390 full batches don't consume is the remainder — here 80 examples, a partial batch.

Step 3. Keep it → $⌈ 50000/128 ⌉ = 391$ steps. Drop it → $390$ steps. Why this step? drop_last=True throws away the partial batch (so its gradient estimate isn't computed on a weirdly-small, higher-variance sample). Keeping it gives one extra, slightly noisier update.

Verify: $390 \times 128 + 80 = 50000$ ✓. And $⌈ 390.625 ⌉ = 391 = 390 + 1$ (one extra step for the remainder) ✓.

Ex 3 — Noise scaling across batch sizes (cell C)

Worked example Statement

Per-example gradient standard deviation $σ = 8$ (per coordinate). Compare the mini-batch gradient noise for $m = 4, 16, 64, 256$ .

Forecast: each time we multiply $m$ by 4, does the noise drop by 4, or by less?

Step 1. Recall the law from the parent note: $std (g_{B}) = σ / m$ . Why this step? This is the whole reason batch size controls noise — it comes from the variance-of-a-mean rule $Var (g_{B}) = σ^{2} / m$ .

Step 2. Plug in each $m$ : $\frac{8}{4} = 4, \frac{8}{16} = 2, \frac{8}{64} = 1, \frac{8}{256} = 0.5$ Why this step? Just evaluating $σ / m$ at each dial setting.

Step 3. Notice: $m$ went up $4 \times$ each time ( $4 \to 16 \to 64 \to 256$ ) but noise only halved each time ( $4 \to 2 \to 1 \to 0.5$ ). Why this step? $4 = 2$ , so quadrupling $m$ divides noise by $2$ — this diminishing return is exactly why huge batches waste compute.

Verify: ratio $\frac{std ( m = 4 )}{std ( m = 256 )} = 4/0.5 = 8 = 256/4 = 64 = 8$ ✓.

Ex 4 — The two degenerate limits (cells D and E)

Worked example Statement

$N = 1000$ , $σ = 5$ . (a) The $m = 1$ extreme (SGD): steps per epoch and noise? (b) The $m = N$ extreme (full-batch GD): steps per epoch and noise?

Forecast: which extreme gives 1000 updates per epoch, and which gives exactly 1?

Step 1 (a, $m = 1$ ). Steps $= ⌈ 1000/1 ⌉ = 1000$ per epoch. Noise $= 5/ 1 = 5$ . Why this step? With one example per batch you update after every example → maximum number of steps, but each gradient is a single sample → maximum noise (equal to the raw $σ$ ).

Step 2 (b, $m = N = 1000$ ). Steps $= ⌈ 1000/1000 ⌉ = 1$ per epoch. Noise $= 5/ 1000 \approx 0.158$ . Why this step? One batch = the whole dataset → the estimate is the true average, so mini-batch sampling noise collapses to its smallest possible value; and you get just one update per epoch.

Step 3. Mini-batch (say $m = 100$ ) sits between: $⌈ 1000/100 ⌉ = 10$ steps, noise $5/ 100 = 0.5$ . Why this step? This shows the whole spectrum in one dataset: $m$ trades step count against noise — the sweet spot lives between the two degenerate ends.

Verify: $m = 1$ gives 1000 steps and noise 5; $m = N$ gives 1 step and noise $5/ 1000 = 0.1581$ ; product (steps × noise) shrinks as $m$ grows, confirming diminishing marginal usefulness of extra samples ✓.

Ex 5 — Unbiasedness by full enumeration (cell F)

Worked example Statement

Four per-example gradients: $g = [3, - 1, 5, - 3]$ . Batch size $m = 2$ , drawn without replacement, all pairs equally likely. Show the mini-batch estimator is unbiased.

Forecast: guess the true gradient first, then whether the average of all batch estimates lands on it.

Step 1. True gradient $= \frac{3 - 1 + 5 - 3}{4} = \frac{4}{4} = 1$ . Why this step? The target is the full-dataset mean $\nabla J$ ; everything else is an estimate of this.

Step 2. List all $(2 4) = 6$ pairs and their means $g_{B} = \frac{g _{i} + g _{j}}{2}$ : ${1, 2} : 1, {1, 3} : 4, {1, 4} : 0, {2, 3} : 2, {2, 4} : - 2, {3, 4} : 1$ Why this step? Enumerating every equally-likely batch lets us compute the exact expectation, not a Monte-Carlo guess.

Step 3. Average the six estimates: $\frac{1 + 4 + 0 + 2 - 2 + 1}{6} = \frac{6}{6} = 1$ . Why this step? $E [g_{B}]$ is the mean over all possible batches — and it equals the true gradient $1$ , proving unbiasedness concretely.

Verify: individual batches range from $- 2$ to $4$ (each single step can mislead), yet their ensemble mean is exactly $1 = \nabla J$ ✓.

Ex 6 — Linear scaling rule (cell G)

Worked example Statement

You trained fine with $m = 64$ , learning rate $η = 0.05$ . You now switch to $m = 256$ to use bigger GPUs. What $η$ keeps the effective step size comparable, per the linear scaling rule?

Forecast: should $η$ go up or down when the batch gets bigger?

Step 1. Scaling factor $k = m_{new} / m_{old} = 256/64 = 4$ . Why this step? The rule keys off how many times bigger the batch is, not the raw sizes.

Step 2. Linear scaling rule: $η_{new} = k \cdot η_{old} = 4 \times 0.05 = 0.20$ . Why this step? A $4 \times$ bigger batch has $4 = 2 \times$ lower gradient noise, so it's safe to take a proportionally bolder step. Multiplying $η$ by $k$ makes the total progress per epoch match. (Pair with warmup — see Learning Rate Scheduling.)

Step 3. Sanity: progress per epoch $\approx (steps) \times η$ . Old: $\frac{N}{64} \times 0.05$ . New: $\frac{N}{256} \times 0.20 = \frac{N}{256} \times 4 \times 0.05 = \frac{N}{64} \times 0.05$ . Why this step? Shows the two schedules cover the same "distance" per epoch — that's why the rule is linear.

Verify: $\frac{N}{64} \cdot 0.05 = \frac{N}{256} \cdot 0.20$ ⟺ $\frac{0.05}{64} = \frac{0.20}{256}$ ⟺ $7.8125 \times 1 0^{- 4} = 7.8125 \times 1 0^{- 4}$ ✓.

Ex 7 — Word problem: a real training budget (cell H)

Worked example Statement

You have $N = 1, 200, 000$ images and $m = 256$ . One gradient step (forward+backward on a batch) takes $0.08$ seconds. You want to train for $90$ epochs. How many total updates, and how long in hours?

Forecast: guess the order of magnitude of hours — 1? 10? 100?

Step 1. Steps per epoch $= ⌈ 1200000/256 ⌉ = ⌈ 4687.5 ⌉ = 4688$ . Why this step? $1200000/256 = 4687.5$ is not whole → the remainder ( $1200000 - 4687 \times 256 = 128$ images) needs one extra (partial) batch, so we round up (cell B logic reused).

Step 2. Total updates $= 90 \times 4688 = 421, 920$ . Why this step? Updates = epochs × steps-per-epoch. Compare: full-batch would give just $90$ updates — mini-batch buys us ~4700× more learning signal per epoch.

Step 3. Wall-clock $= 421920 \times 0.08 s = 33753.6 s = 33753.6/3600 \approx 9.376 hours$ . Why this step? Translate steps into time via seconds-per-step, then convert to hours (÷3600) so the answer is human-readable.

Verify: $4688 \times 256 = 1200128 \geq N$ (last batch partial, 128 imgs) ✓. Time: $421920 \times 0.08 = 33753.6$ s $= 9.376$ h ✓. Units: (steps)(s/step) = s, then s ÷ (s/h) = h ✓.

Ex 8 — Exam twist: invert the noise law (cell I)

Worked example Statement

Per-example gradient std $σ = 12$ . You require the mini-batch gradient std to be at most $1.5$ . What is the smallest valid batch size $m$ ?

Forecast: will the required $m$ be near 8, near 64, or near 500?

Step 1. Write the target as an inequality: $\frac{σ}{m} \leq 1.5$ , i.e. $\frac{12}{m} \leq 1.5$ . Why this step? The exam gives a desired noise and asks for $m$ — so we run the $σ / m$ law backwards instead of forwards.

Step 2. Solve for $m$ : $m \geq \frac{12}{1.5} = 8 \Rightarrow m \geq 8^{2} = 64$ . Why this step? Multiply both sides by $m$ , divide by $1.5$ , then square. Squaring is safe because both sides are positive.

Step 3. Smallest integer batch: $m = 64$ . Why this step? $m$ must be a whole number of examples, and $64$ is already whole and exactly meets the bound — no rounding up needed.

Verify: at $m = 64$ : $12/ 64 = 12/8 = 1.5 \leq 1.5$ ✓; at $m = 63$ : $12/ 63 \approx 1.512 > 1.5$ ✗ (so 64 is genuinely the smallest) ✓.

Recall

Recall Which formula does each cell use?

Steps per epoch ::: $⌈ N / m ⌉$ (round up if there's a remainder batch). Noise (std) of the batch gradient ::: $σ / m$ . Quadruple $m$ → noise does what? ::: Halves (only $4 = 2 \times$ improvement). Linear scaling rule ::: multiply $m$ by $k$ → multiply $η$ by $k$ . Smallest $m$ for std ≤ target $τ$ ::: $m \geq (σ / τ)^{2}$ .

Connections

Mini-batch gradient descent — the parent note these examples drill.
Stochastic Gradient Descent — the $m = 1$ degenerate cell (Ex 4a).
Gradient Descent — the $m = N$ degenerate cell (Ex 4b).
Learning Rate Scheduling — pairs with the linear scaling rule (Ex 6).
Momentum and Adam — optimisers that tame the noise measured here.
Batch Normalization — its statistics depend on the batch sizes above.
Bias-Variance Tradeoff — same variance-of-a-mean math as Ex 3 & 8.
Generalization and Flat Minima — why we don't just crank $m$ to kill all noise.