Mini-batch gradient descent: Question bank — AI-ML deep dive

Symbols this page leans on (defined before use)

Before the traps, pin down every symbol so nothing is used unearned. The picture below anchors them.

The figure shows all three: a cloud of individual gradient arrows (spread $σ$ ) around the true gradient $\nabla_{θ} J$ , and the tighter batch-averaged arrow $g_{B}$ whose spread has shrunk to $σ / m$ .

True or false — justify

Each line states a claim. Decide true/false and give the reason.

The mini-batch gradient

g_{B}

is an unbiased estimate of the true gradient

\nabla_{θ} J

True — since each index is drawn uniformly,

E [g_{B}] = \nabla_{θ} J

; on average we step in the exactly correct direction, the noise cancels over many steps.

If a single mini-batch step increases the loss, something has gone wrong with training.

False —

g_{B}

is only an estimate of the true gradient, so an unlucky batch can point uphill for that step. Judge progress by the epoch/running average, not one step.

Quadrupling the batch size

m

cuts the gradient noise to a quarter.

False — noise falls as

σ / m

, so

4 \times

the batch only halves the noise. That

1/ m

law is exactly why huge batches waste compute.

Full-batch gradient descent (

m = N

) always finds a better final solution than mini-batch.

False — it computes an exact gradient but is noiseless, so it can settle in sharp minima; mini-batch noise nudges solutions toward flatter minima that often generalise better, see Generalization and Flat Minima (picture a marble settling in a wide valley vs a narrow crack — the second figure sketches this).

Shuffling the data before each epoch is optional cosmetics — the examples are identical.

False — without shuffling every epoch sees the same batches in the same order, giving correlated (biased-in-sequence) gradients; if data is class-sorted the model sees one class at a time and the i.i.d. assumption behind unbiasedness breaks.

Larger batches let you safely use a larger learning rate.

True — a bigger batch gives a lower-variance gradient, so a bigger step is less likely to overshoot; the linear scaling rule multiplies

η

\approx k

when

m

is multiplied by

k

(with warmup). See Learning Rate Scheduling.

The variance-of-a-mean derivation

Var (g_{B}) = σ^{2} / m

requires the per-example gradients to be independent.

True — the cross-terms vanish only under independence; that's why shuffling (approximate i.i.d. sampling) matters for the clean

1/ m

noise scaling.

Setting

m = 1

(pure SGD) is strictly worse than mini-batch because it is noisier.

False — the extra noise can be a feature: it helps escape saddle points and shallow minima, and it is the cheapest possible step. See Stochastic Gradient Descent.

Spot the error

Each line contains a flawed statement or reasoning. Name the flaw.

"Since

E [g_{B}] = \nabla J

, every mini-batch gradient equals the true gradient."

Confuses the expectation with a sample: individual batches (e.g. gradients

4

then

- 2

around a true value of

1

) are each wrong; only the ensemble average is exact.

"To halve the noise, just double the batch size."

Wrong scaling — noise

\propto 1/ m

, so halving noise needs a

4 \times

batch, not

2 \times

N = 50000

m = 128

gives exactly

50000/128 = 390

updates per epoch."

Forgets the remainder —

⌈ 50000/128 ⌉ = 391

; the leftover

80

examples form a final (smaller) batch unless drop_last is set.

"Bigger batches always improve generalisation because the gradient is more accurate."

A more accurate gradient does not imply better generalisation; large batches remove the helpful regularising noise and tend to find sharp minima, often hurting test accuracy.

g_{B} = \sum_{i \in B} \nabla_{θ} ℓ_{i}

Missing the

\frac{1}{m}

average — without it the estimate is

m

times too large, changing its scale (and effective learning rate) with batch size.

"Because SGD is a descent method, the loss curve should be monotonically decreasing."

The descent guarantee holds for the true gradient; with a noisy batch gradient the per-step loss can rise, so a jittery-but-downward-trending curve is expected.

"The variance of

g_{B}

m σ^{2}

Sign of the exponent is inverted — averaging reduces variance to

σ^{2} / m

; multiplying by

m

would describe a sum, not a mean.

Why questions

Answer the "why" with a mechanism, not a restatement.

Why does mini-batch GD exist at all instead of always using the full dataset?

The true gradient

\nabla_{θ} J

is an average over

N

examples; computing it over

N = 1 0^{7}

costs a full forward+backward pass before one step. Averaging a random sample estimates it cheaply, giving many fast steps.

Why is the mini-batch estimate unbiased?

Each uniformly-drawn index has

E [\nabla_{θ} ℓ_{i}] = \nabla_{θ} J

, and averaging

m

such unbiased terms keeps the mean unchanged.

Why does gradient noise fall as

1/ m

rather than

1/ m

Variance of a mean of

m

independent terms is

σ^{2} / m

, and the standard deviation is its square root, giving

σ / m

Why can gradient noise actually help training?

It acts like a mild random kick that lets the optimiser hop out of saddle points and sharp/shallow minima, empirically steering toward flatter, better-generalising solutions.

Why should you scale the learning rate up when you increase batch size?

A larger batch lowers gradient variance

σ^{2} / m

, so a bigger step is safer; keeping

η

fixed underuses the cleaner signal and makes big-batch training needlessly slow.

Why must you shuffle every epoch and not just once?

A single shuffle fixes one batch partition forever, so gradients across epochs are correlated; reshuffling keeps successive batches approximately i.i.d., preserving the unbiasedness argument.

Why do optimisers like Adam pair naturally with mini-batch GD?

They smooth the batch-to-batch noise using running averages of the gradient, damping the jitter that mini-batching introduces; see Momentum and Adam (imagine a heavy ball ignoring each small sideways nudge and keeping its overall heading).

Edge cases

Boundary and degenerate scenarios — say what happens and why.

What is mini-batch GD when

m = N

It becomes full-batch (exact) gradient descent — noiseless, smooth path, but one slow, memory-heavy update per epoch; see Gradient Descent.

What is mini-batch GD when

m = 1

It becomes pure stochastic gradient descent — maximally noisy per step but cheapest, good at escaping bad minima; see Stochastic Gradient Descent.

What happens to the noise as

m \to N

The estimator variance

σ^{2} / m

keeps shrinking toward zero (with sampling-without-replacement it is

\frac{σ ^{2}}{m} \cdot \frac{N - m}{N - 1}

, which hits exactly

0

m = N

) — so the batch gradient converges to the exact gradient and noise vanishes entirely.

N

is not divisible by

m

, what does the last batch look like?

It is a smaller remainder batch of

N mod m

examples (e.g.

80

for

N = 50000, m = 128

); you either use it or drop it with drop_last=True.

If every per-example gradient is identical, what is the mini-batch variance?

Zero — with

σ^{2} = 0

the estimate

σ^{2} / m = 0

for any

m

, so every batch already returns the exact gradient and batch size is irrelevant.

What happens if the batch is not drawn randomly (e.g. always the first

m

sorted examples)?

The unbiasedness

E [g_{B}] = \nabla J

fails because the sample no longer represents the whole dataset, biasing every step toward that fixed subset.

How does the connection to the Bias-Variance Tradeoff show up here?

Same estimator math — the mini-batch gradient's variance falls with sample size

m

exactly like any sample-mean estimator, while remaining unbiased.

Recall

Recall One-line trap-busters

Does the loss have to drop every step? ::: No — batch noise can push it up for a step; watch the running average.
$4 \times$ batch → how much less noise? ::: Half (noise $\propto 1/ m$ ), not a quarter.
Is $g_{B}$ a sum or a mean? ::: A mean — the $1/ m$ keeps its scale fixed as $m$ changes.
What is $σ^{2}$ here? ::: The per-example gradient variance — the spread of one example's $\nabla_{θ} ℓ_{i}$ about $\nabla_{θ} J$ .
Why reshuffle each epoch? ::: To keep batches approximately i.i.d. so unbiasedness holds.

Connections

Mini-batch gradient descent — the parent topic these traps sit under.
Stochastic Gradient Descent — the $m = 1$ edge case.
Gradient Descent — the $m = N$ edge case.
Learning Rate Scheduling — the linear scaling rule.
Momentum and Adam — noise-smoothing optimisers.
Batch Normalization — depends on batch statistics.
Bias-Variance Tradeoff — estimator-variance analogy.
Generalization and Flat Minima — why batch-noise helps.