Batch, mini-batch, stochastic gradient descent: Question bank

Before diving in, three words you must already own (all built in the parent note):

Gradient estimate $\overset{g}{^}$ — our guess of the true downhill direction $\nabla_{θ} J$ , built from a subset of data.
Batch size $B$ — how many training examples feed into one $\overset{g}{^}$ (and therefore one parameter step).
==Variance of $\overset{g}{^}$ == — how much $\overset{g}{^}$ jitters around the true gradient from step to step; it scales like $σ^{2} / B$ .

Everything here leans on those and on Gradient Descent, Learning Rate and Schedules, and the non-convex landscapes of Saddle Points and Non-Convex Optimization.

True or false — justify

"Batch GD always reaches a lower final loss than SGD."

False. Batch GD finds the exact gradient per step but takes few steps; SGD's noise often lands in flatter, better-generalising minima, so final loss is comparable or better — see Bias-Variance Tradeoff.

"SGD uses exactly one example per epoch."

False. SGD uses one example per update; an epoch is a full pass over all

N

examples, so pure SGD makes

N

updates per epoch.

"Increasing batch size from 1 to 32 and from 256 to 512 reduce noise by the same factor."

False. Variance falls like

σ^{2} / B

, so

1 \to 32

cuts it 32-fold while

256 \to 512

only halves it — diminishing returns are the reason mid-size batches win.

"Mini-batch GD is a compromise, so it must converge slower than both extremes."

False. It usually beats both in wall-clock time: cheaper steps than Batch, less wasted jitter than SGD, and GPU-friendly parallel arithmetic.

"The update rule

θ \leftarrow θ - η \overset{g}{^}

is different for each of the three methods."

False. The rule is identical; only the definition of

\overset{g}{^}

(the subset it averages over) changes.

"A larger batch always lets you keep the same learning rate safely."

False. Larger

B

shrinks gradient variance, so you can afford a bigger

η

; the linear scaling rule says

η \propto B

. Keeping

η

fixed under-uses the cleaner gradient. See Learning Rate and Schedules.

"Because SGD is unbiased, its every single step points downhill."

False. Unbiased means on average it points downhill (

E [\overset{g}{^}] = \nabla J

); any individual step can point sideways or even uphill.

"With a fixed learning rate, SGD converges to the exact minimum."

False. Fixed

η

leaves SGD bouncing in a neighbourhood of the minimum forever; only a decaying schedule

η_{t} \to 0

pins the true point.

Spot the error

Claim: "Batch gradient

\overset{g}{^} = \sum_{i = 1}^{N} \nabla_{θ} L_{i}

Missing the

\frac{1}{N}

average. Without it

\overset{g}{^}

grows with dataset size, making the effective step wildly

N

-dependent — the definition is the mean

\frac{1}{N} \sum \nabla_{θ} L_{i}

Claim: "For SGD,

\overset{g}{^} = \frac{1}{N} \nabla_{θ} L_{k}

for one random

k

\frac{1}{N}

factor belongs there. SGD's estimate is just

\nabla_{θ} L_{k}

; dividing by

N

would shrink every step by

N

and break unbiasedness.

Claim: "The step is

Δ θ = + η \nabla J

because we descend."

Sign error. Descending means going opposite the gradient:

Δ θ = - η \nabla J

. A plus sign climbs the loss.

Claim: "

N = 1000

B = 100

gives

100

updates per epoch."

Reversed ratio. Updates per epoch

= N / B = 1000/100 = 10

, not

N \cdot B

B

Claim: "Variance of

\overset{g}{^}

scales like

σ^{2} / B

, so noise drops slowly."

The variance itself scales like

σ^{2} / B

; it is the standard deviation (the square root) that scales like

σ / B

. Mixing the two mis-states the trade-off.

Claim: "The Taylor step used the second-order term

\frac{1}{2} Δ θ^{⊤} H Δ θ

to justify descent."

The descent direction argument only needs the first-order term

\nabla J^{⊤} Δ θ

; the Hessian term belongs to curvature methods, not the plain gradient-descent derivation.

Claim: "Averaging

B

i.i.d. gradients halves the variance when

B

doubles, so it is linear in

B

It is

\propto 1/ B

, which does halve variance on doubling — but the phrase "linear in

B

" is wrong wording: variance is inversely proportional to

B

, not linearly proportional.

Why questions

Why do we approximate

\nabla J

with a subset instead of always computing the exact full-data gradient?

The exact sum over all

N

samples is too expensive when

N

is huge (millions of CFD or sensor samples); a subset gives a good-enough direction far more cheaply per step.

Why is a little gradient noise sometimes helpful rather than harmful?

Noise can bounce the parameters out of saddle points and shallow local minima that plague non-convex nets, letting training reach better regions — relevant to Saddle Points and Non-Convex Optimization.

Why does the step point anti-parallel to the gradient specifically?

From the first-order Taylor term

Δ J \approx \nabla J^{⊤} Δ θ

, the dot product is most negative (fastest decrease) when

Δ θ

opposes

\nabla J

, by Cauchy–Schwarz.

Why can very large batches generalise worse despite cleaner gradients?

Low noise tends to settle into sharp minima that fit training data tightly but transfer poorly; some noise nudges toward flatter, more robust minima — a Bias-Variance Tradeoff in disguise.

Why does mini-batch often beat Batch GD in wall-clock time even though each Batch step is "more correct"?

Batch makes only one costly update per epoch; mini-batch makes

N / B

cheap, good-enough updates in the same pass, so total progress per second is greater.

Why must the learning rate eventually shrink for SGD to truly converge?

A fixed

η

keeps the stochastic bounce alive around the minimum; a schedule

η_{t} \to 0

damps that bounce so the iterates settle on the actual minimum. See Learning Rate and Schedules.

Edge cases

What is SGD when the dataset has exactly one sample,

N = 1

All three methods collapse to the same thing: Batch, SGD, and mini-batch each use that single example, so

\overset{g}{^}

is identical and exact every step.

What happens to

Var [\overset{g}{^}]

when

B = N

It becomes the variance of the full-data average, the smallest achievable from this data; for the true objective

J

the estimate is exact, so the sampling noise vanishes.

N

is not divisible by

B

(say

N = 1000

B = 300

), how many updates per epoch?

You get

⌈ 1000/300 ⌉ = 4

updates: three full batches of 300 plus one leftover batch of 100 (the final batch is simply smaller).

What is the effective batch behaviour if you set

B

larger than

N

You cannot draw more distinct samples than exist, so it caps at Batch GD (

B = N

); asking for

B > N

just reuses data or is clamped to the full set.

At a saddle point where

\nabla J = 0

exactly, what does each method do?

Batch GD's

\overset{g}{^} = 0

so it stalls indefinitely; SGD/mini-batch have nonzero noise (

\overset{g}{^} \neq = 0

per sample) that kicks it off the saddle and lets training continue.

What does

η = 0

do regardless of batch size?

The update

θ \leftarrow θ - 0 \cdot \overset{g}{^}

makes no change at all — parameters freeze, so no learning happens no matter how good

\overset{g}{^}

is.

B \to \infty

(infinite i.i.d. data per step), what does the mini-batch gradient approach?

Its variance

σ^{2} / B \to 0

, so

\overset{g}{^}

converges to the true expected gradient

\nabla_{θ} J

— mini-batch becomes indistinguishable from exact Batch GD in direction.

Recall One-line self-test before you close the tab

Cover the answers and re-derive: (1) updates per epoch $= N / B$ ; (2) $Var [\overset{g}{^}] \propto σ^{2} / B$ ; (3) step direction $= - η \overset{g}{^}$ ; (4) unbiased means $E [\overset{g}{^}] = \nabla J$ , not "always correct". If all four came instantly, you own the topic.