Stochastic gradient descent (SGD): Question bank

First, the vocabulary these questions lean on, so no symbol appears un-earned:

$L (θ)$ = the average loss over all $N$ training examples; $θ$ = the parameters (the "knobs") we tune.
$\nabla L$ = the gradient: the vector pointing in the direction of steepest increase of $L$ ; we step the opposite way to go downhill.
$g_{B}$ = the mini-batch gradient: the same average computed on a small random handful $B$ of $m$ examples — a cheap, noisy stand-in for $\nabla L$ .
$η$ = the learning rate: how big a step we take each update.
$σ^{2}$ = the per-example gradient variance: how much one example's gradient scatters around the true average.

True or false — justify

Bigger batch size always gives faster convergence in wall-clock time.

False. Variance falls as

σ^{2} / m

, so std-dev only falls as

1/ m

— doubling

m

doubles cost but cuts noise by just

2

. Past a point you pay more compute for near-zero benefit.

The mini-batch gradient is a biased estimate of the true gradient.

False. With uniform random sampling

E [g_{B}] = \nabla L

exactly — it is unbiased. It is noisy (has variance), which is a different property from being biased.

With a constant learning rate, SGD eventually lands exactly on the minimum.

False. The gradient noise never vanishes with fixed

η

, so

θ

hovers in a ball of radius

\propto η

around the minimum forever. You need a decaying schedule to actually settle.

Full-batch gradient descent can never get stuck at a saddle point.

False. At a saddle the true gradient is

\approx 0

, so deterministic GD stalls there. SGD's noise is exactly what kicks it off — this is one reason the noise is a feature.

Shuffling the data each epoch is optional since every example is used equally.

False. Equal usage isn't the issue; order is. Sorted/unshuffled data makes consecutive batches correlated (e.g. all one class), breaking the i.i.d. assumption and causing biased, oscillating updates.

Pure SGD (

m = 1

) and full-batch GD head toward the same target on average.

True. Both use unbiased directions (

E [g_{B}] = \nabla L

), so in expectation they descend the same way;

m = 1

just zig-zags around that path with much higher variance.

Reducing the learning rate to zero over time is enough to guarantee convergence.

False. It must shrink slowly enough to still travel any distance (

\sum_{t} η_{t} = \infty

) yet fast enough to kill noise (

\sum_{t} η_{t}^{2} < \infty

). Both Robbins–Monro conditions are needed; too-fast decay stalls before reaching the minimum.

One epoch always equals one parameter update.

False. One epoch is one full sweep =

N / m

updates. Only full-batch (

m = N

) makes an epoch a single update.

Spot the error

"SGD is faster than GD because each step moves further downhill."

The error: each SGD step is usually less accurate (noisier), not more powerful. SGD wins by taking many cheap steps per unit compute, not by taking bigger or better individual steps.

"Since

g_{B}

is unbiased, averaging many steps removes all error, so noise is harmless."

Unbiasedness cancels noise in expectation over infinite steps, but at any finite time with fixed

η

the residual jitter has radius

\propto η

. Noise is not harmless — it's why you must decay

η

to converge (and why it helps escape saddles).

"To cut gradient noise in half, double the batch size."

Doubling

m

halves the variance (

σ^{2} / m

) but the noise magnitude is the std-dev,

σ / m

, which only drops by

2 \approx 1.41 \times

. To halve std-dev you need

4 \times

the batch.

"The linear scaling rule says: use a bigger batch to reduce the learning rate."

Reversed. The linear scaling rule says scale

η

up with

m

(

η \propto m

), because a larger, less-noisy batch supports (and needs) a proportionally larger step to keep the same effective progress per epoch.

"SGD's variance formula

σ^{2} / m

proves large batches generalize better."

The formula is only about gradient noise, not generalization. Empirically very large batches often find sharp minima that generalize worse — see Batch Size and Generalization. Less noise

\neq =

better test error.

"Momentum eliminates the need for SGD's stochasticity."

Momentum and Nesterov smooths the trajectory by averaging past gradients; it doesn't remove the sampling noise or the benefit of exploring the loss surface. It complements SGD, it doesn't replace the stochastic estimate.

Why questions

Why does SGD use a random subset instead of just the first

m

examples each step?

Random sampling makes

g_{B}

unbiased (

E [g_{B}] = \nabla L

). A fixed first-

m

slice systematically ignores the rest of the data, giving a biased direction that descends the wrong objective.

Why is the noise variance

\propto 1/ m

and not

1/ m^{2}

The mini-batch gradient averages

m

i.i.d. draws; the variance of an average of

m

independent terms scales as

1/ m

(only the mean helps, cross-terms cancel). This is the ordinary law-of-large-numbers rate.

Why can SGD's noise act as a regularizer?

The jitter prevents

θ

from settling into narrow, sharp valleys and biases it toward flat minima, which tend to generalize better — an implicit link to the Bias-Variance Tradeoff without any explicit penalty term.

Why do we need the gradient at all — why not just try random parameter changes?

The gradient (built by Backpropagation) tells us the steepest downhill direction locally, so each step is informed rather than a blind guess. Random search wastes almost all moves in a high-dimensional space.

Why does the Robbins–Monro condition

\sum_{t} η_{t} = \infty

matter physically?

If the total of all step sizes were finite, you could only ever travel a bounded distance — you might never reach a minimum that's far from your start, no matter how many steps you take.

Why might adaptive optimizers like Adam still keep a stochastic (mini-batch) gradient?

Adam and Adaptive Optimizers rescale how far to step per coordinate, but they still need a gradient estimate to work with — the mini-batch stochasticity (cheap steps, exploration) is orthogonal to per-coordinate scaling and is kept.

Why does shuffling help even though it doesn't change how often each example appears?

It decorrelates successive mini-batches so each behaves like a fresh i.i.d. draw. Without it, structured order injects a repeating pattern into the noise, causing cyclic bias in the updates.

Edge cases

What happens to SGD when

m = N

It becomes exact full-batch Gradient Descent: zero sampling noise, one update per epoch, deterministic and smooth — but expensive and prone to stalling at saddles.

What happens at a data point where the loss is already zero (perfectly fit)?

Its per-example gradient is

0

, so a batch containing it contributes nothing from that point — the update is driven only by the still-imperfect examples in the batch.

If every example were identical, what would the batch size do to the noise?

With identical examples

σ^{2} = 0

: every mini-batch gives the exact true gradient regardless of

m

. Batch size only matters because real examples disagree.

At a flat region where

\nabla L \approx 0

everywhere, how does SGD behave?

The mean step is near zero, but sampling noise still nudges

θ

around, letting it drift and eventually find a slope — whereas deterministic GD would sit motionless.

What if the learning rate

η

is set larger than the curvature allows?

Steps overshoot the minimum and grow each iteration — the loss diverges. This is independent of the noise; it's the same instability full-batch GD suffers, just noisier. See Learning Rate Schedules.

What does "convergence" even mean for constant-

η

SGD, since it never stops moving?

It converges in distribution:

θ

settles into a stationary cloud of radius

\propto η

around the minimum. "Converged" means the loss stops trending down and just fluctuates.

Recall One-line summary of the traps

The recurring lie is "less noise / bigger batch / smaller step = strictly better." Reality: noise is unbiased and useful, its magnitude shrinks only as $1/ m$ , and true convergence needs a decaying schedule (Robbins–Monro), not a big batch or a frozen tiny step.