Mini-batch gradient descent

WHY does this even exist?

WHAT is the problem? For $N = 10^7$ images, computing $\nabla_\theta J$ once means a full forward+backward pass over 10 million images before taking a single step. You'd get maybe a few dozen updates a day. Learning would crawl.

The key statistical insight: the true gradient is an average. Any average can be estimated from a random sample. So pick a random subset $\mathcal{B}$ (the mini-batch) of size $m \ll N$ :

$g_{\mathcal{B}} = \frac{1}{m}\sum_{i \in \mathcal{B}} \nabla_\theta \ell_i$

Derivation-from-scratch: how much noise?

Let $g_i = \nabla_\theta \ell_i$ be per-example gradients with mean $\mu = \nabla J$ and (per-coordinate) variance $\sigma^2$ . The mini-batch estimate is $g_{\mathcal{B}} = \frac{1}{m}\sum_{i=1}^m g_i$ .

Why this step? We want to know how the spread of our estimate depends on batch size $m$ . Use the variance-of-a-mean rule. For independent draws:

$\operatorname{Var}(g_{\mathcal{B}}) = \operatorname{Var}\!\left(\frac{1}{m}\sum_{i=1}^m g_i\right) = \frac{1}{m^2}\sum_{i=1}^m \operatorname{Var}(g_i) = \frac{1}{m^2}\cdot m\,\sigma^2 = \frac{\sigma^2}{m}$

The update rule (plain SGD form): $\theta_{t+1} = \theta_t - \eta\, g_{\mathcal{B}_t}$ where $\eta$ is the learning rate. One pass over all mini-batches (i.e. the whole dataset) = one epoch.

The three regimes (the whole spectrum)

Worked Examples

Common Mistakes (Steel-manned)

Recall

Recall Active-recall checkpoints (hide answers, try first)

Write the mini-batch gradient estimator. → $g_{\mathcal{B}} = \frac{1}{m}\sum_{i\in\mathcal B}\nabla_\theta \ell_i$
Is it biased? → No, $\mathbb{E}[g_{\mathcal B}] = \nabla J$ .
How does its std scale with $m$ ? → $\sigma/\sqrt m$ .
Steps per epoch for $N=1000, m=100$ ? → $10$ .
Why is $m=1$ (SGD) sometimes preferred? → Noise escapes bad minima; cheapest step.

Recall Feynman: explain to a 12-year-old

Imagine you want the average height of everyone in a huge school. Measuring all 5000 kids takes forever. Instead you grab a random handful of 30, average their heights, and use that as a good guess. It's not perfect, but do it again with a new handful and it's usually close. Training a neural network is the same: instead of looking at every photo before nudging the network, you look at a small random pile of photos, figure out which way to nudge, and nudge. Small piles = many quick nudges. Sometimes a pile is odd and nudges you the wrong way, but over thousands of piles you drift the right direction. That's mini-batch gradient descent.

Connections

Stochastic Gradient Descent — the $m=1$ extreme case.
Gradient Descent — the $m=N$ full-batch parent method.
Learning Rate Scheduling — pairs with batch size (linear scaling rule).
Momentum and Adam — optimisers that smooth mini-batch noise.
Batch Normalization — behaviour depends critically on batch statistics.
Bias-Variance Tradeoff — same math: variance of an estimator falls with sample size.
Generalization and Flat Minima — why batch-noise aids generalisation.

What objective does mini-batch GD minimise?

The average loss

J(\theta)=\frac1N\sum_i \ell_i

over the dataset.

What is the mini-batch gradient estimator?

g_{\mathcal B}=\frac1m\sum_{i\in\mathcal B}\nabla_\theta\ell_i

, the mean per-example gradient over a random batch of size

m

Is the mini-batch gradient unbiased?

Yes — for uniformly random batches,

\mathbb E[g_{\mathcal B}]=\nabla_\theta J

How does the standard deviation of the gradient estimate scale with batch size

m

\sigma/\sqrt{m}

(variance

\sigma^2/m

Why is quadrupling the batch size inefficient for reducing noise?

Noise only halves (

1/\sqrt m

) while cost quadruples — diminishing returns.

Define an epoch.

One full pass through all training examples (all mini-batches once).

Steps per epoch formula?

\lceil N/m\rceil

Why can gradient noise improve generalisation?

It helps escape sharp minima and saddle points, biasing toward flatter minima that generalise better.

What are the three regimes by batch size?

m=N

full-batch,

m=1

SGD,

1<m\ll N

mini-batch.

What is the linear scaling rule?

When you multiply batch size by

k

, scale the learning rate by ~

k

(with warmup).

Why must you shuffle data each epoch?

To keep batches approximately i.i.d. so the unbiased-gradient assumption holds and to avoid correlated/class-ordered batches.

Why can mini-batch loss go up on a single step?

The batch gradient is a noisy estimate; individual steps may not descend the true loss even if the average trend does.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, neural network train karne ke liye hume loss ka gradient chahiye poore dataset pe. Lekin agar dataset mein 1 crore images hain, toh har ek step lene se pehle saari images dekhna padega — bahut slow. Dusra extreme yeh hai ki sirf ek image dekh ke step lo (SGD), par woh bahut noisy hota hai. Mini-batch gradient descent beech ka rasta hai: har baar ek chhota random batch (jaise 32, 64, 128 examples) lo, unka average gradient nikaalo, aur ek step le lo. Isse GPU bhi efficiently use hota hai aur updates fast aate hain.

Sabse important baat — yeh estimate unbiased hota hai. Matlab average nikaalo toh $\mathbb{E}[g_{\mathcal B}] =$ sacha gradient. Har single batch thoda galat direction de sakta hai, par hazaaron steps ke baad noise cancel ho jaata hai aur aap sahi jagah pahunch jaate ho. Aur noise ka size batch size $m$ pe depend karta hai: standard deviation $= \sigma/\sqrt{m}$ . Isliye batch 4 guna badhao toh noise sirf aadha hota hai — diminishing returns, isliye bahut bada batch waste hai.

Ek aur mazedaar point: thoda noise acha hota hai. Woh model ko sharp/kharab minima aur saddle points se nikaal deta hai aur flat minima ki taraf le jaata hai jo better generalise karte hain. Isliye full-batch (jisme koi noise nahi) hamesha best nahi hota.

Practical tips: har epoch mein data shuffle karo taaki batches i.i.d. rahein; batch size badhao toh learning rate bhi thoda badhao (linear scaling rule); aur single step ka loss upar-neeche ho toh ghabrao mat — epoch average dekho. Bas yaad rakho: "root-m rules" aur "Mini is Just-Right".

WHY does this even exist?

Derivation-from-scratch: how much noise?

The three regimes (the whole spectrum)

Worked Examples

Common Mistakes (Steel-manned)

Recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Training Deep Networks

Connections