Batch, mini-batch, stochastic gradient descent

WHY does this distinction even exist?

The loss we truly want to minimise is the average over all $N$ training samples:

$J(\theta) = \frac{1}{N}\sum_{i=1}^{N} L\big(f_\theta(x_i),\, y_i\big)$

To move downhill we need the gradient $\nabla_\theta J$. The exact gradient is:

$\nabla_\theta J(\theta) = \frac{1}{N}\sum_{i=1}^{N} \nabla_\theta L_i$

The catch: computing this full sum is expensive when $N$ is huge (millions of aerodynamic CFD samples, sensor logs, etc.). So we approximate the gradient using a subset of data. The three methods are three choices of subset size.

HOW the update rule is derived (from first principles)

WHAT we want: parameters $\theta$ that minimise $J(\theta)$.

Step 1 — Taylor expand the loss around current $\theta$ for a small step $\Delta\theta$:

$J(\theta + \Delta\theta) \approx J(\theta) + \nabla_\theta J(\theta)^\top \Delta\theta$

Why this step? Near $\theta$, the loss is locally linear; the linear term dominates the change.

Step 2 — Choose $\Delta\theta$ to decrease $J$ fastest. To make $\nabla J^\top \Delta\theta$ as negative as possible for a fixed step length, point opposite the gradient:

$\Delta\theta = -\eta\,\nabla_\theta J(\theta)$

Why this step? The dot product $\nabla J^\top \Delta\theta$ is minimised when $\Delta\theta$ is anti-parallel to $\nabla J$ (Cauchy–Schwarz). $\eta > 0$ is the learning rate (step size).

Step 3 — The general update:

$\boxed{\theta \leftarrow \theta - \eta\, \hat{g}}$

where $\hat{g}$ is our estimate of $\nabla_\theta J$. The three methods differ only in $\hat g$:

Why is SGD "unbiased"? If we pick $k$ uniformly at random from $\{1,\dots,N\}$:

$\mathbb{E}_k[\nabla_\theta L_k] = \sum_{k=1}^{N}\frac{1}{N}\nabla_\theta L_k = \nabla_\theta J$

So on average one random sample points the right way — it's just noisy. That noise is the whole story.

The variance trade-off (the 80/20 core)

For a mini-batch of size $B$ drawn from data with per-sample gradient variance $\sigma^2$:

$\operatorname{Var}[\hat g] = \frac{\sigma^2}{B}$

Why? Variance of an average of $B$ i.i.d. terms shrinks like $1/B$.

Worked Example 1 — one SGD step by hand

Fit $y = \theta x$ with squared loss $L_i = \tfrac12(\theta x_i - y_i)^2$. Data: $(x_1,y_1)=(2,4)$, start $\theta = 0$, $\eta = 0.1$.

Step: $\nabla_\theta L_1 = (\theta x_1 - y_1)\,x_1 = (0\cdot2 - 4)\cdot2 = -8$. Why this step? Chain rule: $\frac{d}{d\theta}\tfrac12(\theta x - y)^2 = (\theta x - y)\cdot x$.

Update: $\theta \leftarrow 0 - 0.1(-8) = 0.8$. Why? Gradient is negative, so we step up in $\theta$ toward the true value $\theta^*=2$.

We used one sample — this is SGD.

Worked Example 2 — one Batch step

Same model, now use both points: add $(x_2,y_2)=(1,1)$, $\theta = 0$.

$\nabla L_1 = -8$ (as above). $\nabla L_2 = (0\cdot1 - 1)\cdot1 = -1$.

Batch gradient: $\hat g = \tfrac12(-8 + -1) = -4.5$. Why average? Batch GD uses the mean over all $N=2$ samples — the true $J$ gradient.

Update: $\theta \leftarrow 0 - 0.1(-4.5) = 0.45$. Smoother, single trustworthy step.

Worked Example 3 — mini-batch epoch bookkeeping

$N = 1000$ samples, $B = 100$. How many updates per epoch (one full pass over data)?

$\text{updates} = \frac{N}{B} = \frac{1000}{100} = 10.$ Why? Each mini-batch consumes $B$ samples; you need $N/B$ batches to cover all data once. Batch GD would give 1 update/epoch; pure SGD gives 1000.

Flashcards

Connections

• Gradient Descent — parent algorithm; these are its data-sampling variants.
• Learning Rate and Schedules — $\eta$ must adapt to batch size and time.
• Momentum and Adam — smooth out mini-batch noise using running averages.
• Loss Functions — the $L_i$ we sum over.
• Bias-Variance Tradeoff — the same variance logic governs $\hat g$.
• Saddle Points and Non-Convex Optimization — why noise helps in aerospace surrogate models.
• Backpropagation — how each $\nabla_\theta L_i$ is actually computed.

Concept Map

Hinglish (regional understanding)