Visual walkthrough — Stochastic gradient descent (SGD)

This page zooms in on Stochastic gradient descent (SGD). If "gradient" or "loss surface" is fuzzy, keep Gradient Descent open in a tab.

Step 1 — What is a gradient, as an arrow on a hill?

WHAT. Imagine the loss $L(\theta )$ as the height of a landscape. The horizontal position is $\theta$ — the knobs of the network. The height is how wrong the network is. We want the lowest point.

WHY. Before we can talk about estimating a gradient, we need to see what the true one is. The gradient is nothing mystical: it is the little arrow that points straight uphill, and its length says how steep the slope is.

PICTURE. In the figure, the white curve is the loss. At one point sits a chalk-blue dot (our current $\theta$). The pink arrow $\nabla L$ points up-slope; SGD (and plain GD) walks the opposite way — that's the minus sign in the update.

Term by term, the true gradient is an average over the whole dataset: $\nabla L(\theta )=\underset{\text{share each example gets}}{\underbrace{\frac{1}{N}}}\underset{\text{add over all }N\text{ examples}}{\underbrace{\sum _{i=1}^N}}\underset{\text{arrow from example }i}{\underbrace{\nabla {\ell }_i(\theta )}}.$

Each example $i$ produces its own little arrow $\nabla {\ell }_i$. The true gradient is their average.

Step 2 — Every example has its OWN arrow (they disagree!)

WHAT. Draw the per-example arrows $\nabla {\ell }_1,\nabla {\ell }_2,\dots$ at the same point $\theta$. They scatter — some steep, some shallow, some pointing slightly left, some right.

WHY. This scatter is the whole reason SGD is "stochastic". Any one arrow is a bad guess for the true direction. But their average (the black arrow) is exactly $\nabla L$. This is the picture we must keep in our head.

PICTURE. A fan of faint chalk arrows spraying out from the dot; the bold pale-yellow arrow through the middle is their average — the true gradient.

Step 3 — Grab a handful: the mini-batch estimator

WHAT. Pick $m$ examples uniformly at random — call this set $\mathcal{B}$ (a mini-batch). Average only their arrows: $g_{\mathcal{B}}(\theta )=\underset{\text{share each of the }m\text{ picks gets}}{\underbrace{\frac{1}{m}}}\underset{\text{add over the sampled }m}{\underbrace{\sum _{i\in \mathcal{B}}}}\nabla {\ell }_i(\theta ).$

WHY. We can't afford all $N$ arrows every step (millions of them). Averaging a random handful is cheap. The question the rest of the page answers: does this handful-average point the right way?

PICTURE. From the same fan of arrows, three are circled in chalk-blue (the sampled batch). Their short average arrow $g_{\mathcal{B}}$ sits near — but not exactly on — the true yellow arrow.

Step 4 — WHY the handful is unbiased (points right on average)

WHAT. "Unbiased" means: if you repeated the random draw many times and averaged all the resulting $g_{\mathcal{B}}$ arrows, you'd land exactly on the true gradient. No systematic lean.

WHY expectation, not just averaging? We need a tool that says "the typical value over all possible random draws". That tool is the expectation $\mathbb{E}[\cdot ]$ — literally the probability-weighted average outcome. We use it because we can't try every batch, but we can reason about the average of all of them.

For a single random index $i$, every example is equally likely (probability $1/N$): ${\mathbb{E}}_i[\nabla {\ell }_i]=\underset{\text{over every possible pick}}{\underbrace{\sum _{i=1}^N}}\underset{\text{its probability}}{\underbrace{\frac{1}{N}}}\nabla {\ell }_i=\nabla L(\theta ).$

Averaging $m$ such independent picks doesn't change the mean: $\mathbb{E}[g_{\mathcal{B}}]=\frac{1}{m}{\sum }_{i\in \mathcal{B}}\underset{=\nabla L\text{ each}}{\underbrace{\mathbb{E}[\nabla {\ell }_i]}}=\frac{1}{m}\cdot m\cdot \nabla L=\nabla L.$

PICTURE. Many faint sampled-average arrows drawn on top of each other (many possible batches). Their own average — the bold pale-yellow arrow — sits exactly on the true gradient. Off by a bit each time, dead-on in the long run.

Step 5 — WHY the noise shrinks like ${\sigma }^2/m$

WHAT. Individual $g_{\mathcal{B}}$ arrows scatter around the true one. How wide is that scatter? We measure it with variance — the average squared distance from the mean.

WHY variance, and why $1/m$? We need one number for "how far off a typical batch is". Variance is that number. The magic: averaging $m$ independent noisy arrows shrinks the scatter by a factor of $m$. Let ${\sigma }^2$ be the spread of a single example's arrow. Then $\mathrm{Var}[g_{\mathcal{B}}]=\mathrm{Var}\left[\frac{1}{m}{\sum }_{i\in \mathcal{B}}\nabla {\ell }_i\right]=\underset{\text{from the }\frac{1}{m}\text{ squared}}{\underbrace{\frac{1}{m^2}}}\cdot \underset{m\text{ independent spreads add}}{\underbrace{m{\sigma }^2}}=\frac{{\sigma }^2}{m}.$

The $\frac{1}{m}$ out front gets squared to $\frac{1}{m^2}$; the $m$ independent variances add up to $m{\sigma }^2$; multiply and one $m$ survives in the denominator.

PICTURE. Two clouds of sampled arrows around the true gradient: a wide cloud for $m=1$ (chalk-pink) and a tight cloud for $m=16$ (chalk-blue). Same center, different width.

Step 6 — Edge case A: $m=N$ collapses to plain GD

WHAT. Put $m=N$: the batch is the whole dataset. Then $g_{\mathcal{B}}=\nabla L$ exactly, and the variance ${\sigma }^2/N$ is as small as it can get.

WHY show it. A good formula must reproduce the extremes. At $m=N$ the noise cloud shrinks to a single point — that's Gradient Descent, the deterministic method. SGD contains GD as its zero-noise limit.

PICTURE. The scatter cloud collapsing to a dot as $m$ grows from $1$ toward $N$; the arrow becomes the exact yellow one.

Step 7 — Edge case B: constant $\eta$ never fully settles

WHAT. With a fixed learning rate $\eta$, even at the minimum the noisy arrow is non-zero, so $\theta$ keeps twitching. It settles into a ball of wobble of radius $\propto \eta$, never a point.

WHY. The step is ${\theta }_{t+1}={\theta }_t-\eta g_{{\mathcal{B}}_t}$. At the bottom the true gradient is zero, but $g_{\mathcal{B}}$ still carries noise of size $\sigma /\sqrt{m}$. The pushes never stop. To actually land, shrink $\eta$ over time — the Robbins–Monro conditions $\sum {\eta }_t=\infty$ (still able to travel far) and $\sum {\eta }_t^2<\infty$ (noise eventually averages out). See Learning Rate Schedules.

PICTURE. Two paths into a bowl: constant-$\eta$ (pink) orbiting the minimum in a fuzzy ring; decaying-$\eta$ (blue) spiralling inward to the center.

The one-picture summary

Everything on one board: the cloud of per-example arrows (their center = true gradient), one sampled handful (the noisy estimate that's right on average), the shrinking cloud as $m$ grows, and the downhill walk that reaches the minimum despite the jitter.

Quick self-check