Visual walkthrough — Batch, mini-batch, stochastic gradient descent

We are re-deriving the parent's central result: the gradient-descent step, and how the batch size $B$ controls the noise in that step. Related ideas live in Gradient Descent, Learning Rate and Schedules, Saddle Points and Non-Convex Optimization, and Loss Functions.

---

Step 1 — What is a "loss surface"? (the hill you stand on)

WHAT. Before any formulas, picture a landscape. The ground is a curved sheet. Your left–right position is a number you can tune — call it $\theta$ (pronounced "theta"), the single knob of our toy model. Your height on the sheet is how badly the model is doing right now — a number called the loss, written $J(\theta )$.

• $\theta$ = the horizontal position = the parameter we are allowed to change.
• $J(\theta )$ = the height directly above that position = "how wrong we are."

WHY. Every learning problem is secretly this: "find the position $\theta$ where the height $J$ is lowest." Low height = small error. So learning = walking to the bottom of a valley.

PICTURE. Look at the curve below. The red dot is where we currently stand. The green X at the bottom is the goal — the lowest point, the best $\theta$.

---

Step 2 — What is the gradient? (which way is downhill?)

WHAT. Standing blindfolded on the hill, you want to know: which way slopes down, and how steeply? The answer at your spot is a single number here: the slope of the curve, written ${\nabla }_{\theta }J$ (read "grad J"). For one knob it is just $\frac{dJ}{d\theta }$ — the tilt of the line that just kisses the curve at your point (the tangent line).

• ${\nabla }_{\theta }J$ = the slope = "for a tiny step right, how much does height change?"
• Positive slope → the curve rises to the right → downhill is left.
• Negative slope → the curve rises to the left → downhill is right.

WHY the slope and not something fancier? Because a slope is exactly the tool that answers "which direction lowers height fastest, right here." A limit-based slope (a derivative) is the smallest possible local description of the surface — it is the least you need and the most you can trust near your feet.

PICTURE. The orange tangent line touches the curve at the red dot. Its steepness is ${\nabla }_{\theta }J$. The arrow shows the direction the slope points uphill; downhill is the opposite.

---

Step 3 — Why we step opposite the gradient (the Taylor argument, drawn)

WHAT. We want to take a small step $\Delta \theta$ (a little nudge left or right) that lowers height. Near our feet the curve looks almost straight (the tangent line). So the change in height for a small nudge is approximately:

$\underset{\text{new height}}{\underbrace{J(\theta +\Delta \theta )}}\approx \underset{\text{height now}}{\underbrace{J(\theta )}}+\underset{\text{slope}}{\underbrace{{\nabla }_{\theta }J(\theta )}}\cdot \underset{\text{our nudge}}{\underbrace{\Delta \theta }}$

Read term by term: new height ≈ old height + (slope × nudge). That "slope × nudge" is the only part we control.

WHY this equation? This is a Taylor expansion kept to its linear (straight-line) piece — a fancy name for "replace the curve by its tangent near the point." We use it because for a tiny step the tangent and the curve are indistinguishable, so it tells us truly which nudge lowers height.

To make the height change $\nabla J\cdot \Delta \theta$ as negative as possible for a fixed step size, point the nudge against the slope:

$\boxed{\Delta \theta =-\eta {\nabla }_{\theta }J}$

• $\eta$ (read "eta") = a small positive number, the step size / learning rate.
• The minus sign = "go downhill, opposite the uphill slope."

PICTURE. Two candidate nudges from the red dot: a green one against the slope (height drops) and a coral one with the slope (height rises). The green wins.

---

Step 4 — Where does the gradient actually come from? (it's an average)

WHAT. The height $J$ is not one thing — it is the average unhappiness over all your training examples. Suppose you have $N$ examples (data points). Each example $i$ has its own little loss $L_i$. Then:

$J(\theta )=\frac{1}{N}{\sum }_{i=1}^N{L}_i$

• $N$ = how many training examples you own.
• ${\sum }_{i=1}^N$ = "add up over every example, from the 1st to the $N$th."
• $\frac{1}{N}$ = "divide by the count, i.e. take the average."
• $L_i$ = the loss of a single example.

Because the average of slopes is the slope of the average, the true downhill direction is also an average:

${\nabla }_{\theta }J=\frac{1}{N}{\sum }_{i=1}^N{\nabla }_{\theta }{L}_i$

WHY does this matter? Because that sum is the expensive part. Each ${\nabla }_{\theta }{L}_i$ costs real compute (a full forward + Backpropagation pass). If $N$ is millions, one honest gradient is a small eternity.

PICTURE. Each faint arrow is one example's downhill direction ${\nabla }_{\theta }{L}_i$. They scatter. Their average (the bold arrow) is the true gradient ${\nabla }_{\theta }J$.

---

Step 5 — The one idea that splits Batch / Mini-batch / SGD

WHAT. We cannot always afford the full average. So we estimate it with a subset. Call the estimate $\hat{g}$ ("g-hat"). The general step is the same for everybody:

$\theta \leftarrow \theta -\eta \hat{g}$

The only difference between the three methods is how many examples $B$ we averaged to build $\hat{g}$:

• $\hat{g}=\frac{1}{N}{\sum }_{i=1}^N\nabla L_i$ — Batch: $B=N$, use all examples.
• $\hat{g}=\frac{1}{B}{\sum }_{i\in \mathcal{B}}\nabla L_i$ — Mini-batch: $1<B<N$, use a chunk $\mathcal{B}$.
• $\hat{g}=\nabla L_k$ — SGD: $B=1$, use one random example $k$.

Here $B$ = the batch size = how many examples feed one step. $\mathcal{B}$ = the little set of chosen examples.

WHY choose fewer? Fewer examples → a cheaper $\hat{g}$ → you can take many more steps in the same time. But the price is that $\hat{g}$ is now only a guess at the true arrow.

PICTURE. Same scatter of per-example arrows. Batch averages all → lands on the true arrow. Mini-batch averages a few → close-ish. SGD grabs one → could point noticeably off.

---

Step 6 — Why the guess is unbiased (right on average)

WHAT. Pick one example $k$ uniformly at random (every example equally likely, chance $\frac{1}{N}$ each). The expected value ${\mathbb{E}}_k$ — the long-run average over many random picks — of that single-sample gradient is:

${\mathbb{E}}_k[\nabla L_k]={\sum }_{k=1}^N\underset{\text{chance of }k}{\underbrace{\frac{1}{N}}}\underset{\text{its arrow}}{\underbrace{\nabla L_k}}=\frac{1}{N}{\sum }_{k=1}^N\nabla L_k={\nabla }_{\theta }J$

• ${\mathbb{E}}_k[\cdot ]$ = "average outcome if you repeated the random pick forever."
• Result: the average of the one-sample guesses equals the true gradient.

WHY it's called unbiased. Unbiased means no systematic tilt — the guesses don't lean left or right on average, they scatter around the truth. So SGD is not wrong on average; it is merely noisy.

PICTURE. A dartboard: the bullseye is the true gradient. SGD darts (one example each) scatter widely but center on the bullseye. Mini-batch darts cluster tightly near it. That clustering-vs-scatter is the whole story of the next step.

---

Step 7 — Why more examples means less noise ($1/B$), drawn

WHAT. Let each single example's gradient wobble around the truth with a spread we call variance ${\sigma }^2$ ("sigma-squared" = a number measuring scatter). If we average $B$ independent examples, the scatter of that average shrinks:

$\mathrm{Var}[\hat{g}]=\frac{{\sigma }^2}{B}$

• $\mathrm{Var}[\hat{g}]$ = how much our estimate jiggles from step to step.
• ${\sigma }^2$ = the jiggle of a single example.
• Dividing by $B$ = "the more you average, the calmer the result."

WHY $1/B$ and not $1/B^2$? Averaging cancels independent wobbles partially, not perfectly — the errors partly overlap. The math of averaging $B$ independent numbers gives exactly a $\frac{1}{B}$ shrink in variance (a $\frac{1}{\sqrt{B}}$ shrink in the actual wiggle length). That ​ is why doubling $B$ from $256$ to $512$ barely helps — see the Bias-Variance Tradeoff.

PICTURE. Three descent paths on the same valley: SGD ($B=1$) zig-zags wildly, mini-batch ($B=32$) wiggles gently, Batch ($B=N$) glides smooth but takes few steps. Same destination, different personalities.

---

Step 8 — The degenerate & edge cases (never leave a gap)

Every scenario, drawn so you never meet a surprise:

• $B=N$ (Batch, the maximum). The subset is the whole dataset, so $\hat{g}=\nabla J$ exactly. Variance ${\sigma }^2/N$ — as small as it can get. One update per epoch.
• $B=1$ (SGD, the minimum). Variance ${\sigma }^2/1={\sigma }^2$ — maximum jitter. $N$ updates per epoch.
• $B>N$? Impossible — you cannot average more examples than you own. $B$ is capped at $N$.
• $\eta$ too large. The linear (tangent) approximation of Step 3 only holds for small nudges. A huge $\eta$ overshoots the valley and the loss climbs — divergence. See Learning Rate and Schedules.
• $\eta \to 0$. Steps become infinitesimally tiny; you crawl and effectively never arrive. Some step size is required.
• Zero gradient ($\nabla J=0$). You're at a flat spot: a minimum (done!), a maximum, or a saddle. Batch GD freezes here; SGD's noise can still nudge off a saddle.
• Bumpy valley (non-convex). Multiple dips exist; the path found depends on noise and start point. Small $B$ explores more.

PICTURE. A 2×2 gallery: (a) $\eta$ good → clean descent, (b) $\eta$ too big → overshoot/diverge, (c) flat saddle where Batch freezes but SGD escapes, (d) two-valley landscape where noise picks the deeper one.

---

The one-picture summary

Everything above compressed: the loss curve, the tangent slope, the anti-slope step, and the three batch-size personalities descending the same valley with different amounts of jitter — smooth Batch, gentle mini-batch, wild SGD — all obeying $\theta \leftarrow \theta -\eta \hat{g}$ and $\mathrm{Var}[\hat{g}]={\sigma }^2/B$.

Recall Feynman: the whole walkthrough in plain words

You're blindfolded on a curved hill and want the lowest spot. (1–2) You feel the tilt of the ground under your feet — that tilt is the gradient. (3) To go down, you step against the tilt; a tiny step's height change is just tilt × step, most negative when you go straight opposite. (4) But the "true tilt" is really the average tilt over all your training examples — and averaging over millions of them is exhausting. (5) So you cheat: feel the tilt at just a few spots (or one) and step on that guess. Batch feels all spots (perfect, slow), SGD feels one (fast, wild), mini-batch feels a handful (the smart middle). (6) The one-spot guess isn't biased — over many taps it averages to the truth — it's just noisy. (7) Averaging $B$ taps calms the noise by $1/B$, so bigger batches walk straighter but step less often. (8) A dash of noise is even helpful: it kicks you off flat saddles where a perfect walker would get stuck. That's the entire idea.

Recall

In one sentence, what is the only difference between Batch, mini-batch, and SGD? ::: How many examples $B$ are averaged to estimate the gradient $\hat{g}$ before each step. Why do we step in the $-\nabla J$ direction? ::: Taylor says $\Delta J\approx \nabla J\cdot \Delta \theta$, which is most negative when the step is anti-parallel to the slope. How does the noise of $\hat{g}$ depend on batch size? ::: $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ — more examples, less jitter. Why is a single-sample gradient "unbiased"? ::: ${\mathbb{E}}_k[\nabla L_k]=\nabla J$; on average one random example points the true way.