Worked examples — Batch, mini-batch, stochastic gradient descent

This page is the drill floor for the parent topic. We do not re-explain the theory here — we hit every case the topic can throw at you, one worked example per cell. If a symbol feels unfamiliar, jump back to the parent, then return.

Before anything, a one-line reminder of the machine we are cranking:

---

The scenario matrix

Every problem this topic can pose falls into one of these cells. Each row is a thing that can vary; the examples below are labelled with the cell they cover.

#	Case class	What makes it tricky	Example
C1	Sign of the gradient — positive $\hat{g}$	step must go down in $\theta$, not up	Ex 1
C2	Sign of the gradient — negative $\hat{g}$	step goes up in $\theta$	Ex 2
C3	Zero gradient (degenerate)	already at a flat spot — no move	Ex 3
C4	Batch vs SGD on same data	show noise vs exactness numerically	Ex 4
C5	Variance shrinks as $1/B$	prove the noise-reduction number	Ex 5
C6	Epoch bookkeeping, $N$ not divisible by $B$	leftover last batch	Ex 6
C7	Limiting behaviour — $\eta$ too big	divergence / overshoot	Ex 7
C8	Real-world word problem (aerospace)	translate wall-clock into method choice	Ex 8
C9	Exam twist — linear-scaling rule	scale $\eta$ with $B$ correctly	Ex 9
C10	Escaping a saddle (why noise helps)	zero gradient that noise breaks	Ex 10

The running toy model for the numeric examples is the simplest thing that still has a gradient:

$L_i=\frac{1}{2}(\theta x_i-y_i{)}^2,{\nabla }_{\theta }{L}_i=(\theta x_i-y_i)x_i.$

The middle formula comes from the chain rule: differentiate $\frac{1}{2}(\theta x_i-y_i{)}^2$ and the inner $x_i$ falls out. We use it on every example below. Throughout, we keep the subscript $i$ on ${\nabla }_{\theta }{L}_i$ so it is always clear we mean one sample.

---

C1 — Positive gradient: step must go down in $\theta$

Forecast: we start above the target. Guess — will $\theta$ increase or decrease after one step?

1. Compute the gradient. ${\nabla }_{\theta }{L}_1=(\theta x_1-y_1)x_1=(3\cdot 2-4)\cdot 2=(2)\cdot 2=+4$. Because this is SGD ($B=1$), the estimate equals this exact per-sample gradient: $\hat{g}={\nabla }_{\theta }{L}_1=+4$. Why this step? This is the only quantity the update rule needs; its sign decides direction.
2. Read the sign. $\hat{g}=+4>0$. A positive gradient means "loss rises as $\theta$ rises," so we must move $\theta$ down. Why this step? The rule $\theta -\eta \hat{g}$ subtracts a positive number → $\theta$ shrinks. Sign of $\hat{g}$ = direction, automatically.
3. Update. $\theta \leftarrow 3-0.1(4)=2.6$. Why this step? We took the general rule and plugged $\hat{g}$ in.

Verify: we predicted $\theta$ would drop toward ${\theta }^{\ast }=2$. Distance to the target went from $|3-2|=1$ down to $|2.6-2|=0.6$, so we moved closer. ✓ Positive gradient ⇒ downward step. ✓

---

C2 — Negative gradient: step goes up in $\theta$

Forecast: below target this time. Up or down?

1. Gradient: ${\nabla }_{\theta }{L}_1=(0\cdot 2-4)\cdot 2=-8$. Again SGD, so $\hat{g}={\nabla }_{\theta }{L}_1=-8$. Why this step? Same formula; only $\theta$ changed.
2. Sign: $\hat{g}=-8<0$. Subtracting a negative number adds, so $\theta$ increases. Why this step? $-\eta \hat{g}=-0.1(-8)=+0.8$ — the sign flip is the whole mechanism.
3. Update: $\theta \leftarrow 0-0.1(-8)=0.8$.

Verify: distance to ${\theta }^{\ast }=2$ went $|0-2|=2\to |0.8-2|=1.2$. Closer. ✓ Negative gradient ⇒ upward step. ✓ C1 and C2 together show the sign of $\hat{g}$ always steers us toward the minimum, from either side.

---

C3 — Zero gradient: the degenerate "already there" case

Forecast: we are at ${\theta }^{\ast }$. What should a good algorithm do? (Hint: nothing.)

1. Gradient: ${\nabla }_{\theta }{L}_1=(2\cdot 2-4)\cdot 2=(0)\cdot 2=0$, so $\hat{g}=0$. Why this step? At the minimum the prediction $\theta x_1=4$ equals the target $y_1=4$; the residual $(\theta x_1-y_1)$ is zero, killing the whole gradient.
2. Update: $\theta \leftarrow 2-0.1(0)=2$. No movement. Why this step? $\hat{g}=0\Rightarrow$ step size is $0$ regardless of $\eta$.

Verify: loss $L_1=\frac{1}{2}(4-4{)}^2=0$, already minimal; the algorithm correctly stops moving. ✓ This is the fixed point of the update: $\theta$ is stable iff $\hat{g}=0$. (Careful: this same condition holds at saddle points too — see C10.)

---

C4 — Batch vs SGD on identical data (noise vs exactness)

Forecast: which step is bigger — Batch, or the worst-case single-sample SGD step?

1. Per-sample gradients: ${\nabla }_{\theta }{L}_1=(0-4)\cdot 2=-8$, ${\nabla }_{\theta }{L}_2=(0-1)\cdot 1=-1$. Why this step? We need each one to both average (Batch) and pick (SGD).
2. Batch estimate: ${\hat{g}}_{\text{batch}}=\frac{1}{2}(-8-1)=-4.5$ — here $\hat{g}\ne$ either single ${\nabla }_{\theta }{L}_i$; it is their average over $N=2$, the exact ${\nabla }_{\theta }J$. Update: $\theta \leftarrow 0-0.1(-4.5)=0.45$. Why this step? Batch uses the mean over all $N$ samples.
3. SGD estimate, sample 1: $\hat{g}={\nabla }_{\theta }{L}_1=-8$ → $\theta \leftarrow 0.8$. SGD, sample 2: $\hat{g}={\nabla }_{\theta }{L}_2=-1$ → $\theta \leftarrow 0.1$. Why this step? SGD ($B=1$) picks one random sample, so $\hat{g}$ equals that sample's exact gradient — producing $0.8$ or $0.1$ depending on luck. That luck is the "noise."

Verify: average of the two SGD outcomes $=\frac{1}{2}(0.8+0.1)=0.45=$ the Batch outcome. ✓ This is "SGD is unbiased" made numeric: on average SGD equals Batch, but any single SGD step (0.8 or 0.1) scatters around it. That scatter is the noise.

---

C5 — Variance really does shrink like $1/B$

Forecast: going from $B=1$ to $B=9$ — does noise drop by $9\times$ or by $3\times$?

1. Apply the formula $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ (see Bias-Variance Tradeoff). Why this step? $\hat{g}$ is an average of $B$ i.i.d. gradients; the variance of a mean of $B$ terms is ${\sigma }^2/B$.
2. Plug in: $B=1\Rightarrow 36$; $B=4\Rightarrow 9$; $B=9\Rightarrow 4$.
3. The catch — standard deviation. The noise you feel is the standard deviation $\sqrt{\mathrm{Var}}$: $6,3,2$. From $B=1\to 9$ the std drops $6\to 2$, i.e. a factor of $3=\sqrt{9}$, not $9$. Why this step? This is the "diminishing returns" fact from the parent: noise falls as $1/\sqrt{B}$, so doubling $B$ helps ever less.

Verify: $36/1=36$, $36/4=9$, $36/9=4$; and $\sqrt{36},\sqrt{9},\sqrt{4}=6,3,2$. Ratio $6/2=3=\sqrt{9/1}$. ✓ Cell C5 confirmed.

---

C6 — Epoch bookkeeping when $N$ is NOT divisible by $B$

Forecast: $1000/300=3.33$ — but you can't do a third of an update. Round which way?

1. Full batches: $\lfloor 1000/300\rfloor =3$ full batches of 300 = 900 samples. Why this step? Each update consumes exactly $B$ fresh samples.
2. Leftover: $1000-900=100$ samples remain — one partial last batch. Why this step? We must cover all $N$ data once to finish the epoch, so the remainder still gets its own update.
3. Total updates: $\lceil 1000/300\rceil =4$ updates per epoch (3 full + 1 partial). Why this step? Ceiling, because the remainder still triggers one update.

Verify: $3\cdot 300+100=1000$ ✓, and $\lceil 1000/300\rceil =4$ ✓. Contrast: Batch GD = 1 update/epoch, pure SGD = 1000 updates/epoch. Mini-batch sits between with 4.

---

C7 — Limiting behaviour: learning rate too large ⇒ divergence

Forecast: big $\eta$ = big steps. Does $\theta$ settle at $2$, or fly past it forever?

1. Step 1 gradient: $\hat{g}=(0\cdot 2-4)\cdot 2=-8$. Update: $\theta \leftarrow 0-0.5(-8)=4$. Why this step? We overshot the target ${\theta }^{\ast }=2$ and landed on the other side ($4>2$).
2. Step 2 gradient at $\theta =4$: $\hat{g}=(4\cdot 2-4)\cdot 2=(4)\cdot 2=+8$. Update: $\theta \leftarrow 4-0.5(8)=0$. Why this step? Now positive gradient pushes back — but by the same magnitude, straight back to $0$.
3. Diagnose: $\theta$ ping-pongs $0\to 4\to 0\to 4\dots$ It never converges; loss stays high forever.

Verify: at $\theta =0$, $L_1=\frac{1}{2}(0-4{)}^2=8$; at $\theta =4$, $L_1=\frac{1}{2}(8-4{)}^2=8$ — equal, no progress. ✓ For this quadratic the stability boundary is $\eta <2/x_1^2=2/4=0.5$; we sat exactly on it and got a perpetual oscillation. Any larger and it would blow up. This is why schedules and methods like Momentum and Adam exist.

---

C8 — Real-world aerospace word problem

Forecast: all three touch the same 2M samples per epoch. So which one updates the most within that pass?

1. Compute cost per epoch is the same for all: touching all $N$ = 2M samples once $=2,000,000\times 10^{-6}=2$ seconds. Well inside 30 s. Good — the differentiator is update count, not total compute. Why this step? Number of gradients evaluated per epoch is $N$ for every method; only how you group them into updates (via $B$) changes.
2. Batch GD: groups all 2M into 1 update. You improve $\theta$ once every 2 seconds of compute. Why this step? $B=N$ ⇒ $N/B=1$.
3. SGD: $N/B=2,000,000$ tiny updates — extreme but very jittery and cache-unfriendly.
4. Mini-batch $B=256$: $\lceil 2,000,000/256\rceil =7813$ updates per epoch — many good-enough steps, GPU-friendly. This is the practical winner. Why this step? $2,000,000/256=7812.5$, ceiling $7813$ (last batch has $128$ samples).

Verify: $2,000,000\times 10^{-6}=2$ s ✓ (< 30 s, so all feasible on compute). $7812\cdot 256=1,999,872$, remainder $128$, so $\lceil \cdot \rceil =7813$ ✓. Batch=1, SGD=2,000,000, mini-batch=7813 updates/epoch. Mini-batch gives thousands of trustworthy steps per pass — the 80/20 sweet spot.

---

C9 — Exam twist: the linear-scaling rule for $\eta$

Forecast: batch got $8\times$ bigger. Multiply $\eta$ by $8$, or by $\sqrt{8}$?

1. State the rule. Linear scaling: $\eta \propto B$, so $\frac{{\eta }_{\text{new}}}{{\eta }_{\text{old}}}=\frac{B_{\text{new}}}{B_{\text{old}}}$. Why this step? Larger $B$ ⇒ smaller gradient variance ${\sigma }^2/B$ ⇒ the step is more trustworthy, so you may take a proportionally bolder step.
2. Compute the ratio: $B_{\text{new}}/B_{\text{old}}=512/64=8$.
3. Scale: ${\eta }_{\text{new}}=0.01\times 8=0.08$. Why this step? Direct application of the proportionality.

Verify: $512/64=8$ ✓ and $0.01\times 8=0.08$ ✓. (Sanity: it's the linear rule, so factor $8$, not $\sqrt{8}\approx 2.83$. In practice you also add a warm-up — see Learning Rate and Schedules — but the exam answer is $\eta =0.08$.)

---

C10 — Why a little noise escapes a saddle

Forecast: at $\theta =0$ the gradient is... compute it. Will Batch GD ever move?

1. Batch gradient at 0: ${\nabla }_{\theta }J=3(0{)}^2=0$. Update: $\theta \leftarrow 0-0.1(0)=0$. Stuck forever. Why this step? Exact gradient is exactly zero at the flat spot — Batch GD has no reason to move, even though $\theta =0$ is not a real minimum (loss keeps dropping for $\theta <0$).
2. SGD's noisy estimate. Recall (from the definitions) SGD's estimate is $\hat{g}={\nabla }_{\theta }J+\epsilon$, where $\epsilon$ is the sampling noise: mean zero, variance ${\sigma }^2$, coming from which random sample was drawn. At $\theta =0$ the mean part is $0$, but a single draw of $\epsilon$ is generally nonzero — suppose this draw gives $\epsilon =+0.6$, so $\hat{g}=0+0.6=0.6$. Update: $\theta \leftarrow 0-0.1(0.6)=-0.06$. Why this step? Because $\mathbb{E}[\epsilon ]=0$ but $\mathrm{Var}[\epsilon ]={\sigma }^2>0$, individual steps are nonzero even when the true gradient vanishes. That nonzero kick is precisely what nudges $\theta$ off the flat spot — impossible for noiseless Batch GD.
3. Now the true gradient reappears and carries us downhill. At $\theta =-0.06$: ${\nabla }_{\theta }J=3(-0.06{)}^2=0.0108>0$, so even a noiseless step now continues $\theta \leftarrow -0.06-0.1(0.0108)=-0.06108$, sliding to more-negative $\theta$ where ${\theta }^3$ (the loss) keeps decreasing. Why this step? Once the noise has moved us off the exact saddle, the gradient is no longer zero, so ordinary descent takes over and we keep making real progress. The noise only had to break the tie once.

Verify: $3(0{)}^2=0$ (Batch frozen at the saddle) ✓; one noisy kick $0-0.1(0.6)=-0.06$ ✓; then $3(-0.06{)}^2=0.0108$, giving next $\theta =-0.06-0.1(0.0108)=-0.06108$, which is more negative than $-0.06$ ✓ — i.e. still moving downhill. Conclusion: SGD's zero-mean sampling noise dislodges the parameter from a saddle where Batch GD would sit frozen forever; after one kick, deterministic descent finishes the job. This is the parent's "a little noise is good" made concrete. See Saddle Points and Non-Convex Optimization.

---

Recall Quick self-test on the matrix

Which cell(s) show that the sign of $\hat{g}$ alone decides step direction? ::: C1 (positive) and C2 (negative). In C7, what exact $\eta$ sits on the oscillation boundary for $x_1=2$? ::: $\eta =2/x_1^2=0.5$. In C6, why is the answer 4 and not 3 updates per epoch? ::: The 100 leftover samples form a partial final batch, so we take the ceiling $\lceil 1000/300\rceil =4$. In C5, going $B=1\to 9$ cuts the std of noise by what factor? ::: $\sqrt{9}=3$ (variance by 9, std by 3). When do $\hat{g}$ and ${\nabla }_{\theta }{L}_i$ coincide, and when do they differ? ::: They coincide for SGD ($B=1$, average of one term); they differ for mini-batch and Batch, where $\hat{g}$ averages several ${\nabla }_{\theta }{L}_i$. In C10, why does Batch GD freeze at $\theta =0$ but SGD does not? ::: The exact gradient there is 0, so Batch never moves; SGD's zero-mean noise $\epsilon$ gives a nonzero kick that escapes the flat spot.