Exercises — Batch, mini-batch, stochastic gradient descent

The batch size $B$ equals $N$ ($B=60,000=N$). By the parent definition ==$B=N$ is Batch GD==. It uses the exact average gradient every step — zero noise, maximum cost per step.

$B=64$. Since $1<64<N$ (assuming the dataset is bigger than 64), this is Mini-batch GD. The average of 64 gradient terms is an unbiased estimate whose noise (variance ${\sigma }^2/64$, with ${\sigma }^2$ the per-sample gradient variance defined above) is 64× smaller than a single sample's.

Estimate variance is $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ (recall ${\sigma }^2$ is the per-sample gradient variance), so bigger $B$ = less noise:

• Batch ($B=N$): least noisy (exact, variance $\to 0$).
• Mini-batch ($B=32$): medium.
• SGD ($B=1$): most noisy. Order least → most: Batch < Mini-batch < SGD.

Gradient (chain rule): ${\nabla }_{\theta }{L}_1=(\theta x_1-y_1)x_1=(1\cdot 3-6)\cdot 3=(-3)(3)=-9$. Update: $\theta \leftarrow 1-0.05(-9)=1+0.45=1.45$. Why up? The gradient is negative, so stepping against it (subtracting a negative) increases $\theta$ toward the true value ${\theta }^{\text{true}}=2$.

$\nabla L_1=-9$ (from L2.1). $\nabla L_2=(\theta x_2-y_2)x_2=(1\cdot 1-3)\cdot 1=-2$. Batch gradient (average of $N=2$): $\hat{g}=\frac{1}{2}(-9+(-2))=\frac{1}{2}(-11)=-5.5$. Update: $\theta \leftarrow 1-0.05(-5.5)=1+0.275=1.275$. Why average, not sum? Batch GD estimates the mean loss gradient $\frac{1}{N}\sum \nabla L_i$, so the two disagreeing samples ($-9$ and $-2$) are blended into one trustworthy direction $-5.5$. Why is the step smaller than SGD's? The mild sample ($-2$) pulls the average toward zero, so batch takes a gentler, less jumpy step ($+0.275$) than the single hot sample alone would ($+0.45$) — exactly the smoothing that lower noise buys you.

Updates per epoch $=N/B=50,000/250=200$. Over 10 epochs $=200\times 10=2000$ updates. Why? An epoch means seeing every sample once; each mini-batch consumes $B$ samples, so you need $N/B$ batches to cover the data.

Use $\mathrm{Var}[\hat{g}]={\sigma }^2/B$.

• $B=1$: $8/1=8$.
• $B=4$: $8/4=2$.
• $B=16$: $8/16=0.5$. Quadrupling $B$ from 4 to 16 cut variance by 4× — clean $1/B$ shrinkage.

$1030/250=4.12$, which is not a whole number. You form $\lfloor 1030/250\rfloor =4$ full batches of 250 (using 1000 samples), leaving a remainder of $1030-1000=30$ samples. Standard practice: make one extra smaller last mini-batch of size 30. So the epoch has $5$ updates: four with $B=250$ and one with $B=30$. Why does this matter? That last batch has higher variance (${\sigma }^2/30$ vs ${\sigma }^2/250$), so its step is noisier — some libraries let you drop the remainder (drop_last=True) to keep every step equally clean. Either way, the number of updates per epoch is $\lceil N/B\rceil =\lceil 4.12\rceil =5$.

Per epoch each method processes all 1000 samples once, so both cost $\approx 1000$ time units per epoch (same total gradient computations).

• Batch: $N/N=1$ update per epoch.
• Mini-batch $B=50$: $1000/50=20$ updates per epoch. Same compute, but mini-batch makes 20 course-corrections while Batch makes 1. Each mini-batch step is "good enough" (unbiased, moderate noise), so 20 good-enough steps typically drive the loss down far more than a single perfect step. This is why mini-batch wins in wall-clock time even though its per-step direction is less exact.

Run A (Batch) likely got stuck at a saddle point or shallow local minimum: there the exact gradient $\nabla J\approx 0$, so the update $\theta \leftarrow \theta -\eta \cdot 0$ barely moves — it stalls. Run B (SGD) has a noisy estimate $\hat{g}=\nabla L_k$ whose variance is nonzero even when $\nabla J=0$. That random kick pushes $\theta$ off the flat saddle onto a slope where the gradient is again informative. So the noise, harmful for precision, is helpful for escaping flat traps. This is a central reason noisy small-batch methods are favoured for the non-convex surrogate models in aerospace ML.

Standard deviation $=\sqrt{{\sigma }^2/B}=10/\sqrt{B}$.

• $B:1\to 4$: $10/1=10\to 10/2=5$. Drop of $5$ (halved).
• $B:64\to 256$: $10/8=1.25\to 10/16=0.625$. Drop of $0.625$. Same 4× increase in $B$, but the first jump removes 8× more noise than the second. Because noise falls like $1/\sqrt{B}$, early increases in $B$ pay off hugely and later ones barely move the needle — that is the 80/20 sweet spot justifying $B\in [32,256]$.

Constraint 1 (variance): need ${\sigma }^2/B\le {\sigma }^2/10\Rightarrow B\ge 10$. So the smallest allowed batch is $B=10$. Constraint 2 (budget): each update costs $B$ gradient computations; total budget is 1000. Updates $=\frac{1000}{B}$. To maximise updates we want the smallest legal $B$, i.e. $B=10$. Updates $=1000/10=100$ updates. Why smallest legal B? Fewer computations per step means more steps for the same budget, and $B=10$ still satisfies the noise cap exactly. Any $B<10$ would break the variance requirement; any $B>10$ wastes budget on unnecessary noise reduction.

Ratio $256/32=8$, so scale the learning rate by 8: ${\eta }_{\text{new}}=0.01\times 8=0.08$. Why safe? Bigger $B$ shrinks gradient variance (${\sigma }^2/256$ vs ${\sigma }^2/32$ — 8× less). With a cleaner, more trustworthy direction $\hat{g}$, you can take a bolder step without the noise throwing you off course. Keeping $\eta =0.01$ with $B=256$ would waste the low-noise estimate on tiny, over-cautious steps. See Momentum and Adam for adaptive alternatives that ease this tuning.

Update 1 (Batch A, $B=1$): $\nabla L=(\theta \cdot 2-4)\cdot 2=(0-4)(2)=-8$. Mean over 1 sample $=-8$. $\theta \leftarrow 0-0.1(-8)=0.8$. Update 2 (Batch B, $B=2$), now $\theta =0.8$: $\nabla L_{(1,1)}=(0.8\cdot 1-1)\cdot 1=-0.2$. $\nabla L_{(2,4)}=(0.8\cdot 2-4)\cdot 2=(1.6-4)(2)=-4.8$. Mean $\hat{g}=\frac{1}{2}(-0.2+(-4.8))=-2.5$. $\theta \leftarrow 0.8-0.1(-2.5)=0.8+0.25=1.05$. The parameter climbs $0\to 0.8\to 1.05$ toward ${\theta }^{\text{true}}=2$, with the second (averaged) step smoother than the first.

At the true minimum ${\theta }^{\text{opt}}$, the full gradient $\nabla J({\theta }^{\text{opt}})=0$, but a single-sample estimate $\hat{g}=\nabla L_k$ has nonzero variance ${\sigma }^2$ even there. So the update $\theta \leftarrow {\theta }^{\text{opt}}-\eta \hat{g}$ still moves by a random amount of typical size $\eta \sigma$. The iterate never settles; it bounces inside a ball of radius $\sim \eta \sigma$ around ${\theta }^{\text{opt}}$. A larger $\eta$ ⇒ bigger ball. Fix — a schedule ${\eta }_t\to 0$ (e.g. ${\eta }_t={\eta }_0/t$): as steps grow the bounce amplitude ${\eta }_t\sigma \to 0$, so the neighbourhood shrinks to a point. The classic Robbins–Monro conditions ${\sum }_t{\eta }_t=\infty$ (still travel far enough) and ${\sum }_t{\eta }_t^2<\infty$ (noise damps out) guarantee convergence to ${\theta }^{\text{opt}}$. See Learning Rate and Schedules.

A loss surface has sharp minima (narrow, steep walls) and flat minima (wide basins). Sharp minima fit the training set tightly but a small shift between train and test data lands you on a steep wall → large test loss (poor generalisation). Flat minima are robust: nearby points also have low loss, so train/test shift barely hurts. Batch size controls which you find. Large-batch gradients are nearly noise-free, so the optimiser slides straight into the nearest minimum — often a sharp one. Small-batch noise (variance ${\sigma }^2/B$) acts like a random kick that cannot stay inside a narrow sharp basin — it gets shaken out and only wide flat basins can hold it. So small batches implicitly bias toward flat, generalising minima. In bias–variance terms: the noise is a form of regularisation that trades a little training-set fit (slightly higher bias) for much better test performance (lower variance of the learned function). Hence mid-size batches, not huge ones, dominate practice.

Batch GD is impossible: it needs all $N$ samples in memory to form $\frac{1}{N}\sum \nabla L_i$, but data streams in and cannot be fully stored — $N$ is effectively unbounded/growing. Pure SGD ($B=1$) works with streaming (update per sample) but is very noisy and under-uses each GPU pass. Mini-batch is the right choice: buffer a small window (e.g. $B=64$) of the incoming stream, update, discard, repeat. This (a) fits in memory, (b) reduces variance to ${\sigma }^2/64$ for a stable-enough direction, (c) suits the online/streaming setting, and (d) keeps the useful noise that escapes the saddle points of the non-convex surrogate landscape. Pair it with a decaying schedule so early exploration gives way to precise convergence, and consider Momentum and Adam to smooth the noisy stream further.