Foundations — Stochastic gradient descent (SGD)

This page assumes nothing. Every letter and squiggle in the parent note is unpacked here, in an order where each idea rests on the one before it. If a symbol on the parent page confused you, find it below. (The name SGD always means Stochastic Gradient Descent — "stochastic" is just a fancy word for "randomly sampled.")

---

0. The mental picture we keep returning to

Everything in SGD lives on one image: a landscape of hills and valleys, where low = good. Your position is your parameter setting; the height is how wrong the network is. Walking downhill = learning.

Figure 1 — the mistake landscape. Keep this picture in your head. Every symbol below is either a coordinate on this landscape, the height, or an instruction for how to step.

---

1. Parameters — the symbol $\theta$

Picture. In the landscape image (Figure 1), $\theta$ is your location on the flat ground — your $x$-coordinate (and $y$, and a million more). Turn a dial → you slide to a new spot on the map.

Why the topic needs it. Training = "which setting of the dials is best?" You cannot talk about searching for a good setting without a symbol for "the current setting." That symbol is $\theta$.

The little clock symbol ${\theta }_t$ means "the value of $\theta$ at step $t$" — ${\theta }_0$ is where you start, ${\theta }_1$ after one step, and so on. $t$ is just a counter, like frame numbers in an animation.

---

2. The loss — the symbol ${\ell }_i$ and $L$

The subscript $i$ is a label picking out which example: ${\ell }_1$ is example 1's badness, ${\ell }_2$ is example 2's, up to ${\ell }_N$.

Reading the sum symbol. The big $\sum$ ("sigma") just means add up. ${\sum }_{i=1}^N{\ell }_i$ is a compact way to write ${\ell }_1+{\ell }_2+\cdots +{\ell }_N$. The "$i=1$" underneath is the start, the "$N$" on top is the end. Dividing by $N$ turns "total" into "average."

Picture. $L(\theta )$ is the height of the ground at your location $\theta$. Move to a lower spot → smaller average mistake → better network. The whole surface in Figure 1 is the function $L$.

Why the topic needs it. SGD is a minimizer. You cannot minimize without a single number to shrink — that number is $L$.

---

3. Which direction is downhill? — the gradient ${\nabla }_{\theta }$

Here is the tool the whole method hinges on. We ask: standing at $\theta$, which way is steepest downhill, and how steep? The mathematical answer to exactly that question is the gradient.

A note on the subscript. The upside-down triangle $\nabla$ ("nabla") is the name of this operation: "give me the slope-arrow." The subscript $\theta$ in ${\nabla }_{\theta }$ reminds us we're measuring steepness as the dials $\theta$ change (not, say, as the data changes). We will always write the subscript — ${\nabla }_{\theta }L$ — for consistency, even when it's obvious.

Figure 2 — the gradient arrow. Why this tool and not another? We need one object that answers two questions at once — which way and how much. A plain number (slope) gives "how much" in one dimension only. The gradient generalizes slope to millions of dials: it hands you a full arrow. That is precisely why it, and not a scalar derivative, is the engine of training.

Why the minus sign later. The gradient points uphill (steepest increase). We want down. So we step along $-{\nabla }_{\theta }L$. That single minus sign is the difference between learning and un-learning.

Where does ${\nabla }_{\theta }L$ come from mechanically for a real network? From Backpropagation — the algorithm that computes this arrow efficiently. Here we only need to trust that we can get it.

---

4. The learning rate — the symbol $\eta$

Picture. The gradient tells you which way to walk; $\eta$ tells you how big a stride to take. Big $\eta$ = giant leaps (fast but you might overshoot the valley floor and bounce). Tiny $\eta$ = baby steps (safe but slow).

Figure 3 — stride size. Why the topic needs it. The gradient's length alone would give a stride nobody chose. $\eta$ is the human-tuned knob that scales it into a sensible step. How to shrink $\eta$ over time is the subject of Learning Rate Schedules.

Now we can read the heart of the parent note: ${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }L({\theta }_t)$ In words: new position = old position − (stride) × (uphill arrow). The minus makes it downhill. This is plain Gradient Descent — SGD is one twist away.

---

5. The twist: mini-batches — the symbols $N$, $m$, $\mathcal{B}$

Computing ${\nabla }_{\theta }L$ exactly means summing over all $N$ examples every step. If $N$ is millions, that's brutally slow. The fix: estimate the slope from a small random sample.

Picture. Instead of surveying the entire valley to judge the slope, you feel the ground under just a small patch beneath your feet. That patch is $\mathcal{B}$; its size is $m$.

The notation $i\in \mathcal{B}$ under the sum means "for each example $i$ inside the mini-batch." The three sizes give three flavours: $m=N$ is full-batch, $m=1$ is pure SGD, and in-between is what everyone uses (see Batch Size and Generalization).

Figure 4 — noisy mini-batch arrows. Each blue arrow is one mini-batch's guess $g_{\mathcal{B}}$; the yellow arrow is the true gradient they scatter around.

---

6. Noise words — expectation $\mathbb{E}$ and variance $\mathrm{Var}$

The sampled slope $g_{\mathcal{B}}$ is random (depends which patch you stepped on). Three symbols describe randomness precisely.

Why it matters here. The parent's key claim is $\mathbb{E}[g_{\mathcal{B}}]={\nabla }_{\theta }L$: on average, the cheap patch-slope points the same way as the true slope. This is called unbiased — no systematic error, just wobble.

Picture. In Figure 4, each patch gives a slightly different arrow. ${\sigma }^2$ is the spread of that cloud when the patch is a single example.

What it means. Bigger patch $m$ → tighter cloud → less wobble, but only as $1/m$, so doubling $m$ only halves variance. Diminishing returns.

---

7. Epoch — a housekeeping word

Picture. If the dataset is a deck of $N$ cards and each step deals $m$ cards, one epoch = dealing the whole deck once. You shuffle before each epoch so batches stay random and un-correlated.

---

Prerequisite map

Read it as: dials + mistake give the height; the height gives the downhill arrow; arrow + stride give the step; sampling a batch makes the arrow noisy; noisy step, repeated, is SGD.

---

Equipment checklist

Test yourself — you should be able to answer each before reading the parent note.

---

Next: with these symbols earned, re-read the parent SGD note, then branch into Momentum and Nesterov and Adam and Adaptive Optimizers for smarter stepping rules.