Foundations — Batch, mini-batch, stochastic gradient descent

Before you can read the parent note comfortably, you need to earn every symbol it throws at you. We build them one at a time, each resting on the one before. Nothing here assumes you have seen calculus or statistics — we start from a picture.

1. The training data: $x_i$, $y_i$, $N$

The subscript $i$ is a name tag, not a multiplication. $x_5$ means "the 5th input", the same way "chair #5" points at one chair in a row.

Look at the figure: each dot is one $(x_i,y_i)$ pair plotted on a grid. $N$ is simply the number of dots. That whole cloud of dots is all the data — the thing Batch GD looks at in one go.

2. The model and its knobs: $f_{\theta }$ and $\theta$

Think of $\theta$ as the dial on a radio. Turning it changes what you hear; here, turning $\theta$ changes what the model predicts. Learning = finding the right dial position.

3. Measuring wrongness: the loss $L_i$ and $J(\theta )$

Why squared? Two reasons, both visual:

1. Squaring makes every error positive (a miss above and a miss below both count as "bad").
2. The square is a smooth U-shaped bowl — and a bowl has a clear bottom you can roll down to. See the figure.

Read ${\sum }_{i=1}^N$ out loud as: "add up, for $i$ going from 1 to $N$." The $\frac{1}{N}$ in front turns that big sum into an average. So $J$ is "on average, how wrong is the model right now?"

4. The slope: gradient ${\nabla }_{\theta }$ and the derivative

Now the key question: standing at some $\theta$, which way is downhill on the $J$ landscape? The tool that answers "which way and how steeply does a curve go" is the derivative.

Look at the tangent lines in the figure: the derivative is the tilt of the line that just kisses the curve. Steeper tilt = bigger number.

That single result is what the parent's Worked Example 1 uses — now you know exactly where the $\cdot x$ comes from, and why no $\frac{1}{2}$ survives.

Look at the figure: on a two-knob bowl, the two partial slopes (one per axis) are the two components; join them tip-to-tail and the diagonal arrow is ${\nabla }_{\theta }J$ — the true steepest-uphill direction. To go downhill, we go opposite the arrow: $-{\nabla }_{\theta }J$.

5. The step: learning rate $\eta$ and the update arrow $\Delta \theta$

Putting it together, the ideal update is: $\theta \leftarrow \theta -\eta {\nabla }_{\theta }J$ The arrow $\leftarrow$ means "replace with" — assign the new value to $\theta$. It is not an equation; it is an action.

6. Estimates and averages: $\hat{g}$, $\mathbb{E}$, and $\mathrm{Var}$

Look at the figure: three descent paths on the same bowl — the wobbly one (SGD, $B=1$), the medium one (mini-batch), the smooth one (Batch). Same destination, different amounts of jitter, exactly as ${\sigma }^2/B$ predicts.

7. Bookkeeping words: epoch, batch size $B$

That $N/B$ is the entire content of the parent's Worked Example 3: $1000/100=10$ updates per epoch.

How these foundations feed the topic

Read the chain of dependencies from top to bottom — each box is a foundation from this page, and you cannot skip a link. Data $(x_i,y_i,N)$ define what the model $f_{\theta }$ is judged against; that judgement is the loss $L_i$, averaged into the landscape $J$; the slope of $J$ (derivative → gradient ${\nabla }_{\theta }J$) tells us which way is downhill; the learning rate $\eta$ sizes the step; and finally the estimate $\hat{g}$ (with its expectation and variance) is what turns the ideal step into the three real algorithms.

Use the map to check your reading order: if any box above still feels fuzzy, that is exactly where to re-read before moving to the three descent methods. For where these ideas go next, see Gradient Descent, Learning Rate and Schedules, Loss Functions, Momentum and Adam, Saddle Points and Non-Convex Optimization, Bias-Variance Tradeoff, and Backpropagation (the machine that actually computes ${\nabla }_{\theta }J$ for real networks).

Equipment checklist

Self-test: can you say each answer before revealing it?