Foundations — Optimization — SGD, momentum, Adam — derivations

Before you can read a single line of the parent derivations, you must be able to read the symbols. This page builds every one of them from nothing — plain words first, then a picture, then why the topic can't live without it. Nothing here is assumed.

0. The landscape picture (the mental image everything hangs on)

Picture a hilly valley. Your horizontal position is your choice of numbers (the "settings" of the model). Your height is how wrong the model is at those settings. Lower = better. Optimization = walking to the bottom of the valley.

Keep this picture in your head. Every symbol below is a label on some part of it.

1. $\theta$ — the parameters (your position on the ground)

• Plain words: a bag of dials you are allowed to turn.
• The picture: the flat ground you walk on (the x-y plane under the hills).
• Why the topic needs it: the whole goal of training is to find the best $\theta$, so we need one letter to talk about "all the dials at once."

A single number is written $\theta$; when there are many we still write $\theta$ but mean a vector — an ordered list of numbers, like coordinates $({\theta }_1,{\theta }_2,\dots )$ giving your position in many directions at once.

2. $L(\theta )$ — the loss (your height)

• Plain words: the "badness score" of your current settings.
• The picture: the height of the hill directly above where you stand.
• Why the topic needs it: without a height there is no "downhill." $L$ is the surface we are trying to descend.

The notation $L(\theta )$ just means "$L$ depends on $\theta$" — move your feet ($\theta$ changes), your height ($L$) changes.

3. Subscript $t$ — the time step (which footstep this is)

• Plain words: a stopwatch that ticks once per update.
• The picture: your footprints numbered in the order you made them.
• Why the topic needs it: every optimizer is a rule that turns ${\theta }_t$ into ${\theta }_{t+1}$. The subscript lets us write "old → new" cleanly.

4. The slope: derivative, partial derivative, and $\nabla L$

This is the heart. To walk downhill you must feel the slope. Three levels of "slope," from simplest to the one the topic uses.

4a. Derivative — slope in ONE direction

• Plain words: steepness of the ground under one foot.
• The picture: a straight ramp (tangent line) laid on the curve where you stand — its tilt is the derivative.

Why this tool and not just "look at the graph"? With a million-dimensional landscape you cannot look. The derivative is a formula that gives the slope numerically, no eyes required. That is why calculus, not eyesight, drives training.

4b. Partial derivative $\partial$ — slope while holding others fixed

• The picture: stand still, then slide only east-west and measure the tilt; that is one partial.
• Why needed: each dial has its own slope. We need all of them to know the full downhill direction.

4c. The gradient $\nabla L$ — all slopes bundled into one arrow

• Plain words: the single arrow that says "uphill is that way, and this steep."
• The picture: an arrow drawn on the ground, pointing up the steepest slope of the hill above you.
• Why the topic needs it: the very first update rule is ${\theta }_{t+1}={\theta }_t-\eta \nabla L$. Reading it requires knowing $\nabla L$ is "the uphill arrow" and the minus sign flips it downhill.

The gradient is computed in practice by Backpropagation and rests on the Taylor Expansion idea that near any point a curved surface looks like a flat ramp.

5. $\eta$ — the learning rate (your step length)

• Plain words: the size of your stride.
• The picture: the length of the step-arrow you place on the ground.
• Why the topic needs it: the gradient gives a direction; $\eta$ turns it into an actual move. Too big → you overshoot the valley and bounce; too small → you crawl. Learning Rate Scheduling is entirely about tuning $\eta$ over time.

6. $g_t$ — the estimated gradient from a mini-batch

• Plain words: a quick guess of the downhill direction from a few examples.
• The picture: the true gradient arrow, but wobbling a little because you only glanced at part of the data.
• Why the topic needs it: real datasets are too big to feel the exact slope every step. $g_t$ is what SGD, Momentum, and Adam actually use. It is unbiased (right on average) but jittery — and that jitter is the whole reason the later optimizers exist.

Here ${\ell }_i$ is the loss on one example $i$; $|B|$ is how many examples are in the batch; $\sum$ means "add them all up."

7. $\beta$ and the exponential moving average — smoothing the past

• Plain words: a smoothing knob. $\beta$ near $1$ = long memory (smooth, slow to react); $\beta$ near $0$ = short memory (jumpy).
• The picture: past gradients fading out, each older one weaker than the last.

• Why the topic needs it: the noisy $g_t$ is unreliable step-to-step. Averaging many recent gradients cancels the jitter and reveals the true trend. This EMA is the engine of both Momentum ($v_t$) and Adam ($m_t$, $v_t$). Full detail in Exponential Moving Average.

The "exponential" part: unrolling the recurrence, the gradient from $k$ steps ago is weighted by ${\beta }^k$ — and ${\beta }^k$ shrinks geometrically, so old news fades fast. That is exactly what the picture above shows.

8. Small helper symbols (so nothing surprises you)

• Why they matter: $ϵ$ keeps Adam's division safe; $\mathbb{E}$ is how we say "on average, over the noise"; the dot product $\nabla L^{\top }\Delta \theta$ is the number that tells us whether a step goes up or down.

9. Two danger-features of the landscape

• Why the topic needs them: these two shapes are the villains the whole parent note fights. Noise (SGD) escapes saddles; memory (Momentum/Adam) glides through ravines. More in Saddle Points and Loss Landscapes.

Prerequisite map

Read it as: the plain symbols on top feed the gradient, the gradient plus step size gives the update rule, batching gives SGD, smoothing gives Momentum, and everything together gives Adam.

Equipment checklist

Test yourself — cover the right side.