Optimization — SGD, momentum, Adam — derivations: Visual walkthrough

Step 1 — A ball on a hill: what "descend" even means

WHAT. Picture a landscape. The height of the ground at position $θ$ is a number we call the loss $L (θ)$ . "Training" means: roll the ball to the lowest point.

WHY a picture first. Before any formula, you must see that optimization is just walking downhill. Every optimizer on this page is one rule for "which way, how far?"

PICTURE. In the figure, $θ$ (the parameter) runs left–right. $L (θ)$ (the loss) is the curve's height. The ball sits at $θ_{0}$ ; the arrow shows the only sensible move — toward the valley.

Figure — Optimization — SGD, momentum, Adam — derivations

Step 2 — The slope tells us which way is down

WHAT. At the ball's position, draw the line that just kisses the curve — the tangent. Its steepness is the derivative, written $L^{'} (θ)$ or, in many-dimensions language, the gradient $\nabla L (θ)$ .

WHY this tool and not another. We need a local answer — "which way is down right here?" — using only what we can measure at one point. The derivative is exactly that: the rise-over-run of the tangent. A positive slope means the ground climbs to the right, so we must step left; a negative slope means step right. Either way, we step against the slope.

PICTURE. The green tangent line leans uphill. The red arrow $- L^{'} (θ_{0})$ points the opposite way: downhill. That single sign-flip is the seed of every method here.

Why anti-parallel is optimal (not just "a" downhill direction) is the Taylor Expansion + Cauchy–Schwarz argument in the parent note; here we just trust the picture: opposite the slope drops fastest.

Step 3 — The gradient is noisy: enter SGD

WHAT. In real training we cannot measure the true slope $g_{t}$ ; we estimate it from a small random mini-batch, giving a wobbly arrow $\overset{g}{^}_{t} = g_{t} + (noise)$ .

WHY. Computing the exact slope over millions of samples is too slow. A mini-batch is cheap and, on average, correct — its expected value equals the true slope (an unbiased estimate). See Backpropagation for how each $\overset{g}{^}$ is actually computed.

PICTURE. The true downhill arrow is one clean direction. The mini-batch arrows scatter around it — some too long, some tilted. Follow them blindly and your path zig-zags.

Step 4 — The ravine problem, and momentum's answer

WHAT. Many loss surfaces are ravines: steep across, nearly flat along. SGD bounces off the steep walls (fast oscillation) while crawling along the flat floor (slow progress). Momentum keeps a running average called the velocity $v_{t}$ so consistent directions build up and oscillating ones cancel.

WHY an average. Oscillation means the across-arrows keep flipping sign; adding them cancels. The along-arrows all point the same way; adding them accumulates. A decaying running sum does both at once — this is an Exponential Moving Average.

PICTURE. Left: SGD's saw-tooth path across the ravine. Right: momentum's smoothed path — the sideways wiggles annihilate, the forward drive grows.

Step 5 — Why momentum is a $\frac{1}{1 - β}$ amplifier

WHAT. Unroll the velocity: $v_{t} = \sum_{k = 0}^{t} β^{k} g_{t - k}$ . On a steady slope $g_{t} = g$ (constant), this geometric sum settles to $v_{\infty} = \frac{g}{1 - β}$ .

WHY it matters. With $β = 0.9$ , the factor $\frac{1}{1 - β} = 10$ : momentum sprints 10× faster than SGD along any direction the gradient keeps agreeing on. That is the "acceleration."

PICTURE. The bars $β^{k}$ shrink geometrically ( $1, 0.9, 0.81, \dots$ ); their heights sum to the dashed ceiling at $\frac{1}{1 - β}$ . Recent gradients count most; ancient ones fade.

Step 6 — Adam's second moment: measuring gradient size

WHAT. Adam keeps two running averages per parameter: the mean $m_{t}$ (momentum, "which way") and the mean square $v_{t}$ (magnitude, "how big and how noisy"). $m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}, v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2} .$

WHY square it. Squaring throws away the sign, so $v_{t}$ measures raw strength regardless of direction. A big $v_{t}$ means "this coordinate has large or noisy gradients"; a small $v_{t}$ means "quiet." This lets each parameter get its own step size — vital when early and late network layers have wildly different scales.

PICTURE. Two coordinates: one with a large steady gradient, one with a tiny one. Their $m$ (blue) and $v$ (orange) EMAs track direction and magnitude separately.

Step 7 — The cold-start bias, and its fix

WHAT. Both averages start at zero: $m_{0} = v_{0} = 0$ . Early on they are dragged toward zero — an underestimate. Unrolling with a constant $g$ gives $E [m_{t}] = g (1 - β_{1}^{t})$ , which is smaller than $g$ when $t$ is tiny. The fix is to divide the shrinkage out: $\overset{m}{^}_{t} = \frac{m _{t}}{1 - β _{1}^{t}}, \overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}} .$

WHY. Without correction the very first Adam steps would be absurdly small (starting from an empty memory), stalling training. Dividing by $1 - β_{1}^{t}$ exactly undoes the cold-start.

PICTURE. The raw $m_{t}$ (dashed) creeps up from $0$ ; the corrected $\overset{m}{^}_{t}$ (solid) sits at the true value $g$ from step one. As $t$ grows, $1 - β_{1}^{t} \to 1$ and the two curves merge — the correction politely fades away.

Step 8 — Putting it together: the unit step (all cases)

WHAT. Assemble the update $θ_{t + 1} = θ_{t} - η \frac{m ^ _{t}}{v ^ _{t} + ϵ} .$ Now check every regime the ratio $\overset{m}{^}_{t} / \overset{v}{^}_{t}$ can hit:

Regime	$\overset{m}{^}_{t}$	$\overset{v}{^}_{t}$	ratio	step taken
steady slope $g$	$\to g$	$\to g^{2}$	$\frac{g}{∣ g ∣} = \pm 1$	full $\pm η$ — confident
pure noise (mean 0)	$\to 0$	$\to Var > 0$	$\to 0$	~nothing — ignore noise
dead coordinate $g = 0$	$0$	$0$	$\frac{0}{0 + ϵ} = 0$	$0$ — $ϵ$ saves the divide

WHY. This is the whole point of Adam: divide direction by magnitude so the stride length is $\approx η$ regardless of how big the gradient is. Confident directions get a clean unit step; noisy directions get damped; degenerate zeros are made safe by $ϵ$ (no division by zero).

PICTURE. Three coordinates side by side, each with its raw gradient (grey) and its Adam step (magenta). Big and small raw gradients both produce the same-length magenta step; the noisy one produces almost none.

The one-picture summary

Everything at once: a ball descends a ravine. SGD zig-zags; momentum smooths and speeds along the floor; Adam rescales each axis so the stride is a steady unit step everywhere. The boxed formula is the destination.

Recall Feynman retelling (plain words)

You are blindfolded on a hill and want the bottom. You can feel the slope under your feet — that is the gradient; step against it (Step 2). But your feet are shaky, so each felt slope is a bit wrong — that is SGD's noise (Step 3). To stop staggering, you build momentum: remember your recent direction and keep rolling, so honest directions add up (10× faster!) and the shaky sideways stumbles cancel (Steps 4–5). Adam adds a second trick: for each foot separately, it notices how big your steps usually are and divides that out, so you always take one confident stride — big-slope feet don't leap off a cliff, tiny- slope feet still move, and totally still feet stay put (Steps 6–8). Because your memory starts empty, the first strides would be too timid, so you scale them up until the memory fills (Step 7). Direction from momentum, stride-size from magnitude, safety from $ϵ$ — that is the whole box.

Recall

One-line "why $\pm η$ ?" ::: On a steady slope $\overset{m}{^}_{t} \to g$ and $\overset{v}{^}_{t} \to g^{2}$ , so the ratio is $g / ∣ g ∣ = \pm 1$ and the step is exactly $\pm η$ . One-line "why bias correction?" ::: Zero-initialized EMAs read low by a factor $1 - β^{t}$ ; dividing by it restores the true value from step one.