Optimization — SGD, momentum, Adam — derivations: Exercises

Throughout, symbols mean exactly what the parent note defined. One naming warning up front: the letter $v$ is used for two different things in this chapter, and mixing them is the #1 source of confusion — so we give each its own name here and never write a bare $v_{t}$ again.

Definition Symbol dictionary (earned before use)

$θ$ — the parameter we tune. Picture a bead on a wire; its position is $θ$ .
$L (θ)$ — the loss, the height of the hill above position $θ$ . We want the valley.
$g_{t} = \nabla L (θ_{t})$ — the gradient at step $t$ : an arrow pointing uphill. We walk the other way.
$η$ — the learning rate: how long a stride we take (a positive number).
$u_{t}$ — the momentum velocity (in the parent note this was written $v_{t}$ ; we rename it $u_{t}$ here so it can never be confused with Adam's second moment). It is a running memory of past gradients: $u_{t} = β u_{t - 1} + g_{t}$ .
$β$ — the momentum decay rate (a number in $[0, 1)$ ): the fraction of old velocity kept each step. Big $β$ = long memory. Typical $β = 0.9$ .
$m_{t}$ — Adam's first moment: an average of the gradient $g_{t}$ itself.
$s_{t}$ — Adam's second moment (the parent note called this $v_{t}$ ; we rename it $s_{t}$ to avoid the clash with momentum's velocity). It is an average of the gradient squared, $g_{t}^{2}$ .
$β_{1}$ — Adam's decay rate for the first moment $m_{t}$ (typical $0.9$ ).
$β_{2}$ — Adam's decay rate for the second moment $s_{t}$ (typical $0.999$ , i.e. very long memory).
$ϵ$ — a tiny positive safety constant (typical $1 0^{- 8}$ ) added under the square root so we never divide by zero.
$\overset{m}{^}_{t}, \overset{s}{^}_{t}$ — the bias-corrected first and second moments, rescaled so early steps aren't shrunk.

Level 1 — Recognition

L1.1 Which direction?

You stand at $θ$ and the gradient is $g = + 3$ (loss goes up as $θ$ grows). With $η = 0.1$ , does plain gradient descent move $θ$ left (smaller) or right (larger), and to what value from $θ_{0} = 5$ ?

Recall Solution

WHAT: the update is $θ_{1} = θ_{0} - η g$ . WHY: $- g$ points downhill; a positive $g$ means uphill is to the right, so we step left. $θ_{1} = 5 - 0.1 \cdot 3 = 4.7.$ Moves left (smaller), to $θ_{1} = 4.7$ . ✔

L1.2 Name the term

In $θ_{t + 1} = θ_{t} - η u_{t}$ with $u_{t} = β u_{t - 1} + g_{t}$ , what is $β$ called and what does $β = 0$ collapse to?

Recall Solution

$β$ is the momentum coefficient (a decay/memory factor). With $β = 0$ : $u_{t} = 0 \cdot u_{t - 1} + g_{t} = g_{t} \Rightarrow θ_{t + 1} = θ_{t} - η g_{t},$ which is plain SGD. So momentum with no memory is SGD. ✔

Level 2 — Application

L2.1 One SGD step on a quadratic

$L (θ) = \frac{1}{2} θ^{2}$ , so $g = θ$ . From $θ_{0} = 2$ , $η = 0.1$ , compute $θ_{1}$ and $θ_{2}$ .

Recall Solution

WHAT: apply $θ_{t + 1} = θ_{t} - η g_{t}$ with $g_{t} = θ_{t}$ . WHY $g = θ$ : differentiating $L = \frac{1}{2} θ^{2}$ gives $\frac{d L}{d θ} = θ$ — the slope of a parabola at position $θ$ is exactly $θ$ , so the gradient literally equals where you stand. $g_{0} = θ_{0} = 2$ : $θ_{1} = 2 - 0.1 \cdot 2 = 1.8$ . WHY it shrank by exactly $0.9 \times$ : substituting $g_{t} = θ_{t}$ makes the rule $θ_{t + 1} = θ_{t} - η θ_{t} = (1 - η) θ_{t}$ — every step scales $θ$ by the same factor. $g_{1} = θ_{1} = 1.8$ : $θ_{2} = 1.8 - 0.1 \cdot 1.8 = 1.62$ . So $θ_{t} = 2 \cdot 0. 9^{t}$ — a clean geometric decay toward the valley at $θ = 0$ . ✔

L2.2 Two momentum steps

Same $L$ , $θ_{0} = 2$ , $η = 0.1$ , $β = 0.9$ , $u_{- 1} = 0$ . Compute $θ_{1}$ and $θ_{2}$ , and compare $θ_{2}$ to SGD's $1.62$ .

Recall Solution

WHAT: run $u_{t} = β u_{t - 1} + g_{t}$ then $θ_{t + 1} = θ_{t} - η u_{t}$ . Step 0: $g_{0} = 2$ , $u_{0} = 0.9 \cdot 0 + 2 = 2$ , $θ_{1} = 2 - 0.1 \cdot 2 = 1.8$ . WHY step 0 matches SGD: with $u_{- 1} = 0$ there is no stored velocity yet, so the first velocity $u_{0}$ is just $g_{0}$ — momentum has nothing extra to add on its very first move. Step 1: $g_{1} = 1.8$ , $u_{1} = 0.9 \cdot 2 + 1.8 = 3.6$ , $θ_{2} = 1.8 - 0.1 \cdot 3.6 = 1.44$ . WHY step 1 pulls ahead: $u_{1}$ keeps $0.9 \times$ of the old velocity ( $= 1.8$ ) and adds the fresh gradient ( $1.8$ ), so the velocity $3.6$ is nearly double the raw gradient — the ball has picked up speed in the consistent downhill direction. Momentum reaches $1.44$ vs SGD's $1.62$ — faster, because $u_{1}$ accumulated the first gradient. ✔

L2.3 One Adam step

Same $L$ , $θ_{0} = 2$ , $η = 0.1$ , $β_{1} = 0.9, β_{2} = 0.999, ϵ = 1 0^{- 8}$ . Compute $θ_{1}$ .

Recall Solution

WHAT: compute the two moments, bias-correct, then step. $g_{0} = 2$ . $m_{0} = (1 - 0.9) \cdot 2 = 0.2$ , bias-correct: $\overset{m}{^}_{0} = \frac{0.2}{1 - 0.9} = 2$ . WHY the correction: $m_{0}$ starts from $0$ , so a single update only reaches $(1 - β_{1}) g = 0.2$ — a shrunken version of the true gradient $2$ . Dividing by $1 - β_{1} = 0.1$ undoes that exact shrink and recovers the honest estimate $\overset{m}{^}_{0} = 2$ . $s_{0} = (1 - 0.999) \cdot 2^{2} = 0.004$ , bias-correct: $\overset{s}{^}_{0} = \frac{0.004}{1 - 0.999} = 4$ , so $\overset{s}{^}_{0} = 2$ . WHY divide by $\overset{s}{^}_{0}$ : $\overset{s}{^}_{0} = 4$ estimates $g^{2}$ , so $\overset{s}{^}_{0} = 2 = ∣ g ∣$ — dividing by it cancels the gradient's size, leaving only its sign. $θ_{1} = 2 - 0.1 \cdot \frac{2}{2 + 1 0 ^{- 8}} \approx 2 - 0.1 = 1.9.$ Adam's first step is $\approx η \cdot sign (g) = 0.1$ — magnitude-independent. ✔

Reading the figure below. The horizontal axis is the parameter $θ$ , the vertical axis is the loss $L = \frac{1}{2} θ^{2}$ (the pale chalk parabola). All three optimizers start at $θ_{0} = 2$ (top right). Three coloured dotted-with-dots tracks show successive positions: blue = SGD, pink = Momentum, yellow = Adam. Each track is drawn slightly above the true parabola so you can tell them apart — the vertical offset is cosmetic only, read the horizontal position for the real $θ$ . Takeaway: watch how the pink Momentum dots pull ahead (land further left per step) while blue SGD lags, and yellow Adam takes a deliberately modest, evenly-scaled first stride — the three L2 answers made visual.

Figure — Optimization — SGD, momentum, Adam — derivations

Level 3 — Analysis

L3.1 The $1/ (1 - β)$ amplification

On a steady slope the gradient is a constant $g$ . Show the momentum velocity converges to $u_{\infty} = \frac{g}{1 - β}$ and evaluate for $β = 0.9$ .

Recall Solution

WHAT: unroll $u_{t} = β u_{t - 1} + g$ from $u_{- 1} = 0$ : $u_{t} = \sum_{k = 0}^{t} β^{k} g = g \frac{1 - β ^{t + 1}}{1 - β} .$ WHY that sum: each step keeps a $β$ -fraction of the old velocity and adds the fresh $g$ — a geometric series (see Exponential Moving Average). As $t \to \infty$ , $β^{t + 1} \to 0$ (since $0 \leq β < 1$ ), so $u_{\infty} = \frac{g}{1 - β}$ . For $β = 0.9$ : $u_{\infty} = \frac{g}{0.1} = 10 g$ — a 10× effective learning rate on steady directions. ✔

L3.2 Oscillation cancellation

Suppose a coordinate's gradient alternates: $g_{t} = (- 1)^{t} a$ (steep ravine wall bouncing). Compute $u_{1}, u_{2}, u_{3}$ with $β = 0.9$ , $a = 1$ , $u_{- 1} = 0$ , and explain what happens to the running velocity versus a same-sign case.

Recall Solution

$g_{0} = + 1, g_{1} = - 1, g_{2} = + 1, g_{3} = - 1$ . $u_{0} = 0.9 \cdot 0 + 1 = 1.0$ $u_{1} = 0.9 \cdot 1.0 - 1 = - 0.1$ $u_{2} = 0.9 \cdot (- 0.1) + 1 = 0.91$ $u_{3} = 0.9 \cdot 0.91 - 1 = - 0.181$ After the first entry $u_{0} = 1.0$ , the velocity magnitude stays small (from $u_{1}$ on it bounces between about $\pm 0.1$ and $\pm 0.91$ ) because opposite-sign gradients keep cancelling. In a same-sign case it would instead have grown toward $u_{\infty} = 10$ . That is exactly how momentum damps oscillations while accelerating consistent directions. ✔

L3.3 Why SGD escapes a saddle

At a saddle point $\nabla L = 0$ , so full-batch GD's update is $θ - η \cdot 0 = θ$ — it never moves. Explain, using $E [g_{t}] = \nabla L$ , why an SGD mini-batch step generically does move.

Recall Solution

The mini-batch gradient $g_{t}$ is unbiased: $E [g_{t}] = \nabla L (θ_{t}) = 0$ at the saddle. But unbiased only means the average is zero — the actual $g_{t}$ has nonzero variance, so a typical draw is $g_{t} \neq = 0$ . That random kick pushes $θ$ off the saddle along some direction; if it lands on a downhill escape direction (there always is one at a saddle) the loss drops. See Saddle Points and Loss Landscapes. This is why SGD's noise is a feature, not a bug. ✔

Level 4 — Synthesis

L4.1 Derive Adam's bias correction

With $m_{0} = 0$ and stationary gradients of mean $g$ , show $E [m_{t}] = g (1 - β_{1}^{t})$ and hence why we divide by $(1 - β_{1}^{t})$ .

Recall Solution

WHAT: unroll $m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}$ from $m_{0} = 0$ : $m_{t} = (1 - β_{1}) \sum_{k = 0}^{t - 1} β_{1}^{k} g_{t - k} .$ WHY: each new gradient enters with weight $(1 - β_{1})$ and old ones decay by $β_{1}$ . Take expectation with $E [g_{t - k}] = g$ :

=(1-\beta_1)\,g\cdot\frac{1-\beta_1^{t}}{1-\beta_1}=g\,(1-\beta_1^{t}).$$ Since $1-\beta_1^{t}<1$ for small $t$, raw $m_t$ **underestimates** $g$ (biased toward 0). Dividing: $$\hat m_t=\frac{m_t}{1-\beta_1^{t}} \;\Rightarrow\; \mathbb{E}[\hat m_t]=g.$$ Same algebra gives $\hat s_t=s_t/(1-\beta_2^{t})$. ✔

L4.2 Bias-correction shrink factor

By how much does the bias correction inflate the first-step first moment for $β_{1} = 0.9$ (i.e. the factor $1/ (1 - β_{1}^{1})$ )? And for the second moment at $t = 1$ with $β_{2} = 0.999$ ?

Recall Solution

First moment, $t = 1$ : $\frac{1}{1 - 0. 9 ^{1}} = \frac{1}{0.1} = 10$ . Second moment, $t = 1$ : $\frac{1}{1 - 0.99 9 ^{1}} = \frac{1}{0.001} = 1000$ . These are exactly the inflations we used in L2.3: $m_{0} = 0.2 \to \overset{m}{^}_{0} = 2$ ( $\times 10$ ) and $s_{0} = 0.004 \to \overset{s}{^}_{0} = 4$ ( $\times 1000$ ). ✔

Reading the figure below. The horizontal axis is the step counter $t$ ; the vertical axis (on a log scale, so equal spacing means equal multiplying factor) is the inflation factor $1/ (1 - β^{t})$ that bias correction applies. Pink is the first-moment factor $1/ (1 - 0. 9^{t})$ ; blue is the second-moment factor $1/ (1 - 0.99 9^{t})$ . The yellow dashed line at 1 is "no correction needed." Takeaway: at $t = 1$ the pink factor is $10 \times$ and the blue is a huge $1000 \times$ — wildly different — but both curves slide down toward the yellow line as $t$ grows, so the correction quietly switches itself off once training is underway. This is why skipping it only hurts the very first steps.

L4.3 Adam on a noise-only coordinate

A coordinate receives zero-mean noise: gradients $g_{t}$ with $E [g_{t}] = 0$ , $E [g_{t}^{2}] = σ^{2} > 0$ . Argue the long-run Adam step $\to 0$ , and contrast with a steady-gradient coordinate where the step $\to η$ .

Recall Solution

Noise case: $\overset{m}{^}_{t} \to E [g] = 0$ but $\overset{s}{^}_{t} \to E [g^{2}] = σ^{2} > 0$ . $step = η \frac{m ^ _{t}}{s ^ _{t} + ϵ} \to η \frac{0}{σ} = 0.$ Steady case ( $g_{t} = g$ ): $\overset{m}{^}_{t} \to g$ , $\overset{s}{^}_{t} \to g^{2}$ , so $step \to η \frac{g}{∣ g ∣} = η sign (g) .$ Adam trusts consistent directions (unit step) and ignores pure noise (zero step) — a per-coordinate signal-to-noise filter. ✔

Level 5 — Mastery

L5.1 One-step descent guarantee

Using the first-order Taylor Expansion $L (θ - η g) \approx L (θ) - η g^{⊤} g$ with $g = \nabla L (θ)$ , show the loss strictly decreases (to first order) whenever $g \neq = 0$ and $η > 0$ . What breaks the guarantee?

Recall Solution

WHAT: substitute the GD update $Δ θ = - η g$ into the Taylor expansion: $L (θ - η g) \approx L (θ) - η g^{⊤} g = L (θ) - η ∥ g ∥^{2} .$ Here $g^{⊤} g = ∥ g ∥^{2}$ is just the definition of the squared length of a vector (an identity, not the Cauchy–Schwarz inequality — the steepest-descent direction used Cauchy–Schwarz back in the parent note; this step is a plain dot-product-with-itself). WHY it decreases: $∥ g ∥^{2} \geq 0$ and equals $0$ only if $g = 0$ . With $g \neq = 0, η > 0$ , the change $- η ∥ g ∥^{2} < 0$ , so $L$ drops. What breaks it: the approximation is only first-order; if $η$ is large the ignored second-order (curvature) term can dominate and $L$ may rise — the overshoot from the L1/L3 traps. ✔

L5.2 Convergence rate on the quadratic

For $L = \frac{1}{2} θ^{2}$ (so $g = θ$ ), the SGD map is $θ_{t + 1} = (1 - η) θ_{t}$ . For what $η$ does $θ_{t} \to 0$ , and what $η$ gives the fastest convergence? Cover all cases including $η = 1$ , $η = 2$ , and the non-positive $η \leq 0$ .

Recall Solution

WHAT: iterate the linear map to closed form. Since each step multiplies by the constant $r = 1 - η$ , after $t$ steps $θ_{t} = (1 - η)^{t} θ_{0} = r^{t} θ_{0} .$ WHY the multiplier $r$ decides everything: $r^{t} θ_{0} \to 0$ exactly when the powers $r^{t}$ shrink, i.e. when $∣ r ∣ < 1$ . Solve $∣1 - η ∣ < 1$ : $- 1 < 1 - η < 1 ⟹ 0 < η < 2.$ Now walk every case of the multiplier $r = 1 - η$ :

$η \leq 0$ (non-positive rate): $r = 1 - η \geq 1$ , so $∣ r ∣ \geq 1$ and $∣ θ_{t} ∣$ never shrinks. In fact $η < 0$ steps in the $+ g$ (uphill) direction — pure gradient ascent, loss grows. $η = 0$ gives $r = 1$ : $θ_{t} = θ_{0}$ frozen forever. So $η > 0$ is mandatory.
$0 < η < 1$ : $0 < r < 1$ — smooth monotone shrink toward $0$ (our L2.1 case, $r = 0.9$ ).
$η = 1$ : $r = 0$ , so $θ_{1} = 0 \cdot θ_{0} = 0$ — one-step convergence, the fastest possible.
$1 < η < 2$ : $- 1 < r < 0$ — still converges, but $θ_{t}$ alternates sign each step (oscillating shrink).
$η = 2$ : $r = - 1$ , so $θ_{t} = (- 1)^{t} θ_{0}$ — bounces forever, magnitude never shrinks (marginal, no convergence).
$η > 2$ : $∣ r ∣ > 1$ — diverges, $∣ θ_{t} ∣$ grows without bound.

Summary: converges $⟺ 0 < η < 2$ ; fastest at $η = 1$ ; frozen at $η = 0$ ; ascends for $η < 0$ ; marginal bounce at $η = 2$ ; diverges for $η > 2$ . ✔

L5.3 Synthesize: choose the optimizer

An aerospace CFD surrogate net has (a) a noisy mini-batch gradient, (b) a deep ravine loss valley, and (c) layers whose gradient magnitudes differ by $\sim 1 0^{3}$ . Which single optimizer addresses all three and by which mechanism each?

Recall Solution

Adam. Mechanism-by-mechanism:

(a) Noise → the first moment $\overset{m}{^}_{t}$ (an Exponential Moving Average of gradients) averages out zero-mean noise, just like momentum.
(b) Ravine → the same momentum-like averaging accelerates the consistent along-valley direction and cancels the oscillating across-valley one (L3.2).
(c) $1 0^{3}$ scale mismatch → dividing by $\overset{s}{^}_{t}$ rescales every coordinate to a $\sim η$ step, so the $1 0^{3}$ -larger gradients don't explode and the tiny ones don't stall (L4.3). One optimizer, three mechanisms. That combination is why Adam is the default for such nets. ✔

Recall Self-test checklist

One-step SGD from $θ_{0} = 2, η = 0.1$ on $\frac{1}{2} θ^{2}$ ::: $θ_{1} = 1.8$ Momentum $θ_{2}$ (same setup, $β = 0.9$ ) ::: $1.44$ Adam $θ_{1}$ (same setup) ::: $1.9$ Momentum steady-slope amplification for $β = 0.9$ ::: $10 \times$ Adam first-step second-moment inflation, $β_{2} = 0.999$ ::: $1000 \times$ SGD converges on $\frac{1}{2} θ^{2}$ for which $η$ ? ::: $0 < η < 2$ , fastest at $η = 1$ ; diverges/ascends for $η \leq 0$ or $η \geq 2$