Worked examples — Optimization — SGD, momentum, Adam — derivations

Here $g$ = the gradient (slope of the loss), $\eta$ = learning rate (step-size knob), $\beta$ = decay/memory of the running average, and $ϵ=10^{-8}$ a tiny number to avoid dividing by zero.

---

The scenario matrix

Every worked example below is tagged with the cell it covers. The matrix lists every case class this topic can throw at you.

#	Cell (scenario class)	What is special	Example
A	Positive gradient, plain step	$g>0$, basic SGD walk	Ex 1
B	Negative gradient / sign flip	$g<0$ and $g$ crossing zero	Ex 2
C	Consistent direction → momentum builds	same-sign $g$, $1/(1-\beta )$ speed-up	Ex 3
D	Oscillating direction → momentum cancels	alternating-sign $g$ (a ravine)	Ex 4
E	Zero / degenerate gradient (saddle)	$g=0$: who moves, who stalls	Ex 5
F	Exploding gradient	huge $g$: Adam's $\sqrt{\hat{v}}$ rescue	Ex 6
G	Pure-noise direction (mean 0)	$\hat{m}\to 0$, $\hat{v}\to \text{Var}$, step $\to 0$	Ex 7
H	Limiting behaviour $t\to \infty$	bias correction fades, EMA saturates	Ex 8
I	Real-world word problem	CFD surrogate LR schedule	Ex 9
J	Exam twist	derive $\beta$ needed for target speed-up	Ex 10

---

Example 1 — Cell A: positive gradient, one plain SGD step

1. Compute the gradient. $g_0={\theta }_0=4$. Why this step? SGD needs the slope at where we stand; for $L=\frac{1}{2}{\theta }^2$ the derivative is exactly $\theta$.
2. Apply the rule ${\theta }_1={\theta }_0-\eta g_0=4-0.1\cdot 4=3.6$. Why this step? Positive slope means the loss rises as $\theta$ grows, so we move left (down the bowl), toward $0$.

Verify: the minimum of $\frac{1}{2}{\theta }^2$ is at $\theta =0$, and $3.6$ is closer to $0$ than $4$ — loss dropped from $8$ to $6.48$. Units: $\theta$ dimensionless, $\eta g$ dimensionless. ✓

---

Example 2 — Cell B: negative gradient and a sign flip

1. Step 1 gradient: $g_0={\theta }_0=-3$ (negative — loss decreases as $\theta$ grows). Why this step? Sign of $g$ tells direction: $g<0$ means go right (increase $\theta$).
2. Update: ${\theta }_1=-3-0.1\cdot (-3)=-3+0.3=-2.7$. Why this step? Subtracting a negative adds — we correctly move toward $0$ from the left.
3. Step 2 gradient: $g_1={\theta }_1=-2.7$, still negative → keep moving right. ${\theta }_2=-2.7-0.1\cdot (-2.7)=-2.43$.

Verify: $|\theta |$ shrank $3\to 2.7\to 2.43$; each step multiplies by $(1-\eta )=0.9$. Indeed $3\cdot 0.9=2.7$ and $2.7\cdot 0.9=2.43$. ✓ No overshoot because $\eta =0.1<1$.

---

Example 3 — Cell C: consistent direction, momentum builds

1. Unroll velocity. $v_0=0.9\cdot 0+1=1$; $v_1=0.9\cdot 1+1=1.9$; $v_2=0.9\cdot 1.9+1=2.71$. Why this step? Momentum sums the recent gradients (an exponentially weighted sum); same-sign terms add up.
2. Limit. Constant $g$ gives a geometric series $v_{\infty }={\sum }_{k\ge 0}{\beta }^{k}g=\frac{g}{1-\beta }=\frac{1}{1-0.9}=10$. Why this step? $\sum {\beta }^k=1/(1-\beta )$ for $|\beta |<1$; this is the source of the famous $1/(1-\beta )$ speed-up.
3. Effective step $=\eta v_{\infty }=0.1\cdot 10=1$, i.e. a 10× amplification of the raw $\eta g=0.1$.

Verify: $v_2=2.71$ against the partial geometric sum ${\sum }_{k=0}^{2}0.9^k=1+0.9+0.81=2.71$. ✓ And $1/(1-0.9)=10$. ✓

---

Example 4 — Cell D: oscillating direction, momentum cancels

1. $v_0=0.9\cdot 0+1=1.0$.
2. $v_1=0.9\cdot 1.0+(-1)=-0.1$. Why this step? The old velocity $+0.9$ is almost cancelled by the flipped $-1$; the alternating signs destructively interfere.
3. $v_2=0.9\cdot (-0.1)+1=0.91$; $v_3=0.9\cdot 0.91-1=-0.181$.

Verify: SGD's velocity would be exactly $g_t=\pm 1$ every step (swing amplitude $1$). Momentum's swing settles near $\pm 0.1$/$\pm 0.9$ — the cross-ravine wobble is damped roughly $\times (1-\beta )$ on the cancelling part. Compare Ex 3 (consistent → grows to $10$) with this (oscillating → stays small): same rule, opposite outcome, exactly the design goal. ✓

---

Example 5 — Cell E: zero gradient at a saddle

1. Plain SGD: step $=-\eta \cdot 0=0$. Stalls instantly. Why this step? SGD has zero memory; no slope → no motion. This is why deterministic full-batch GD gets stuck at saddles.
2. Momentum: $v_t=0.9\cdot 0.5+0=0.45$, step $=-\eta \cdot 0.45\ne 0$. Keeps rolling. Why this step? Stored velocity carries it across the flat spot — like a ball coasting.
3. Adam: $m_t=0.9\cdot 0.5+0.1\cdot 0=0.45$, $v_t^{\text{Adam}}=0.999\cdot 0.04+0.001\cdot 0=0.03996$. Step $\propto \hat{m}/\sqrt{\hat{v}}\ne 0$. Also coasts.

Verify: $0.9\cdot 0.5=0.45$ ✓; $0.999\cdot 0.04=0.03996$ ✓. The degenerate $g=0$ case cleanly separates memoryless SGD (stuck) from the memory-based methods (unstuck) — the whole practical argument for momentum/Adam. ✓

---

Example 6 — Cell F: exploding gradient, Adam rescues

1. SGD step: $-\eta g=-0.1\cdot 1000=-100$. A catastrophic jump — likely diverges. Why this step? SGD's step is proportional to $g$; a big gradient means a big, dangerous leap.
2. Adam step (settled): $-\eta \frac{\hat{m}}{\sqrt{\hat{v}}+ϵ}=-0.1\cdot \frac{1000}{\sqrt{10^6}+10^{-8}}=-0.1\cdot \frac{1000}{1000}\approx -0.1$. Why this step? Dividing by $\sqrt{\hat{v}}=|g|$ normalizes the magnitude away, leaving a $\pm \eta$ unit step. This is the per-coordinate rescaling Adam is famous for.

Verify: $\sqrt{10^6}=1000$, so $1000/1000=1$, times $\eta =0.1$ → $0.1$. The exploding-gradient scenario shows Adam's step is $\approx \eta \text{sign}(g)$, independent of scale. ✓

---

Example 7 — Cell G: pure-noise direction, step dies

1. First moment $m$ (mean tracker): $m_1=0.9\cdot 0+0.1\cdot 2=0.2$; $m_2=0.9\cdot 0.2+0.1\cdot (-2)=-0.02$; $m_3=0.9\cdot (-0.02)+0.1\cdot 2=0.182$; $m_4=0.9\cdot 0.182+0.1\cdot (-2)=-0.0362$. Why this step? $m$ estimates $\mathbb{E}[g]$; with alternating signs it hovers near 0.
2. Second moment $v$ (magnitude tracker), using $g^2=4$ each step: $v_1=0.001\cdot 4=0.004$; $v_2=0.999\cdot 0.004+0.004=0.007996$; and it climbs toward $4$. Why this step? $v$ estimates $\mathbb{E}[g^2]=4$ regardless of sign — squaring erases the flip.
3. Settled step $\propto \hat{m}/\sqrt{\hat{v}}\to 0/\sqrt{4}=0$. Adam freezes noisy directions.

Verify: $0.9\cdot 0.182-0.2=-0.0362$ ✓; $0.999\cdot 0.004+0.001\cdot 4=0.007996$ ✓. Small $\hat{m}$ over larger $\sqrt{\hat{v}}$ ⇒ tiny step — the signal-to-noise behaviour. Contrast Ex 6 (steady $g$ → $\pm \eta$) with this (noise → $0$). ✓

---

Example 8 — Cell H: limiting behaviour, bias correction fades

1. Raw EMA at step $t$: with $g=1$, $m_t=(1-{\beta }_1){\sum }_{k=0}^{t-1}{\beta }_1^k=1-{\beta }_1^t$. Why this step? Starting from $m_0=0$ drags the average toward zero early; $\mathbb{E}[m_t]=g(1-{\beta }_1^t)$ from the parent's series argument.
2. Correction factors: $1-0.9^1=0.1$, $1-0.9^2=0.19$, $1-0.9^{10}=0.6513$.
3. Corrected: ${\hat{m}}_t=m_t/(1-{\beta }_1^t)=(1-{\beta }_1^t)/(1-{\beta }_1^t)=1=g$ exactly, at every $t$. Why this step? Dividing by the same $1-{\beta }_1^t$ cancels the bias — that's the whole point of bias correction; it matters most when $t$ is small (factor $0.1$ at $t=1$).

Verify: $0.9^{10}=0.34868$, so $1-0.9^{10}=0.65132$ ✓; corrected mean is exactly $1$. As $t\to \infty$ the factor $\to 1$, so bias correction quietly switches itself off. ✓

---

Example 9 — Cell I: real-world CFD surrogate word problem

1. $t=0$: ${\eta }_0=0.05/(1+0)=0.05$. Why this step? Early on we want big exploratory steps to escape saddles.
2. $t=100$: $0.05/(1+0.01\cdot 100)=0.05/2=0.025$. Halved.
3. $t=900$: $0.05/(1+9)=0.05/10=0.005$. Why this step? Late training wants tiny steps so the noisy path stops zig-zagging and settles into the minimum.

Verify: $1+0.01\cdot 900=10$, so $0.05/10=0.005$; ratio $0.05/0.005=10$. Units: learning rate dimensionless throughout. ✓ This matches the parent's ${\eta }_t={\eta }_0/(1+kt)$ schedule.

---

Example 10 — Cell J: exam twist, solve for $\beta$

1. Set up: $\frac{1}{1-\beta }=20\Rightarrow 1-\beta =\frac{1}{20}=0.05\Rightarrow \beta =0.95$. Why this step? The steady-state velocity of momentum is $g/(1-\beta )$ (geometric sum, Ex 3); invert it for the target.
2. Default check: $\beta =0.9\Rightarrow 1/(1-0.9)=10\times$.
3. Bump: $\beta =0.99\Rightarrow 1/(1-0.99)=100\times$ — a tiny $\beta$ change (0.9→0.99) multiplies the speed-up 10-fold, because $1-\beta$ shrinks nonlinearly. Why this step? The map $\beta \mapsto 1/(1-\beta )$ blows up near $\beta =1$; that sensitivity is why $\beta$ is tuned in the top decimals.

Verify: $1/(1-0.95)=20$ ✓, $1/(1-0.9)=10$ ✓, $1/(1-0.99)=100$ ✓.

---

Recall Self-test: name the cell, then answer

Which optimizer moves at a saddle where $g=0$ right now, and why? ::: Momentum and Adam (stored velocity / $m$ coasts through); SGD stalls because it has no memory (Cell E). On a steady slope with $\beta =0.9$, what is the momentum speed-up and where does it come from? ::: $10\times$, from the geometric sum $v_{\infty }=g/(1-\beta )$ (Cell C). Why does Adam give a $\approx \pm \eta$ step for both $g=1000$ and $g=0.001$? ::: It divides by $\sqrt{\hat{v}}\approx |g|$, cancelling magnitude — a per-coordinate normalization (Cell F). In a pure-noise direction why does Adam's step go to zero? ::: $\hat{m}\to 0$ (mean gradient) but $\sqrt{\hat{v}}\to \sqrt{\text{Var}}>0$, so the ratio $\to 0$ (Cell G). Why does bias correction matter most at small $t$? ::: The factor $1-{\beta }_1^t$ is tiny early (0.1 at $t=1$), so uncorrected $m$ badly underestimates $g$ (Cell H).

Back to the parent derivations · Hinglish version: 5.6.09 Optimization — SGD, momentum, Adam — derivations (Hinglish)