Worked examples — Optimization — SGD, momentum, Adam — derivations

5.6.9 · D3 · Coding › Machine Learning (Aerospace Applications) › Optimization — SGD, momentum, Adam — derivations

Yahan $g$ = gradient (loss ka slope), $\eta$ = learning rate (step-size knob), $\beta$ = decay/memory of running average, aur $ϵ=10^{-8}$ ek tiny number hai taaki zero se divide na ho.

---

Scenario matrix

Neeche har worked example us cell ke saath tagged hai jo woh cover karta hai. Matrix har case class list karta hai jo yeh topic aap par throw kar sakta hai.

#	Cell (scenario class)	Kya special hai	Example
A	Positive gradient, plain step	$g>0$, basic SGD walk	Ex 1
B	Negative gradient / sign flip	$g<0$ aur $g$ ka zero cross karna	Ex 2
C	Consistent direction → momentum builds	same-sign $g$, $1/(1-\beta )$ speed-up	Ex 3
D	Oscillating direction → momentum cancels	alternating-sign $g$ (ek ravine)	Ex 4
E	Zero / degenerate gradient (saddle)	$g=0$: kaun move karta hai, kaun stall karta hai	Ex 5
F	Exploding gradient	huge $g$: Adam ka $\sqrt{\hat{v}}$ rescue	Ex 6
G	Pure-noise direction (mean 0)	$\hat{m}\to 0$, $\hat{v}\to \text{Var}$, step $\to 0$	Ex 7
H	Limiting behaviour $t\to \infty$	bias correction fade hoti hai, EMA saturate hoti hai	Ex 8
I	Real-world word problem	CFD surrogate LR schedule	Ex 9
J	Exam twist	target speed-up ke liye $\beta$ derive karo	Ex 10

---

Example 1 — Cell A: positive gradient, ek plain SGD step

1. Gradient compute karo. $g_0={\theta }_0=4$. Yeh step kyun? SGD ko wahan slope chahiye jahan hum khade hain; $L=\frac{1}{2}{\theta }^2$ ke liye derivative exactly $\theta$ hai.
2. Rule apply karo ${\theta }_1={\theta }_0-\eta g_0=4-0.1\cdot 4=3.6$. Yeh step kyun? Positive slope matlab loss badhti hai jab $\theta$ barhta hai, toh hum left move karte hain (bowl ke neeche), $0$ ki taraf.

Verify: $\frac{1}{2}{\theta }^2$ ka minimum $\theta =0$ pe hai, aur $3.6$, $4$ se zyada $0$ ke paas hai — loss $8$ se $6.48$ tak giri. Units: $\theta$ dimensionless, $\eta g$ dimensionless. ✓

---

Example 2 — Cell B: negative gradient aur ek sign flip

1. Step 1 gradient: $g_0={\theta }_0=-3$ (negative — loss decrease hoti hai jab $\theta$ barhta hai). Yeh step kyun? $g$ ka sign direction batata hai: $g<0$ matlab right jao ($\theta$ badhao).
2. Update: ${\theta }_1=-3-0.1\cdot (-3)=-3+0.3=-2.7$. Yeh step kyun? Negative subtract karna add karne jaisa hai — hum sahi tarah se $0$ ki taraf left se move karte hain.
3. Step 2 gradient: $g_1={\theta }_1=-2.7$, abhi bhi negative → right move karte raho. ${\theta }_2=-2.7-0.1\cdot (-2.7)=-2.43$.

Verify: $|\theta |$ shrink hua $3\to 2.7\to 2.43$; har step $(1-\eta )=0.9$ se multiply hota hai. Wakai $3\cdot 0.9=2.7$ aur $2.7\cdot 0.9=2.43$. ✓ Koi overshoot nahi kyunki $\eta =0.1<1$.

---

Example 3 — Cell C: consistent direction, momentum builds

1. Velocity unroll karo. $v_0=0.9\cdot 0+1=1$; $v_1=0.9\cdot 1+1=1.9$; $v_2=0.9\cdot 1.9+1=2.71$. Yeh step kyun? Momentum recent gradients ko sum karta hai (ek exponentially weighted sum); same-sign terms add up hote hain.
2. Limit. Constant $g$ ek geometric series deta hai $v_{\infty }={\sum }_{k\ge 0}{\beta }^{k}g=\frac{g}{1-\beta }=\frac{1}{1-0.9}=10$. Yeh step kyun? $\sum {\beta }^k=1/(1-\beta )$ jab $|\beta |<1$; yahi famous $1/(1-\beta )$ speed-up ka source hai.
3. Effective step $=\eta v_{\infty }=0.1\cdot 10=1$, yaani raw $\eta g=0.1$ ka 10× amplification.

Verify: $v_2=2.71$ partial geometric sum ${\sum }_{k=0}^{2}0.9^k=1+0.9+0.81=2.71$ ke against. ✓ Aur $1/(1-0.9)=10$. ✓

---

Example 4 — Cell D: oscillating direction, momentum cancels

1. $v_0=0.9\cdot 0+1=1.0$.
2. $v_1=0.9\cdot 1.0+(-1)=-0.1$. Yeh step kyun? Purani velocity $+0.9$ almost cancel ho jaati hai flipped $-1$ se; alternating signs destructively interfere karte hain.
3. $v_2=0.9\cdot (-0.1)+1=0.91$; $v_3=0.9\cdot 0.91-1=-0.181$.

Verify: SGD ki velocity exactly $g_t=\pm 1$ hoti har step (swing amplitude $1$). Momentum ka swing $\pm 0.1$/$\pm 0.9$ ke paas settle hota hai — cross-ravine wobble roughly $\times (1-\beta )$ damped hoti hai cancelling part pe. Ex 3 (consistent → $10$ tak badhta hai) ko is se (oscillating → chota rehta hai) compare karo: same rule, opposite outcome, exactly design goal. ✓

---

Example 5 — Cell E: saddle pe zero gradient

1. Plain SGD: step $=-\eta \cdot 0=0$. Instantly stall karta hai. Yeh step kyun? SGD ki zero memory hai; koi slope nahi → koi motion nahi. Isliye deterministic full-batch GD saddles pe stuck ho jaata hai.
2. Momentum: $v_t=0.9\cdot 0.5+0=0.45$, step $=-\eta \cdot 0.45\ne 0$. Rolling rehta hai. Yeh step kyun? Stored velocity ise flat spot ke across carry karti hai — jaise ek ball coasting kare.
3. Adam: $m_t=0.9\cdot 0.5+0.1\cdot 0=0.45$, $v_t^{\text{Adam}}=0.999\cdot 0.04+0.001\cdot 0=0.03996$. Step $\propto \hat{m}/\sqrt{\hat{v}}\ne 0$. Yeh bhi coast karta hai.

Verify: $0.9\cdot 0.5=0.45$ ✓; $0.999\cdot 0.04=0.03996$ ✓. Degenerate $g=0$ case clearly separate karta hai memoryless SGD (stuck) ko memory-based methods (unstuck) se — yahi momentum/Adam ka poora practical argument hai. ✓

---

Example 6 — Cell F: exploding gradient, Adam rescues

1. SGD step: $-\eta g=-0.1\cdot 1000=-100$. Ek catastrophic jump — likely diverge karega. Yeh step kyun? SGD ka step $g$ ke proportional hai; bada gradient matlab bada, dangerous leap.
2. Adam step (settled): $-\eta \frac{\hat{m}}{\sqrt{\hat{v}}+ϵ}=-0.1\cdot \frac{1000}{\sqrt{10^6}+10^{-8}}=-0.1\cdot \frac{1000}{1000}\approx -0.1$. Yeh step kyun? $\sqrt{\hat{v}}=|g|$ se divide karna magnitude ko normalize kar deta hai, ek $\pm \eta$ unit step chodh ke. Yahi Adam ki famous per-coordinate rescaling hai.

Verify: $\sqrt{10^6}=1000$, toh $1000/1000=1$, times $\eta =0.1$ → $0.1$. Exploding-gradient scenario dikhata hai Adam ka step $\approx \eta \text{sign}(g)$ hai, scale se independent. ✓

---

Example 7 — Cell G: pure-noise direction, step dies

1. First moment $m$ (mean tracker): $m_1=0.9\cdot 0+0.1\cdot 2=0.2$; $m_2=0.9\cdot 0.2+0.1\cdot (-2)=-0.02$; $m_3=0.9\cdot (-0.02)+0.1\cdot 2=0.182$; $m_4=0.9\cdot 0.182+0.1\cdot (-2)=-0.0362$. Yeh step kyun? $m$ estimate karta hai $\mathbb{E}[g]$; alternating signs ke saath yeh near 0 hover karta hai.
2. Second moment $v$ (magnitude tracker), $g^2=4$ har step pe use karke: $v_1=0.001\cdot 4=0.004$; $v_2=0.999\cdot 0.004+0.004=0.007996$; aur yeh $4$ ki taraf climb karta hai. Yeh step kyun? $v$ estimate karta hai $\mathbb{E}[g^2]=4$ regardless of sign — squaring flip erase kar deta hai.
3. Settled step $\propto \hat{m}/\sqrt{\hat{v}}\to 0/\sqrt{4}=0$. Adam noisy directions ko freeze kar deta hai.

Verify: $0.9\cdot 0.182-0.2=-0.0362$ ✓; $0.999\cdot 0.004+0.001\cdot 4=0.007996$ ✓. Chota $\hat{m}$ bade $\sqrt{\hat{v}}$ ke upar ⇒ tiny step — signal-to-noise behaviour. Ex 6 (steady $g$ → $\pm \eta$) ko is se (noise → $0$) contrast karo. ✓

---

Example 8 — Cell H: limiting behaviour, bias correction fades

1. Raw EMA at step $t$: $g=1$ ke saath, $m_t=(1-{\beta }_1){\sum }_{k=0}^{t-1}{\beta }_1^k=1-{\beta }_1^t$. Yeh step kyun? $m_0=0$ se start karna average ko early zero ki taraf drag karta hai; $\mathbb{E}[m_t]=g(1-{\beta }_1^t)$ parent ke series argument se.
2. Correction factors: $1-0.9^1=0.1$, $1-0.9^2=0.19$, $1-0.9^{10}=0.6513$.
3. Corrected: ${\hat{m}}_t=m_t/(1-{\beta }_1^t)=(1-{\beta }_1^t)/(1-{\beta }_1^t)=1=g$ exactly, har $t$ pe. Yeh step kyun? Same $1-{\beta }_1^t$ se divide karna bias cancel karta hai — bias correction ka poora point yahi hai; yeh sabse zyada matter karta hai jab $t$ small ho (factor $0.1$ at $t=1$).

Verify: $0.9^{10}=0.34868$, toh $1-0.9^{10}=0.65132$ ✓; corrected mean exactly $1$ hai. Jab $t\to \infty$ factor $\to 1$ hota hai, toh bias correction quietly khud ko switch off kar leta hai. ✓

---

Example 9 — Cell I: real-world CFD surrogate word problem

1. $t=0$: ${\eta }_0=0.05/(1+0)=0.05$. Yeh step kyun? Early on hum bade exploratory steps chahte hain saddles escape karne ke liye.
2. $t=100$: $0.05/(1+0.01\cdot 100)=0.05/2=0.025$. Halved.
3. $t=900$: $0.05/(1+9)=0.05/10=0.005$. Yeh step kyun? Late training mein tiny steps chahiye taaki noisy path zig-zagging band kare aur minimum mein settle ho.

Verify: $1+0.01\cdot 900=10$, toh $0.05/10=0.005$; ratio $0.05/0.005=10$. Units: learning rate throughout dimensionless. ✓ Yeh parent ke ${\eta }_t={\eta }_0/(1+kt)$ schedule se match karta hai.

---

Example 10 — Cell J: exam twist, $\beta$ ke liye solve karo

1. Set up: $\frac{1}{1-\beta }=20\Rightarrow 1-\beta =\frac{1}{20}=0.05\Rightarrow \beta =0.95$. Yeh step kyun? Momentum ki steady-state velocity $g/(1-\beta )$ hai (geometric sum, Ex 3); target ke liye invert karo.
2. Default check: $\beta =0.9\Rightarrow 1/(1-0.9)=10\times$.
3. Bump: $\beta =0.99\Rightarrow 1/(1-0.99)=100\times$ — ek tiny $\beta$ change (0.9→0.99) speed-up ko 10-fold multiply kar deta hai, kyunki $1-\beta$ nonlinearly shrink hoti hai. Yeh step kyun? Map $\beta \mapsto 1/(1-\beta )$ near $\beta =1$ blow up karta hai; wahi sensitivity hai isliye $\beta$ top decimals mein tune hoti hai.

Verify: $1/(1-0.95)=20$ ✓, $1/(1-0.9)=10$ ✓, $1/(1-0.99)=100$ ✓.

---

Recall Self-test: pehle cell ka naam lo, phir answer do

Kaun sa optimizer saddle pe move karta hai jahan $g=0$ abhi hai, aur kyun? ::: Momentum aur Adam (stored velocity / $m$ coast karta hai); SGD stall karta hai kyunki uski koi memory nahi (Cell E). Steady slope pe $\beta =0.9$ ke saath, momentum speed-up kya hai aur yeh kahan se aata hai? ::: $10\times$, geometric sum $v_{\infty }=g/(1-\beta )$ se (Cell C). Adam $g=1000$ aur $g=0.001$ dono ke liye $\approx \pm \eta$ step kyun deta hai? ::: Yeh $\sqrt{\hat{v}}\approx |g|$ se divide karta hai, magnitude cancel karta hai — ek per-coordinate normalization (Cell F). Pure-noise direction mein Adam ka step zero kyun ho jaata hai? ::: $\hat{m}\to 0$ (mean gradient) lekin $\sqrt{\hat{v}}\to \sqrt{\text{Var}}>0$, toh ratio $\to 0$ (Cell G). Bias correction chote $t$ pe sabse zyada kyun matter karta hai? ::: Factor $1-{\beta }_1^t$ early pe tiny hota hai ($t=1$ pe 0.1), toh uncorrected $m$ badly underestimate karta hai $g$ ko (Cell H).

Parent derivations pe wapas jao parent derivations · Hinglish version: 5.6.09 Optimization — SGD, momentum, Adam — derivations (Hinglish)