Visual walkthrough — Optimization — SGD, momentum, Adam — derivations

5.6.9 · D2 · Coding › Machine Learning (Aerospace Applications) › Optimization — SGD, momentum, Adam — derivations

---

Step 1 — Ek ball ek hill par: "descend" ka matlab kya hai

KYA HAI. Ek landscape imagine karo. Position $\theta$ par ground ki height ek number hai jise hum loss $L(\theta )$ kehte hain. "Training" ka matlab hai: ball ko lowest point tak roll karo.

PEHLE PICTURE KYU. Kisi bhi formula se pehle, tumhe dekhna hoga ki optimization bas downhill chalna hai. Is page ke har optimizer ka ek rule hota hai — "kis taraf, kitna door?"

PICTURE. Figure mein, $\theta$ (parameter) left–right jaata hai. $L(\theta )$ (loss) curve ki height hai. Ball ${\theta }_0$ par baithi hai; arrow wohi sensible move dikhata hai — valley ki taraf.

---

Step 2 — Slope humein batata hai neeche kaun sa taraf hai

KYA HAI. Ball ki position par, woh line draw karo jo curve ko bas chhuti hai — yeh tangent hai. Iska steepness derivative hai, jise $L^{\prime }(\theta )$ likhte hain, ya many-dimensions language mein gradient $\nabla L(\theta )$.

YEH TOOL KYU AUR KOI NAHI. Humein ek local jawab chahiye — "yahan abhi neeche kaun sa taraf hai?" — sirf woh use karke jo hum ek point par measure kar sakte hain. Derivative exactly wahi hai: tangent ka rise-over-run. Positive slope matlab ground right ki taraf chadhti hai, toh hum left jaayein; negative slope matlab right jaao. Dono taraf, hum slope ke khilaf step lete hain.

PICTURE. Green tangent line uphill jhuk rahi hai. Red arrow $-L^{\prime }({\theta }_0)$ opposite direction mein point karta hai: downhill. Woh single sign-flip yahan ke har method ka seed hai.

Anti-parallel kyun optimal hai (sirf "ek" downhill direction nahi) — yeh Taylor Expansion + Cauchy–Schwarz argument parent note mein hai; yahan hum picture par trust karte hain: slope ke opposite sabse fast girta hai.

---

Step 3 — Gradient noisy hai: SGD ka entry

KYA HAI. Real training mein hum true slope $g_t$ measure nahi kar sakte; hum ise ek chhote random mini-batch se estimate karte hain, jo ek wobbly arrow deta hai ${\hat{g}}_t=g_t+(\text{noise})$.

KYU. Millions of samples par exact slope compute karna bahut slow hai. Mini-batch sasta hai aur, average par, correct hai — iski expected value true slope ke barabar hoti hai (ek unbiased estimate). Backpropagation dekho ki har $\hat{g}$ actually kaise compute hota hai.

PICTURE. True downhill arrow ek clean direction hai. Mini-batch arrows uske around scatter hote hain — kuch bahut lambe, kuch tilted. Inhe blindly follow karo aur tumhara path zig-zag karega.

---

Step 4 — Ravine problem, aur momentum ka jawab

KYA HAI. Bahut saare loss surfaces ravines hote hain: across steep, along almost flat. SGD steep walls se bounce karta hai (fast oscillation) aur flat floor par creep karta hai (slow progress). Momentum ek running average rakhta hai jise velocity $v_t$ kehte hain, taaki consistent directions build up hon aur oscillating wale cancel ho jaayein.

AVERAGE KYU. Oscillation ka matlab hai across-arrows baar baar sign flip karte hain; unhe add karne se cancel hota hai. Along-arrows sab ek hi taraf point karte hain; unhe add karne se accumulate hota hai. Ek decaying running sum dono ek saath karta hai — yeh ek Exponential Moving Average hai.

PICTURE. Left: SGD ka saw-tooth path ravine ke across. Right: momentum ka smoothed path — sideways wiggles annihilate ho jaate hain, forward drive badhta hai.

---

Step 5 — Momentum $\frac{1}{1-\beta }$ amplifier kyun hai

KYA HAI. Velocity unroll karo: $v_t={\sum }_{k=0}^t{\beta }^k{g}_{t-k}$. Steady slope par $g_t=g$ (constant), yeh geometric sum settle hota hai $v_{\infty }=\frac{g}{1-\beta }$ par.

YEH KYU MATTER KARTA HAI. $\beta =0.9$ ke saath, factor $\frac{1}{1-\beta }=10$: momentum 10× faster sprint karta hai SGD se us kisi bhi direction mein jis par gradient agree karta rehta hai. Yahi "acceleration" hai.

PICTURE. Bars ${\beta }^k$ geometrically shrink karte hain ($1,0.9,0.81,\dots$); unki heights $\frac{1}{1-\beta }$ par dashed ceiling tak sum hoti hain. Recent gradients sabse zyada count hote hain; purane fade ho jaate hain.

---

Step 6 — Adam ka second moment: gradient size measure karna

KYA HAI. Adam do running averages rakhta hai har parameter ke liye: mean $m_t$ (momentum, "kaun sa taraf") aur mean square $v_t$ (magnitude, "kitna bada aur kitna noisy"). $m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2.$

SQUARE KYU KARTE HAIN. Squaring sign throw away kar deta hai, toh $v_t$ raw strength measure karta hai direction se regardless. Bada $v_t$ matlab "is coordinate mein large ya noisy gradients hain"; chhota $v_t$ matlab "quiet." Isse har parameter apna khud ka step size pa sakta hai — vital jab early aur late network layers ke scales bilkul alag hon.

PICTURE. Do coordinates: ek mein large steady gradient, ek mein tiny. Unke $m$ (blue) aur $v$ (orange) EMAs direction aur magnitude ko alag alag track karte hain.

---

Step 7 — Cold-start bias, aur uska fix

KYA HAI. Dono averages zero se start hote hain: $m_0=v_0=0$. Shuruaat mein woh zero ki taraf drag hote hain — ek underestimate. Constant $g$ ke saath unroll karo toh $\mathbb{E}[m_t]=g(1-{\beta }_1^t)$ milta hai, jo $g$ se chhota hota hai jab $t$ tiny ho. Fix yeh hai ki shrinkage divide kar do: ${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}.$

KYU. Correction ke bina Adam ke pehle steps absurdly small honge (empty memory se start karte hue), training stall ho jaayegi. $1-{\beta }_1^t$ se divide karna exactly cold-start undo karta hai.

PICTURE. Raw $m_t$ (dashed) $0$ se creep karta hai; corrected ${\hat{m}}_t$ (solid) step one se hi true value $g$ par baith jaata hai. Jaise $t$ badhta hai, $1-{\beta }_1^t\to 1$ aur dono curves merge ho jaate hain — correction politely fade ho jaata hai.

---

Step 8 — Sab kuch jodte hain: unit step (saare cases)

KYA HAI. Update assemble karo: ${\theta }_{t+1}={\theta }_t-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}.$ Ab har regime check karo jo ratio ${\hat{m}}_t/\sqrt{{\hat{v}}_t}$ hit kar sakta hai:

Regime	${\hat{m}}_t$	${\hat{v}}_t$	ratio	step liya
steady slope $g$	$\to g$	$\to g^2$	$\frac{g}{|g|}=\pm 1$	poora $\pm \eta$ — confident
pure noise (mean 0)	$\to 0$	$\to \text{Var}>0$	$\to 0$	~kuch nahi — noise ignore
dead coordinate $g=0$	$0$	$0$	$\frac{0}{0+ϵ}=0$	$0$ — $ϵ$ divide bachata hai

KYU. Yahi Adam ka poora point hai: direction ko magnitude se divide karo taaki stride length $\approx \eta$ ho chahe gradient kitna bhi bada ho. Confident directions clean unit step lete hain; noisy directions damp ho jaate hain; degenerate zeros $ϵ$ se safe ho jaate hain (division by zero nahi).

PICTURE. Teen coordinates side by side, har ek ke saath raw gradient (grey) aur uska Adam step (magenta). Bade aur chhote raw gradients dono same-length magenta step produce karte hain; noisy wala almost kuch nahi produce karta.

---

Ek-picture summary

Sab ek saath: ek ball ravine descend kar rahi hai. SGD zig-zag karta hai; momentum floor ke saath smooth aur speed karta hai; Adam har axis ko rescale karta hai taaki stride har jagah steady unit step ho. Boxed formula destination hai.

Recall Feynman retelling (plain words)

Tum ek hill par blindfolded ho aur bottom chahte ho. Tum apne pair ke neeche slope feel kar sakte ho — yahi gradient hai; uske khilaf step lo (Step 2). Lekin tumhare pair kaanpte hain, toh har felt slope thoda galat hota hai — yahi SGD ka noise hai (Step 3). Staggering rokne ke liye, tum momentum banate ho: apni recent direction yaad rakho aur roll karte raho, toh honest directions add up hote hain (10× faster!) aur kaanpne wale sideways stumbles cancel ho jaate hain (Steps 4–5). Adam ek doosra trick add karta hai: har pair ke liye alag se, woh notice karta hai ki tumhare steps usually kitne bade hote hain aur use divide kar deta hai, taaki tum hamesha ek confident stride lete ho — bade-slope pair cliff se nahi kudte, tiny-slope pair phir bhi hilte hain, aur bilkul still pair rukhe rehte hain (Steps 6–8). Kyunki tumhari memory khaali se start hoti hai, pehle strides bahut timid honge, toh tum unhe scale up karte ho jab tak memory fill na ho jaaye (Step 7). Direction momentum se, stride-size magnitude se, safety $ϵ$ se — yahi poora box hai.

Recall

Ek-line "$\pm \eta$ kyun?" ::: Steady slope par ${\hat{m}}_t\to g$ aur ${\hat{v}}_t\to g^2$, toh ratio $g/|g|=\pm 1$ hai aur step exactly $\pm \eta$ hai. Ek-line "bias correction kyun?" ::: Zero-initialized EMAs $1-{\beta }^t$ factor se low padhte hain; usse divide karna step one se hi true value restore karta hai.