Question bank — Optimization — SGD, momentum, Adam — derivations

False. Ye $L$ tabhi decrease karta hai jab $\eta$ itna chhota ho ki first-order Taylor Expansion $L(\theta )+\nabla L^{\top }\Delta \theta$ accurate rahe; bahut bada step valley ki doosri wall par overshoot kar sakta hai.

False. Ye sirf steepest local descent ki direction mein point karta hai; ek curved ya ravine-shaped loss landscape mein minimum tak ka raasta minimum ki taraf straight line se kaafi door zig-zag kar sakta hai.

False. Ye true gradient plus zero-mean noise ke barabar hota hai — ye unbiased hai ($\mathbb{E}[g_t]=\nabla L$), isliye error average out ho jaata hai, ye koi constant offset nahi hai.

False. $1/(1-\beta )=10\times$ amplification sirf un directions mein hoti hai jahan gradient ka sign consistent ho; oscillating directions mein alternating terms add hone ki jagah cancel ho jaate hain.

False. Adam phir bhi $\eta$ se multiply karta hai; $\sqrt{{\hat{v}}_t}$ term sirf step ko roughly $\pm \eta$ par normalize karta hai, ye overall scale set nahi karta.

False. Factor $1-{\beta }_1^t\to 1$ jaise $t$ badhta hai, isliye correction early mein bahut badi hoti hai ($t=1$ par, $\hat{m}=m/(1-{\beta }_1)=10m$) aur baad mein negligible ho jaati hai.

False. Decaying schedule ${\eta }_t={\eta }_0/(1+kt)$ (dekho Learning Rate Scheduling) ke saath noise term waqt ke saath shrink hota hai, jisse SGD minimum mein settle ho sakta hai.

False. Ye exact hai lekin deterministic hai, isliye ye saddle points par ruk jaata hai, aur ye per step $O(N)$ hai — SGD ka noise saddles se escape karta hai aur per second kaafi zyada updates deta hai.

Error ye hai: velocity sirf tabhi shrink hoti hai jab naye gradients aane band ho jaayein; jab tak $g_t$ same sign ke saath aata rahe, $v_t$ zero ki taraf nahi balki $g/(1-\beta )$ ki taraf badhta hai.

Ye ${\hat{v}}_t$ se nahi, $\sqrt{{\hat{v}}_t}+ϵ$ se divide karta hai. Square root essential hai: ${\hat{v}}_t\approx g^2$, isliye $\sqrt{{\hat{v}}_t}\approx |g|$ aur ratio $\hat{m}/\sqrt{\hat{v}}\approx g/|g|=\pm 1$ — ek unit step. $g^2$ se divide karne par galat scale $1/|g|$ milti.

Error ye hai: $g_t^2$ gradient vector ki element-wise squaring hai, squared sum nahi. Har coordinate ka apna second-moment estimate hota hai.

Sign error: downhill $-\eta \nabla L$ hai. Gradient add karne se steepest increase ki taraf chadhaai hogi, isliye loss badhega.

Error ye hai: $ϵ$ ek numerical guard hai jo zero division rokta hai jab ${\hat{v}}_t\to 0$; ye learning rate nahi hai.

Error ye hai: ye $1-{\beta }_1^t$ se divide karta hai (dhyan do exponent $t$ par), jo har step par change hota hai aur sirf $t=1$ par $1-{\beta }_1$ ke barabar hota hai.

Error ye hai: unroll karne par $\mathbb{E}[m_t]=g(1-{\beta }_1^t)$ milta hai, jo early mein $g$ se neeche hota hai — exactly isliye correction exist karta hai.

Weights ${\beta }^k$ recent gradients ko dominate karne dete hain jabki purane fade ho jaate hain, isliye velocity changing landscape ke saath adapt hoti hai instead of stale directions se drag hone ke — ye Exponential Moving Average ka ek form hai.

Pure noise mein ${\hat{m}}_t\to 0$ (mean cancel ho jaata hai) lekin ${\hat{v}}_t\to \text{Var}>0$, isliye ratio $\hat{m}/\sqrt{\hat{v}}\to 0$ — Adam wahan move karne se mana kar deta hai jahan koi consistent signal nahi hai.

Normalize karna har jagah equal-length steps deta, ye ignore karte hue ki minimum ke paas gradient shrink hoti hai aur hum chahte hain chhote steps; $-\eta \nabla L$ ko un-normalized rakhne se flat regions ke paas auto-slow ho jaata hai.

Variance parameters ko exact saddle points se perturb karta hai, jahan true gradient zero hoti hai aur deterministic Gradient Descent hamesha ke liye freeze ho jaata.

Per-coordinate $\sqrt{{\hat{v}}_t}$ har parameter ke step ko ~$\pm \eta$ par rescale karta hai, isliye huge-gradient weight explode nahi hoti jabki tiny-gradient weight stall nahi hoti — ye ek aircraft-sensor net ki early vs late layers mein common situation hai.

Laakhon nonlinear parameters ke saath is equation ka koi closed form nahi hai aur countless solutions hain (minima, maxima, saddles); iterative descent hi ek tractable raasta hai.

$v_0=\beta \cdot 0+g_0=g_0$ ke saath, pehla momentum step ek plain SGD step ke barabar hota hai — momentum SGD se sirf tabhi alag hota hai jab history accumulate ho jaaye.

Dono ${\hat{m}}_t$ aur ${\hat{v}}_t$ 0 rehte hain, isliye update $0/(0+ϵ)=0$ hoga — woh parameter kabhi move nahi karta, jo sahi hai kyunki uska koi signal nahi hai.

Velocity recursion $v_t=g_t$ par collapse ho jaati hai, aur update plain SGD ban jaata hai — zero memory wala momentum sirf SGD hai.

Amplification $1/(1-\beta )\to \infty$ ho jaati hai aur purane gradients almost kabhi fade nahi hote, isliye ball barely naye terrain ko feel karta hai — ye badly overshoot karta hai aur unstable ho sakta hai.

${\hat{m}}_t\to g$, ${\hat{v}}_t\to g^2$, isliye step $-\eta g/|g|=\mp \eta$ hai — ek clean unit-scaled step sahi downhill direction mein, $|g|$ se independent.

Deterministic GD mein $\nabla L=0$ hai isliye ye completely ruk jaata hai; SGD ka noisy $g_t\ne 0$ ise saddle se nudge karta hai aur descent resume hone deta hai.

Step local linear model ko overshoot kar deta hai, isliye $L$ increase ho sakta hai aur iterates oscillate ya diverge kar sakte hain — poora "descent" guarantee chhote $\eta$ par depend karta hai.