AdaGrad and RMSprop — Hinglish notes

3.2.4 · AI-ML › Training Deep Networks

Adaptive learning-rate optimizers jo har parameter ko uski apni step size dete hain, uske gradients ki history ke hisaab se scale karke.

Core problem (YE KYUN EXIST KARTE HAIN)

KYA chahiye hame: ek aisi rule jo automatically un parameters ke liye step shrink kare jinke gradients bade/frequent hain, aur un parameters ke liye step bada rakhe jinke gradients chhote/rare hain.

KAISE: har parameter ke liye accumulated squared gradient track karo, aur learning rate ko uske square root se divide karo.

AdaGrad — scratch se derivation

Maano $g_{t} = \nabla_{θ} L$ step $t$ par, aur ek coordinate $i$ consider karo to $g_{t, i}$ ek scalar hai.

Idea: coordinate $i$ mein gradient ka "typical size" uske squares ko sum karke capture hota hai. Bade gradients ⇒ bada sum ⇒ hame ek chhoti effective step chahiye. Kisi "size" ko divisor mein convert karne ka natural tarika yeh hai ki us se divide karo. Squared isliye kyunki yeh hamesha positive rehta hai aur bade gradients ko zyada weight deta hai.

Running sum of squares define karo: $G_{t, i} = \sum_{τ = 1}^{t} g_{τ, i}^{2} = G_{t - 1, i} + g_{t, i}^{2}$

Tab update yeh hai:

$G$ kyun, $G$ kyun nahi? Units ki wajah se. $G$ ke units (gradient) $^{2}$ hain; $g$ ko $G$ se divide karne par ratio dimensionless-ish ban jaata hai aur ek well-behaved, bounded step milti hai. $G$ se divide karne par over-shrink ho jaata.

AdaGrad ki fatal kharabi

Kyunki $G_{t, i}$ ek sum hai jo sirf badhta hai, $G_{t, i} \to \infty$ , isliye effective learning rate $\to 0$ . Deep learning training kaafi steps chalti hai, AdaGrad mar jaata hai — yeh bahut jaldi seekhna band kar deta hai. Convex / sparse problems (NLP embeddings) ke liye achha hai, lekin lambi deep-net training ke liye bura hai.

RMSprop — fix

KYUN: hum nahi chahte ki poori history hamesha ke liye accumulate hoti rahe; hame gradient magnitude ka ek recent estimate chahiye. Purane gradients bhool jaane chahiye.

KAISE: badhte sum ki jagah squared gradients ka ek exponentially weighted moving average (EWMA) use karo. Yeh hamesha badhne wale sum ki jagah ek decaying memory rakhta hai.

Decay $ρ$ ke saath define karo (typically $0.9$ ): $E [g^{2}]_{t, i} = ρ E [g^{2}]_{t - 1, i} + (1 - ρ) g_{t, i}^{2}$

Yeh ek weighted average hai: recent squares puri tarah count hote hain, purane geometrically fade ho jaate hain. Ab yeh blow up nahi hota — yeh recent mean-square gradient par settle ho jaata hai.

Kyunki $E [g^{2}]$ ek bounded average rehta hai (sum nahi), effective learning rate zero tak decay nahi hoti — training chalti rehti hai.

Worked example 1 — do coordinates par AdaGrad

Maano $η = 0.1$ , $ϵ = 0$ . Coordinate A hamesha $g = 1$ deta hai; coordinate B hamesha $g = 0.1$ deta hai.

Step 1:

$G_{A} = 1^{2} = 1$ , step $= \frac{0.1}{1} \cdot 1 = 0.1$ . Kyun? Pehla gradient scale set karta hai.
$G_{B} = 0. 1^{2} = 0.01$ , step $= \frac{0.1}{0.01} \cdot 0.1 = \frac{0.1}{0.1} \cdot 0.1 = 0.1$ .

Interesting kyun hai: bhaale B ka raw gradient 10× chhota hai, AdaGrad ne uski step ko rescale karke A ke barabar kar diya. Yahi equalizing effect hai.

Step 2:

$G_{A} = 2$ , step $= \frac{0.1}{2} \cdot 1 \approx 0.0707$ . Kyun? History accumulate ho gayi, step shrink ho gayi.

Worked example 2 — RMSprop alive rehta hai

$η = 0.1$ , $ρ = 0.9$ , $ϵ = 0$ , constant $g = 1$ .

$E [g^{2}]$ converge karta hai: steady state mein $E = 0.9 E + 0.1 (1) \Rightarrow E = 1$ . To step $= \frac{0.1}{1} \cdot 1 = 0.1$ hamesha ke liye. Yeh step kyun? EWMA true mean-square (1) par saturate ho jaata hai, ek constant effective rate deta hai — koi death nahi. AdaGrad se compare karo jahan $G_{t} = t \to \infty$ aur step $\to 0$ .

Flashcards

AdaGrad/RMSprop kaun sa global problem solve karte hain?

Ek single global learning rate un parameters ke liye suitable nahi ho sakta jinke gradient magnitudes bahut alag hain; yeh per-parameter adaptive rates dete hain.

AdaGrad accumulator update?

G_{t} = G_{t - 1} + g_{t}^{2}

(sum of all past squared gradients).

AdaGrad parameter update?

θ_{t + 1} = θ_{t} - \frac{η}{G _{t} + ϵ} g_{t}

AdaGrad eventually seekhna kyun band kar deta hai?

G_{t}

bina bound ke badhta rehta hai, isliye

η / G_{t} \to 0

; effective learning rate zero tak decay ho jaati hai.

RMSprop use fix kaise karta hai?

Badhte sum ki jagah EWMA use karta hai:

E [g^{2}]_{t} = ρE [g^{2}]_{t - 1} + (1 - ρ) g_{t}^{2}

, jo bounded rehta hai.

RMSprop update?

θ_{t + 1} = θ_{t} - \frac{η}{E [ g ^{2} ] _{t} + ϵ} g_{t}

RMSprop ke liye typical

ρ

aur

η

ρ \approx 0.9

η \approx 1 0^{- 3}

Gradient ko square kyun karte hain (abs value kyun nahi)?

Squaring bade gradients ko zyada weight deta hai aur ek differentiable, variance-jaisa measure deta hai;

proper units restore karta hai.

Constant gradient

g

ke liye steady-state RMSprop step?

E [g^{2}] \to g^{2}

, isliye effective step

\to η

(constant, AdaGrad se alag).

ϵ

ka role?

Numerical stability — zero se division rokta hai jab accumulator tiny ho.

Recall Feynman: 12 saal ke bachche ko explain karo

Socho tum fog mein sabse nichle point tak pahunchne ke liye pahaad se neeche ja rahe ho. Kuch directions steep cliffs hain, kuch gentle slopes. Agar tum har jagah same size ka step loge, to cliffs par giroge aur slopes par crawl karoge. AdaGrad dekhta hai ki har direction kitni bumpy rahi hai aur bumpy directions par chhote steps leta hai, smooth directions par bade steps. Problem: AdaGrad hamesha ke liye yaad rakhta hai, isliye kuch der baad yeh har jagah micro-steps le raha hota hai aur basically ruk jaata hai. RMSprop zyada smart hai — yeh sirf recent bumpiness yaad rakhta hai aur purane bumps bhool jaata hai, isliye yeh poore raaste achhi pace se chalta rehta hai.

Connections

Stochastic Gradient Descent — yeh baseline hai jise yeh improve karte hain
Momentum — gradients smooth karta hai (first moment) vs yeh jo variance se scale karte hain (second moment)
Adam Optimizer — literally = RMSprop + momentum + bias correction
Exponentially Weighted Moving Averages — RMSprop ke peeche ka math engine
Learning Rate Schedules — adaptivity ka alternative/complement
Vanishing and Exploding Gradients — per-coordinate scaling kyun help karta hai