Learning rate scheduling — Hinglish notes

3.2.6 · AI-ML › Training Deep Networks

YEH HAI KYA?

ISKE LIYE KYU ZARURAT HAI?

Convergence guarantees. SGD ko noisy objective ke minimum tak converge karne ke liye, classical theory (Robbins–Monro) require karti hai ki step sizes shrink hon. Constant $η$ parameters ko minimum ke aas paas "buzz" karte rehne deta hai jisme variance $η$ ke proportional hota hai — yeh kabhi truly settle nahi hote.
Speed vs precision trade-off. Bada $η$ early = buri regions se fast escape. Chota $η$ late = optimum ke paas fine-tuning.
Plateaus se escape karna / warmup. Training ke bilkul shuru mein, gradients aur Adam ke running statistics noisy hote hain; ek warmup (chote se start karke ramp up karna) blowing up se bachata hai.

KAISE: derive karte hain kyun LR decay hona chahiye

Steady-state variance ki derivation (scratch se). Dono sides ka variance lo, assume karte hue ki $ξ_{t}$ , $θ_{t}$ se independent hai: $Var (θ_{t + 1}) = (1 - η a)^{2} Var (θ_{t}) + η^{2} σ^{2} .$ Steady state par $Var (θ_{t + 1}) = Var (θ_{t}) = V$ : $V = (1 - η a)^{2} V + η^{2} σ^{2} \Rightarrow V (1 - (1 - η a)^{2}) = η^{2} σ^{2} .$ $1 - (1 - η a)^{2} = 2 η a - η^{2} a^{2} = η a (2 - η a)$ expand karo: $V = \frac{η σ ^{2}}{a ( 2 - η a )} \approx \frac{η σ ^{2}}{2 a} for small η .$

Convergence ke liye AUR actually progress banane ke liye, classic Robbins–Monro conditions hain: $\sum_{t} η (t) = \infty (arbitrarily door travel kar sakte hain), \sum_{t} η (t)^{2} < \infty (noise khatam ho jaata hai) .$ Schedule $η (t) = η_{0} / t$ dono satisfy karta hai (harmonic series diverge karta hai, $\sum 1/ t^{2}$ converge karta hai) — original theoretical schedule yahi hai.

Common schedules (har ek derive / motivate kiya hua)

Warmup + cosine (modern default) neeche dikhaya gaya hai.

Worked examples

Common mistakes (steel-manned)

Forecast-then-verify

Flashcards

SGD ko true minimum tak converge karne ke liye learning rate kyun decay honi chahiye?

Stochastic gradient noise ek steady-state variance

V \propto η

deta hai; sirf

η \to 0

hi us residual jitter ko zero tak drive karta hai.

Schedule

η (t)

par Robbins–Monro conditions bolo.

\sum_{t} η (t) = \infty

(door travel kar sakte hain) aur

\sum_{t} η (t)^{2} < \infty

(noise khatam ho jaata hai).

Ek schedule do jo Robbins–Monro ko exactly satisfy kare.

η (t) = η_{0} / (1 + λ t)

(

1/ t

schedule).

Cosine annealing formula?

η (t) = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + cos (π t / T))

Cosine annealing mein

t = 0

aur

t = T

par

η

kya hote hain?

t = 0

par

η_{m a x}

t = T

par

η_{m i n}

Learning-rate warmup kya hai aur kyun zaroori hai?

Pehle

T_{w}

steps mein

η

ko ~0 se

η_{m a x}

tak linearly ramp karo; zaroori hai kyunki early Adam variance estimates unreliable hote hain, isliye unscaled early steps diverge kar sakte hain.

Step decay formula?

η (t) = η_{0} γ^{⌊ t / s ⌋}

Constant

η

ke under quadratic minimum ke paas steady-state variance?

V = \frac{η σ ^{2}}{a ( 2 - η a )} \approx \frac{η σ ^{2}}{2 a}

Sirf tiny constant LR kyun use nahi karna chahiye?

Tum phir bhi minimum ke upar plateau karte ho (residual variance) AUR compute waste hota hai; decaying dono speed aur precision deta hai.

Exponential decay

η_{0} e^{- λ t}

ki half-life?

t_{1/2} = ln 2/ λ

Recall Feynman: 12-saal ke bachche ko samjhao

Tum fog mein lowest point tak pahunchne ke liye pahadi se neeche chal rahe ho. Upar se tum bade confident steps lete ho taaki jaldi neeche aao. Jaise tumhe lagta hai tum bottom ke paas ho, tum bahut chote careful steps lete ho taaki lowest spot par trip na karo aur wapas upar na chale jao. Ek learning rate schedule bas yeh plan hai ki tum apne steps kitni jaldi shrink karte ho. Aur bilkul shuru mein, jab tum kuch bhi nahi dekh sakte, tum apna balance paane ke liye kuch slow steps lete ho — yahi "warmup" hai.

Connections

Stochastic Gradient Descent — update rule jisme $η$ multiply hota hai.
Adam and Adaptive Optimizers — kyun warmup adaptive methods ke saath kaam karta hai.
Loss Landscapes and Minima — flat vs sharp basins jinhe schedule navigate karta hai.
Robbins-Monro Stochastic Approximation — decay ke peeche convergence theory.
Batch Size and Learning Rate scaling — linear scaling rule schedules ke saath interact karta hai.
Warm Restarts (SGDR) — cosine schedule ko periodically reset karna.
Hyperparameter Tuning — $η_{0}$ , $T_{w}$ , $γ$ search targets ke roop mein.