Noesis
3.2.6Training Deep Networks

Learning rate scheduling

2,004 words9 min readdifficulty · medium6 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is it?

WHY do we need it?

  1. Convergence guarantees. For SGD to converge to a minimum of a noisy objective, classical theory (Robbins–Monro) requires the step sizes to shrink. Constant η\eta makes the parameters "buzz" around the minimum with variance proportional to η\eta — they never truly settle.
  2. Speed vs precision trade-off. Big η\eta early = fast escape from bad regions. Small η\eta late = fine-tuning near the optimum.
  3. Escaping plateaus / warmup. Very early in training, gradients and Adam's running statistics are noisy; a warmup (starting small and ramping up) avoids blowing up.

HOW: deriving why the LR must decay

Derivation of the steady-state variance (from scratch). Take the variance of both sides, assuming ξt\xi_t is independent of θt\theta_t: Var⁡(θt+1)=(1−ηa)2Var⁡(θt)+η2σ2.\operatorname{Var}(\theta_{t+1}) = (1-\eta a)^2 \operatorname{Var}(\theta_t) + \eta^2\sigma^2. At steady state Var⁡(θt+1)=Var⁡(θt)=V\operatorname{Var}(\theta_{t+1})=\operatorname{Var}(\theta_t)=V: V=(1−ηa)2V+η2σ2  ⇒  V(1−(1−ηa)2)=η2σ2.V = (1-\eta a)^2 V + \eta^2 \sigma^2 \;\Rightarrow\; V\big(1-(1-\eta a)^2\big)=\eta^2\sigma^2. Expand 1−(1−ηa)2=2ηa−η2a2=ηa(2−ηa)1-(1-\eta a)^2 = 2\eta a - \eta^2 a^2 = \eta a(2-\eta a): V=η σ2a (2−ηa)≈η σ22a  for small η.\boxed{V = \frac{\eta\,\sigma^2}{a\,(2-\eta a)} \approx \frac{\eta\,\sigma^2}{2a}\ \text{ for small }\eta.}

For convergence AND to actually make progress, the classic Robbins–Monro conditions are: ∑tη(t)=∞(can travel arbitrarily far),∑tη(t)2<∞(noise dies out).\sum_{t} \eta(t) = \infty \quad(\text{can travel arbitrarily far}),\qquad \sum_{t}\eta(t)^2 < \infty \quad(\text{noise dies out}). The schedule η(t)=η0/t\eta(t)=\eta_0/t satisfies both (harmonic series diverges, ∑1/t2\sum 1/t^2 converges) — the original theoretical schedule.

Common schedules (each derived / motivated)

Warmup + cosine (the modern default) is shown below.

Figure — Learning rate scheduling

Worked examples

Common mistakes (steel-manned)

Forecast-then-verify

Flashcards

Why must the learning rate decay for SGD to converge to the true minimum?
The stochastic gradient noise gives a steady-state variance V∝ηV\propto\eta; only η→0\eta\to0 drives that residual jitter to zero.
State the Robbins–Monro conditions on a schedule η(t)\eta(t).
∑tη(t)=∞\sum_t \eta(t)=\infty (can travel far) and ∑tη(t)2<∞\sum_t \eta(t)^2<\infty (noise vanishes).
Give a schedule satisfying Robbins–Monro exactly.
η(t)=η0/(1+λt)\eta(t)=\eta_0/(1+\lambda t) (the 1/t1/t schedule).
Cosine annealing formula?
η(t)=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(πt/T))\eta(t)=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})(1+\cos(\pi t/T)).
What are η\eta at t=0t=0 and t=Tt=T for cosine annealing?
ηmax⁡\eta_{\max} at t=0t=0, ηmin⁡\eta_{\min} at t=Tt=T.
What is learning-rate warmup and why is it needed?
Linearly ramp η\eta from ~0 to ηmax⁡\eta_{\max} over the first TwT_w steps; needed because early Adam variance estimates are unreliable, so unscaled early steps can diverge.
Step decay formula?
η(t)=η0γ⌊t/s⌋\eta(t)=\eta_0\gamma^{\lfloor t/s\rfloor}.
Steady-state variance near a quadratic minimum under constant η\eta?
V=ησ2a(2−ηa)≈ησ22aV=\dfrac{\eta\sigma^2}{a(2-\eta a)}\approx\dfrac{\eta\sigma^2}{2a}.
Why not just use a tiny constant LR?
You still plateau above the minimum (residual variance) AND waste compute; decaying gives both speed and precision.
Half-life of exponential decay η0e−λt\eta_0e^{-\lambda t}?
t1/2=ln⁡2/λt_{1/2}=\ln 2/\lambda.
Recall Feynman: explain to a 12-year-old

You're walking downhill in fog to reach the lowest point. At the top you take big confident steps to get down fast. As you sense you're near the bottom, you take tiny careful steps so you don't trip over the lowest spot and walk back up. A learning rate schedule is just the plan for how quickly you shrink your steps. And right at the start, when you can't see anything yet, you take a few slow steps to get your balance — that's "warmup."

Connections

  • Stochastic Gradient Descent — the update rule η\eta multiplies.
  • Adam and Adaptive Optimizers — why warmup pairs with adaptive methods.
  • Loss Landscapes and Minima — flat vs sharp basins the schedule navigates.
  • Robbins-Monro Stochastic Approximation — convergence theory behind decay.
  • Batch Size and Learning Rate scaling — linear scaling rule interacts with schedules.
  • Warm Restarts (SGDR) — periodically resetting the cosine schedule.
  • Hyperparameter Tuning — η0\eta_0, TwT_w, γ\gamma as search targets.

Concept Map

varied by

causes

quantified by

proportional to eta

goal of

must satisfy

met by

big early small late

early phase uses

avoids

Learning rate eta

LR schedule eta of t

Constant eta

Buzzing noise floor

Steady-state variance V

Decay eta to zero

Robbins-Monro conditions

Schedule eta0 over t

Warmup ramp-up

Speed vs precision

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, learning rate η\eta ka matlab hai ki har step pe tum apne weights ko kitna bada update karte ho. Agar hamesha bada step rakhoge, to minimum ke aas-paas ball bounce karti rahegi — kabhi settle nahi hogi. Agar hamesha chhota rakhoge, to training bahut slow ho jaayegi. Isliye smart idea ye hai: shuruaat me bade steps (fast travel), aur dheere-dheere steps chhote karo (gently settle). Isko hi hum learning rate scheduling kehte hain.

Iske peeche maths bhi solid hai. SGD me gradient me thoda noise hota hai. Agar η\eta constant rakho, to minimum ke paas bhi weights ka ek "buzzing" reh jaata hai jiska variance V∝ηV \propto \eta hota hai. Matlab jab tak η\eta ko zero ki taraf nahi le jaoge, tum true minimum tak pahunchoge hi nahi — bas uske thoda upar ghoomte rahoge. Yahi reason hai ki decay zaroori hai (Robbins–Monro conditions: ∑η=∞\sum\eta=\infty aur ∑η2<∞\sum\eta^2<\infty).

Practical schedules: step decay (har kuch epochs me aadha kar do), cosine annealing (smooth curve jo start me high rehti hai, phir aaram se girti hai), aur warmup (shuruaat me η\eta ko chhote se bada karo). Transformers me warmup bahut important hai kyunki Adam ke variance estimates shuru me kachche hote hain — bina warmup ke loss phat sakta hai. Modern default: warmup + cosine — yaad rakho "Warm up, then Cool down".

Ek common galti: socho ki cosine linear girta hai. Nahi! Cosine start aur end pe flat hota hai, beech me sabse fast girta hai. Isliye halfway time pe value halfway hoti hai (symmetry se), par baaki jagah curve straight nahi hai.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,157 wordsD2Visual walkthroughThe derivation in pictures · 2,292 wordsD3Worked examplesEvery scenario covered · 2,649 wordsD4ExercisesGraded, full solutions · 3,138 wordsD5Question bankConcept traps, quick reveals · 2,375 words

Test yourself — Training Deep Networks

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Stochastic gradient descent (SGD)AI-ML · 3.2.1Adam and AdamW optimizersAI-ML · 3.2.5Hyperparameter tuning for deep netsAI-ML · 3.2.15Local vs global minima - maximaAI-ML · 1.2.9Gradient descent intuition and update ruleAI-ML · 1.2.12Learning rate effects on convergenceAI-ML · 1.2.13Grid search and random searchAI-ML · 2.6.13Stochastic gradient descent (SGD)AI-ML · 3.2.1Mini-batch gradient descentAI-ML · 3.2.2