Learning rate warmup

3.2.7 · AI-ML › Training Deep Networks

WHAT hai warmup?

Do sabse common shapes:

• Linear warmup: $\eta (t)={\eta }_{\text{peak}}\cdot \frac{t}{T_w}$ for $t\le T_w$.
• Exponential/constant-then-ramp: kam common; linear practically zyada use hoti hai.

WHY help karta hai? (first-principles reasoning)

Socho ek SGD step kya karta hai: ${\theta }_{t+1}={\theta }_t-\eta g_t,g_t={\nabla }_{\theta }\mathcal{L}({\theta }_t;{\text{batch}}_t)$

Reason 1 — Kharab early gradients. Random init ke saath loss surface chaotic hoti hai; $g_t$ ki magnitude badi aur batches mein variance bahut zyada hota hai. Bada $\eta$ us noise ko multiply karta hai → step $\eta g_t$ enormous ho sakta hai → weights blow up ho jaate hain (loss → NaN) ya kisi bure basin mein land karte hain.

Reason 2 — Adam ka variance estimate early mein unreliable hota hai. Adaptive optimizers ek running second moment $v_t$ estimate karte hain (jo $\mathbb{E}[g^2]$ ka estimate hai). Shuruaat mein, sirf kuch samples ke saath, $v_t$ ek bekar estimate hota hai, isliye effective step size $\eta /\sqrt{v_t}$ bahut zyada swing karta hai. Warmup $\eta$ ko chhota rakhta hai jab tak $v_t$ ke paas trustworthy hone ke liye kaafi samples na aa jaayein. (Yahi RAdam ke peeche exact motivation hai.)

Reason 3 — Large-batch training. Jab tum batch size $B$ scale karte ho, tum aksar $\eta$ bhi upar scale karte ho (linear scaling rule). Lekin step 0 se bada $\eta$ unstable hai, isliye warmup hi hai jo bade LR ko usable banata hai — "ImageNet in 1 hour" kaam mein famous tarike se.

HOW compute karo — linear schedule derive karo

Hum chahte hain ek function $\eta (t)$ jo:

1. $t=0$ par ${\eta }_{\text{start}}$ ke barabar ho,
2. $t=T_w$ par ${\eta }_{\text{peak}}$ ke barabar ho,
3. beech mein ek straight line ho (sabse simple continuous choice).

Do points $(0,{\eta }_{\text{start}})$ aur $(T_w,{\eta }_{\text{peak}})$ se ek line: $\eta (t)={\eta }_{\text{start}}+({\eta }_{\text{peak}}-{\eta }_{\text{start}})\cdot \frac{t}{T_w},0\le t\le T_w$

${\eta }_{\text{start}}=0$ set karna (common default) simplify karke yeh deta hai: $\boxed{\eta (t)={\eta }_{\text{peak}}\cdot \frac{t}{T_w}}$

Warmup ke baad ($t>T_w$) hum main schedule ko de dete hain. Ek bahut common full recipe hai warmup + cosine decay total steps $T$ par:

Worked examples

Common mistakes

Active recall

Connections

• Learning rate schedules
• Cosine annealing
• Adam optimizer
• RAdam
• Linear scaling rule (large batch training)
• Gradient clipping
• Weight initialization
• Transformer training recipe

Concept Map