Think about what one SGD step does:
θt+1=θt−ηgt,gt=∇θL(θt;batcht)
Reason 1 — Bad early gradients. With random init the loss surface is chaotic; gt has large magnitude and huge variance across batches. A large η multiplies that noise → the step ηgt can be enormous → weights blow up (loss → NaN) or land in a bad basin.
Reason 2 — Adam's variance estimate is unreliable early. Adaptive optimizers estimate a running second moment vt (an estimate of E[g2]). Early on, with only a few samples, vt is a poor estimate, so the effective step size η/vt swings wildly. Warmup keeps η small until vt has enough samples to be trustworthy. (This is the exact motivation behind RAdam.)
Reason 3 — Large-batch training. When you scale batch size B, you often scale η up too (linear scaling rule). But a big η from step 0 is unstable, so warmup is what makes the large LR usable — famously in the "ImageNet in 1 hour" work.
Imagine you just woke up and your legs are stiff. If you sprint immediately you'll pull a muscle and be out for the day. So you jog slowly first, get warm, then sprint. A neural network is the same: right after it's "born" (random weights), it's clumsy and would hurt itself with big steps. So we let it take tiny learning steps first, grow them slowly, and only then take big confident steps. That slow-start-then-speed-up is warmup.
Increasing the LR from a small value to the peak value over the first Tw steps before the main schedule begins.
Why are early gradients dangerous for a large LR?
With random init, gradients are large and high-variance; ηgt can blow up weights or land in a bad basin.
Formula for linear warmup with zero start?
η(t)=ηpeakt/Tw for t≤Tw.
Why does warmup especially help Adam?
The second-moment estimate vt is unreliable with few samples early, making the effective step η/vt swing wildly; small η tames this until vt is trustworthy.
General two-point linear warmup formula?
η(t)=ηstart+(ηpeak−ηstart)t/Tw.
Value of cosine decay at its midpoint (progress = 0.5)?
Exactly ηpeak/2, since cos(π/2)=0.
What must you almost always pair warmup with?
A decay schedule (e.g., cosine or step) for the post-warmup phase.
In what units is Tw measured?
Optimizer steps, not epochs.
Which optimizer method builds warmup-like behavior in automatically?
RAdam (rectified Adam), by rectifying the variance early.
How does warmup enable the linear-scaling rule for large batches?
It lets you use the large scaled LR safely by ramping into it instead of applying it from step 0.
Dekho, training ke bilkul shuru mein tumhare model ke weights random hote hain, aur gradients bahut bade aur noisy hote hain. Agar tum wahin pe bada learning rate laga do, to ek hi kharab batch tumhare weights ko itna hila dega ki loss NaN ho jayega ya model kabhi recover nahi karega. Isliye warmup karte hain: pehle chhota LR se start karo, phir dheere-dheere linearly usko peak value tak badhao (say pehle 1000 steps mein), uske baad normal schedule (jaise cosine decay) le lo. Bilkul jaise gym mein pehle warm-up karte ho, phir heavy weight uthate ho.
Warmup ka sabse bada faayda stability hai jab model "kaccha" hota hai, aur speed jab surface smooth ho jaata hai. Adam jaise adaptive optimizers ke liye ye aur bhi zaroori hai, kyunki wo vt (gradient ke square ka average) estimate karte hain — shuru mein kam samples hone ki wajah se ye estimate galat hota hai, aur effective step η/vt pagalon ki tarah jump karta hai. Warmup us waqt LR chhota rakh ke ise sambhal leta hai.
Formula simple hai: linear warmup mein η(t)=ηpeak⋅t/Tw. Yaani agar aadhe raste ho to aadha peak LR. Ek important baat: Tw ko steps mein naapo, epochs mein nahi — kyunki batch size badalne se steps-per-epoch badal jaata hai aur tumhara warmup chupke se chhota-bada ho jaata hai. Aur warmup ke baad decay lagana mat bhoolna, warna peak pe hi atke rahoge.
