Exercises — Learning rate warmup

Answer: (b). A constant tiny LR is already stable at the start, so warmup adds nothing there. What warmup adds on top of a tiny LR is the ability to eventually reach the large peak LR that trains fast. Compared to a constant huge LR the answer would be (a); compared to a tiny one it is the speed. Since the question is specifically vs. "constant tiny," the missing piece is speed later → (b).

Optimizer steps. One step = one weight update = one call to the optimizer on one batch. Epochs are unreliable because changing the batch size changes how many steps fit in an epoch, silently changing your true warmup length.

False. It became famous via the Transformer paper, but it helps any setup with shaky early gradients — deep ResNets, large-batch training, and adaptive optimizers all benefit. See Transformer training recipe for where it is most famous, but it is a general stability tool.

Zero-start linear warmup: $\eta (t)={\eta }_{\text{peak}}t/T_w$. Fraction through warmup: $200/800=0.25$. $\eta (200)=2\times 10^{-3}\cdot 0.25=5\times 10^{-4}.$ WHAT/WHY: a quarter of the way up the orange line → a quarter of the peak height.

Use the general two-point line: $\eta (200)=1\times 10^{-4}+(2\times 10^{-3}-1\times 10^{-4})\cdot \frac{200}{800}.$ The gap is $2\times 10^{-3}-1\times 10^{-4}=1.9\times 10^{-3}$. Scale by $0.25$: $4.75\times 10^{-4}$. $\eta (200)=1\times 10^{-4}+4.75\times 10^{-4}=5.75\times 10^{-4}.$ WHY it differs from 2.1: the fraction now scales the gap between start and peak, not the peak itself — then you add back the start offset.

Past warmup → cosine branch. Progress: $x=\frac{11000-1000}{21000-1000}=\frac{10000}{20000}=\mathrm{0.5.}$ With ${\eta }_{\text{end}}=0$ the cosine collapses to $\frac{{\eta }_{\text{peak}}}{2}(1+\cos \pi x)$: $\eta =\frac{10^{-3}}{2}(1+\cos (\pi \cdot 0.5))=\frac{10^{-3}}{2}(1+0)=5\times 10^{-4}.$ WHY: at exactly half the decay, $\cos (\pi /2)=0$, so cosine sits at its midpoint → half of peak.

Evaluate both branches at $t=T_w$ (take ${\eta }_{\text{end}}=0$). Warmup branch: ${\eta }_{\text{peak}}\cdot T_w/T_w={\eta }_{\text{peak}}$. Cosine branch at $x=\frac{T_w-T_w}{T-T_w}=0$: $\frac{{\eta }_{\text{peak}}}{2}(1+\cos 0)=\frac{{\eta }_{\text{peak}}}{2}(1+1)={\eta }_{\text{peak}}$. They match → the schedule is continuous at the handoff, no jump. That is exactly why cosine was chosen: it starts at ${\eta }_{\text{peak}}$ with (as the parent showed) zero slope, so the transition looks seamless on the figure — the orange line's top and the teal curve's start touch smoothly.

Zero-start linear warmup: $\eta (t)={\eta }_{\text{peak}}t/T_w$. The fraction $t/T_w=500/1000=0.5$ is unchanged. Before: $10^{-3}\cdot 0.5=5\times 10^{-4}$. After: $2\times 10^{-3}\cdot 0.5=1\times 10^{-3}$. Ratio $=1\times 10^{-3}/5\times 10^{-4}=2$. Analysis: the whole warmup line scales linearly with ${\eta }_{\text{peak}}$, so every point (including step 500) doubles. This is why warmup makes the scaled LR usable: the big LR is only reached gradually, giving the noisy large-batch gradients time to settle. See Adam optimizer on why the second-moment estimate also needs that time.

Set ${\eta }_{\text{peak}}t/T_w={\eta }_{\text{peak}}/2$. Cancel ${\eta }_{\text{peak}}$ (allowed, it is nonzero): $t/T_w=1/2\Rightarrow t=T_w/2$. For $T_w=1000$: $t=500$. WHY it's clean: the warmup line is linear, so "half the height" happens at "half the width." (Contrast with cosine, where half-height happens at $x=0.5$ of the decay window, not half of total training.)

(a) $T_w=0.06\times 50000=3000$ steps. (Check $0<T_w<T$: $0<3000<50000$ ✓.) (b) Step $1500\le T_w$, so warmup branch: $3\times 10^{-4}\cdot \frac{1500}{3000}=3\times 10^{-4}\cdot 0.5=1.5\times 10^{-4}$. (c) Step $28000>T_w$, cosine branch (with ${\eta }_{\text{end}}=0$). Progress: $x=\frac{28000-3000}{50000-3000}=\frac{25000}{47000}\approx \mathrm{0.53191.}$ $\cos (\pi \cdot 0.53191)=\cos (1.67096\text{rad})\approx -0.10017$. $\eta =\frac{3\times 10^{-4}}{2}(1+(-0.10017))=1.5\times 10^{-4}\cdot 0.89983\approx 1.3497\times 10^{-4}.$ WHY these branches: step 1500 is inside the ramp (use the line); step 28000 is past it (use cosine with the shifted progress). Just past the decay midpoint, so the LR sits a touch below half of peak — consistent with the teal curve dropping through its midpoint around step ~26500.

Mechanism: the update size is $\Vert \eta g_t\Vert =\eta \Vert g_t\Vert$. With random init, $\Vert g_t\Vert$ is huge and noisy, so a fixed $\eta =5\times 10^{-3}$ produces an enormous jump $\to$ weights overflow $\to$ NaN. Warmup shrinks $\eta$ near step 0, so even a huge $\Vert g_t\Vert$ is multiplied by a tiny number, keeping the step bounded. Estimate at step 3: zero-start warmup gives $\eta (3)=5\times 10^{-3}\cdot \frac{3}{1000}=1.5\times 10^{-5}$. That is $\frac{5\times 10^{-3}}{1.5\times 10^{-5}}\approx 333$ times smaller than the constant LR — enough to keep the first few explosive steps safe. (An alternative complementary fix is Gradient clipping, which caps $\Vert g_t\Vert$ directly.)

Value continuity. Two-point warmup at $t=T_w$: ${\eta }_{\text{start}}+({\eta }_{\text{peak}}-{\eta }_{\text{start}})\cdot \frac{T_w}{T_w}={\eta }_{\text{peak}}$. Cosine (with ${\eta }_{\text{end}}=0$) at $x=0$: $\frac{{\eta }_{\text{peak}}}{2}(1+\cos 0)={\eta }_{\text{peak}}$. Both equal ${\eta }_{\text{peak}}$ ⇒ continuous for any ${\eta }_{\text{start}}$ (the start value never enters the handoff because at $t=T_w$ the warmup line has fully reached peak). Slope continuity. The warmup line has constant slope $({\eta }_{\text{peak}}-{\eta }_{\text{start}})/T_w>0$ (rising). For the cosine we need its derivative in $t$. WHY the chain rule enters: the cosine is written in terms of $x$, but we want the slope in $t$, so we must convert "rate per unit $x$" into "rate per unit $t$" by multiplying by how fast $x$ moves as $t$ moves — that conversion factor is $\frac{dx}{dt}$. Since $x=\frac{t-T_w}{T-T_w}$ is a straight line in $t$ with $T_w,T$ constant, its slope is $\frac{dx}{dt}=\frac{1}{T-T_w}$ (the numerator changes by $1$ for each $+1$ in $t$, over the fixed denominator $T-T_w$). Chain rule then gives: $\frac{d}{dt}\left[\frac{{\eta }_{\text{peak}}}{2}(1+\cos \pi x)\right]=\frac{{\eta }_{\text{peak}}}{2}\cdot (-\pi \sin \pi x)\cdot \underset{=1/(T-T_w)}{\underbrace{\frac{dx}{dt}}}=-\frac{{\eta }_{\text{peak}}\pi }{2(T-T_w)}\sin (\pi x),$ which at $x=0$ is $0$ (since $\sin 0=0$). So the cosine arrives with zero slope while the ramp leaves with positive slope ⇒ there is a corner (value-continuous but not slope-continuous). Why cosine anyway: cosine's zero-slope end was chosen so the finish (at $t=T$) is gentle and the top is flat, giving a smooth plateau exactly where the model is most sensitive. The slope corner at handoff is harmless because both sides sit at the same peak value — the LR does not jump, it merely stops rising and starts (flatly) falling. Compare with Cosine annealing for the decay-only view and Learning rate schedules for the family.

Average. The average of a straight line from $0$ to ${\eta }_{\text{peak}}$ over $[0,T_w]$ is the midpoint value: $\bar{\eta }=\frac{1}{T_w}{\int }_0^{T_w}{\eta }_{\text{peak}}\frac{t}{T_w}dt=\frac{{\eta }_{\text{peak}}}{T_w^2}\cdot \frac{T_w^2}{2}=\frac{{\eta }_{\text{peak}}}{2}.$ So on average during warmup you apply half the peak LR — a tidy way to see warmup as "spending half your LR budget while the model settles." Budget integral: ${\int }_0^{T_w}{\eta }_{\text{peak}}\frac{t}{T_w}dt=\frac{{\eta }_{\text{peak}}{T}_w}{2}=\frac{10^{-3}\cdot 1000}{2}=\mathrm{0.5.}$ WHY the integral matters: it is the total learning-rate "mass" injected during warmup; using it lets you compare warmup lengths fairly. RAdam (see RAdam) can be read as an automatic way of shaping this early budget.

(a) By construction the cosine branch reaches ${\eta }_{\text{end}}$ at $x=1$ (step $T$), so it ends at $1\times 10^{-4}$ — it decays toward the floor, not to zero. (b) Progress $x=\frac{11000-1000}{21000-1000}=0.5$, so $\cos (\pi \cdot 0.5)=0$: $\eta ={\eta }_{\text{end}}+\frac{{\eta }_{\text{peak}}-{\eta }_{\text{end}}}{2}(1+0)=10^{-4}+\frac{10^{-3}-10^{-4}}{2}=10^{-4}+\frac{9\times 10^{-4}}{2}=10^{-4}+4.5\times 10^{-4}=5.5\times 10^{-4}.$ WHY not ${\eta }_{\text{peak}}/2=5\times 10^{-4}$: the midpoint is halfway between the two endpoints ${\eta }_{\text{peak}}$ and ${\eta }_{\text{end}}$, i.e. their average $\frac{10^{-3}+10^{-4}}{2}=5.5\times 10^{-4}$ — the floor lifts the whole curve up by ${\eta }_{\text{end}}$. Only when ${\eta }_{\text{end}}=0$ does the midpoint collapse to ${\eta }_{\text{peak}}/2$.