Learning rate warmup: Visual walkthrough — AI-ML deep dive

Step 0 — The picture we are trying to draw

Before any formula, let us agree on the canvas. We are drawing a graph.

The horizontal axis is $t$ = the training step number. One step = one weight update from one batch. So $t = 0$ is the very first update, $t = 1$ the second, and so on. It only ever counts up.
The vertical axis is $η$ (the Greek letter "eta"). This is the learning rate: how big a step the optimizer takes. Look back at the update rule $θ_{t + 1} = θ_{t} - η g_{t}$ — a bigger $η$ means a bigger jump for the weights $θ$ .

Our whole job on this page is to decide: for each step $t$ , what value should $η$ have? That is a function $η (t)$ , and a function is just a curve on this canvas.

Step 1 — Two dots pin down the ramp

WHAT. We place exactly two points on the empty canvas:

Point $P_{0} = (0, η_{start})$ — "at step $0$ , be small."
Point $P_{1} = (T_{w}, η_{peak})$ — "at step $T_{w}$ , be at the peak."

WHY. The parent note gave two hard requirements: start small, end at peak. Each requirement is one dot. Two dots is exactly enough to define one straight line — no more, no less. We choose a straight line because it is the simplest continuous path connecting them; nothing in the problem demands a fancier curve.

PICTURE. Two red dots floating on the axes, nothing between them yet. This is the skeleton every warmup curve hangs on.

Step 2 — Connect the dots: the line equation, term by term

WHAT. We draw the straight line through $P_{0}$ and $P_{1}$ and write its equation.

WHY a line and not a curve? A line has constant slope — the LR grows by the same tiny amount every step. That predictable, gentle rise is exactly the "warm the engine slowly" behaviour we want. No sudden jumps.

How to build a line from two dots. Start at the left dot's height, then add "how far up we still need to go" times "how far along we are":

$η (t) = where we begin η_{start} + total rise needed (η_{peak} - η_{start}) \cdot fraction of the way there \frac{t}{T _{w}}$

Read each piece:

$η_{start}$ — the base camp. At $t = 0$ the last two factors vanish and $η = η_{start}$ . ✓
$(η_{peak} - η_{start})$ — the gap we must climb. Not the peak itself — the difference.
$\frac{t}{T _{w}}$ — the progress fraction. At $t = 0$ it is $0$ ; at $t = T_{w}$ it is $1$ . It slides from $0$ to $1$ as we march through warmup, handing us that fraction of the gap.

Check the right end: at $t = T_{w}$ , the fraction is $1$ , so $η = η_{start} + (η_{peak} - η_{start}) = η_{peak}$ . ✓ Both dots satisfied.

Step 3 — The common special case: start at zero

WHAT. Set $η_{start} = 0$ (the default in most recipes) and watch the formula collapse.

WHY. Starting from literally $0$ means the very first update does nothing — the safest possible beginning when the gradients $g_{t}$ are wild. It also makes the algebra clean, which is why libraries default to it.

Substitute $η_{start} = 0$ : $η (t) = 0 + (η_{peak} - 0) \cdot \frac{t}{T _{w}} = η_{peak} \cdot \frac{t}{T _{w}}$

Now the height is simply "peak times progress fraction." At a quarter of the way, $t / T_{w} = 0.25$ , you get a quarter of peak.

PICTURE. The same ramp, but now the line launches from the origin $(0, 0)$ — the red segment leaves the corner of the axes.

Step 4 — What happens after $T_{w}$ ? We must not stop here.

WHAT. The ramp only covers $t \leq T_{w}$ . Past that, the line would keep rising forever — nonsense. We need a second piece of the curve.

WHY. A permanently high LR never lets the model settle into a fine minimum (see the parent's mistake list). So after we reach the peak we want to come back down gently toward $0$ . We need a decay that:

starts at $η_{peak}$ exactly where warmup ended (no jump),
ends near $0$ at the final step $T$ ,
has a flat slope at both ends — flat at the top so the handoff from warmup is smooth, flat at the bottom so training finishes calmly.

PICTURE. The ramp reaches the peak, then a big question mark hovers over the region $t > T_{w}$ : which downhill curve fits all three demands?

Step 5 — Why a cosine, and how to bend it into place

WHAT. We answer the Step-4 question with a half-cosine curve. See Cosine annealing for the full treatment.

WHY cosine and not a straight line down? A straight downhill line has the same steepness everywhere, including at the very top and very bottom — that violates our "flat at both ends" wish. The cosine function $cos$ is special: it is perfectly flat (zero slope) at its peaks and troughs. That is exactly the shape we need.

Building it, factor by factor. Take the raw cosine on the interval where its argument runs from $0$ to $π$ . There $cos$ falls from $+ 1$ down to $- 1$ . We reshape it in three moves:

$η (t) = \frac{η _{peak}}{2} 1 + cos (π progress x \frac{t - T _{w}}{T - T _{w}})$

$x = \frac{t - T _{w}}{T - T _{w}}$ — a fresh progress fraction for the decay phase. It is $0$ right when warmup ends ( $t = T_{w}$ ) and $1$ at the final step ( $t = T$ ). Same trick as Step 2, new interval.
$π x$ — stretches that $0 \to 1$ progress into the angle range $0 \to π$ , exactly the stretch of cosine that falls from $+ 1$ to $- 1$ .
$1 + cos (\dots)$ — lifts the cosine up so it runs from $2$ (at the top) down to $0$ (at the bottom) instead of $+ 1$ to $- 1$ . No more negative learning rates.
$\frac{η _{peak}}{2} \cdot (\dots)$ — the half and the peak together rescale that $2 \to 0$ range into $η_{peak} \to 0$ .

Check the joins:

At $t = T_{w}$ : $x = 0$ , $cos (0) = 1$ , so $η = \frac{η _{peak}}{2} (1 + 1) = η_{peak}$ . Matches the top of the ramp — no jump. ✓
At $t = T$ : $x = 1$ , $cos (π) = - 1$ , so $η = \frac{η _{peak}}{2} (1 - 1) = 0$ . Ends at zero. ✓

Step 6 — Degenerate cases: don't let the curve break

WHAT. We poke the formula at its extreme inputs to be sure nothing explodes.

WHY. A schedule that divides by zero or goes negative can silently NaN a whole training run. We check the corners.

$T_{w} = 0$ (no warmup). The ramp $η_{peak} t / T_{w}$ would divide by zero. In practice " $T_{w} = 0$ " means skip the ramp entirely: jump straight to the peak at $t = 0$ and start cosine immediately. The curve is just the decay half.
$T_{w} = T$ (all warmup, no decay). You ramp the entire run and never come down — the parent's "forgot to decay" mistake. The cosine piece never runs.
$t > T$ (training past the planned end). The progress $x > 1$ , $cos$ turns back upward and $η$ would rise again — undesired. Real code clamps $η$ to $0$ (or holds the final value) for $t > T$ .
$η_{start} = η_{peak}$ . The gap $(η_{peak} - η_{start}) = 0$ , so the ramp is a flat horizontal line — effectively no warmup, just a constant then decay.

The one-picture summary

Here is the whole life of the learning rate on one axis: the red curve rises linearly from $η_{start}$ to $η_{peak}$ across the first $T_{w}$ steps (Steps 1–3), then cosine-decays smoothly to $0$ by step $T$ (Steps 4–5), with flat slopes stitching the two pieces together. Everything in the Adam optimizer / RAdam / large-batch story is about why this red shape beats a flat line.

Recall Feynman: the whole walkthrough in plain words

I want to decide how big a step my network should take at every moment of training. I draw an empty graph: time going right, step-size going up. I put down two dots — "tiny at the start" and "big at step $T_{w}$ " — and connect them with a straight line, because a straight line grows the step-size by the same small amount each moment. That is warmup. But if I keep climbing forever the network never settles, so after the peak I want to come back down smoothly. A straight line down is too abrupt at the ends, so I use a cosine, because a cosine is naturally flat where it turns around — flat at the top (smooth handoff) and flat at the bottom (calm finish). I shift and shrink that cosine so it slides from the peak down to zero. Finally I check the weird cases — no warmup, all warmup, running past the end — and clamp the curve so it never divides by zero or bounces back up. The result is one red curve: rise gently, cruise the peak briefly, glide down to zero.

Visual walkthrough — Learning rate warmup