Learning rate warmup: Worked examples — AI-ML deep dive

The scenario matrix

Every warmup question is really "which piece of the piecewise schedule am I in, and what are the corner cases of that piece?" Here is the full grid.

Cell	Regime	What makes it tricky	Example
A	$0 < t < T_{w}$ , zero start	plain proportional rise	Ex 1
B	$t = 0$ and $t = T_{w}$ exactly	the two endpoints (boundary)	Ex 2
C	$0 < t < T_{w}$ , non-zero start	scale the gap, not the peak	Ex 3
D	$t > T_{w}$ , cosine, midpoint	$cos (π /2) = 0$ sanity value	Ex 4
E	$t > T_{w}$ , cosine, general $x$	full substitution, any progress	Ex 5
F	$t = T_{w}$ handoff & $t = T$ end	both cosine ends, slope check	Ex 6
G	Degenerate: $T_{w} = 0$ , or $t > T$	division by zero, clamping	Ex 7
H	Real word problem (large batch)	pick $η_{peak}$ , count steps	Ex 8
I	Exam twist: solve for $t$	invert the schedule	Ex 9

We reuse these symbols throughout (all from the parent):

The one picture that holds the whole matrix: the warmup-ramp rising, peaking, then the cosine falling.

Cell A — inside warmup, zero start

Worked example Ex 1 · plain proportional rise

$η_{peak} = 1 \times 1 0^{- 3}$ , $T_{w} = 800$ , $η_{start} = 0$ . Find $η$ at step $t = 200$ .

Forecast: we're a quarter of the way up the ramp — guess a quarter of the peak before reading on.

Confirm we are in the warmup piece: is $t \leq T_{w}$ ? Yes, $200 \leq 800$ . Why this step? The schedule is piecewise; the first thing you always do is decide which branch applies. Using the cosine formula here would be wrong.
Use the zero-start line $η (t) = η_{peak} \frac{t}{T _{w}}$ . Why this step? With $η_{start} = 0$ the general two-point line collapses to this simple proportion — the LR is a fixed fraction of the peak.
Plug in: $η = 1 0^{- 3} \cdot \frac{200}{800} = 1 0^{- 3} \cdot 0.25 = 2.5 \times 1 0^{- 4}$ . Why this step? $200/800 = 0.25$ is the fraction of warmup completed; multiplying scales the peak by that fraction.

Verify: at $25%$ of the way, a straight line from $0$ to $1 0^{- 3}$ must sit at $25%$ of $1 0^{- 3}$ — exactly $2.5 \times 1 0^{- 4}$ . Units: LR is dimensionless-per-gradient, unchanged. ✓

Cell B — the two warmup endpoints

Cell C — inside warmup, non-zero start

Worked example Ex 3 · scale the gap, not the peak

$η_{start} = 1 \times 1 0^{- 5}$ , $η_{peak} = 4 \times 1 0^{- 4}$ , $T_{w} = 2000$ . Find $η$ at $t = 500$ .

Forecast: we're $25%$ up — but $25%$ of what? Not the peak. Guess before continuing.

Use the general two-point line $η (t) = η_{start} + (η_{peak} - η_{start}) \frac{t}{T _{w}}$ . Why this step? With a non-zero floor the LR does not start at 0, so the simple proportion of Ex 1 would undershoot. We must add the floor back in.
Compute the gap: $η_{peak} - η_{start} = 4 \times 1 0^{- 4} - 1 \times 1 0^{- 5} = 3.9 \times 1 0^{- 4}$ . Why this step? The line only climbs across this gap; the floor is a constant offset that's always present.
Fraction of warmup done: $500/2000 = 0.25$ . Climb: $3.9 \times 1 0^{- 4} \times 0.25 = 9.75 \times 1 0^{- 5}$ . Why this step? $25%$ of the way means $25%$ of the climb has happened.
Add the floor: $η = 1 \times 1 0^{- 5} + 9.75 \times 1 0^{- 5} = 1.075 \times 1 0^{- 4}$ . Why this step? Total LR = floor you started at + how far you've climbed.

Verify: at $t = 0$ this formula gives the floor $1 0^{- 5}$ ; at $t = 2000$ it gives $1 0^{- 5} + 3.9 \times 1 0^{- 4} = 4 \times 1 0^{- 4} = η_{peak}$ . Our intermediate value $1.075 \times 1 0^{- 4}$ sits between them. ✓

Cell D — cosine at its midpoint

Worked example Ex 4 · the

$cos (π /2) = 0$ sanity value $η_{peak} = 1 0^{- 3}$ , $T_{w} = 1000$ , $T = 11000$ . Find $η$ at $t = 6000$ .

Forecast: step 6000 is exactly halfway through the decay portion. What fraction of peak lives at the halfway point of a cosine fall?

Which branch? $t = 6000 > T_{w} = 1000$ , so we are past warmup → cosine. Why this step? Always classify the branch first.
Normalized progress through decay: $x = \frac{t - T _{w}}{T - T _{w}} = \frac{6000 - 1000}{11000 - 1000} = \frac{5000}{10000} = 0.5$ . Why this step? Cosine decay is parameterized by $x \in [0, 1]$ measuring how far into decay we are, not by raw $t$ .
$η = \frac{η _{peak}}{2} (1 + cos (π \cdot 0.5)) = \frac{1 0 ^{- 3}}{2} (1 + 0) = 5 \times 1 0^{- 4}$ . Why this step? $cos (π /2) = 0$ , so the bracket is $1$ — the half-cosine is precisely at its middle.

Verify: cosine decay goes from $η_{peak}$ (at $x = 0$ ) to $0$ (at $x = 1$ ). Halfway in progress the value is exactly $η_{peak} /2 = 5 \times 1 0^{- 4}$ . This is the single easiest cell to sanity-check — memorize it. ✓ See Cosine annealing.

Cell E — cosine at a general progress

Cell F — both cosine ends and the slope check

Worked example Ex 6 · handoff at

$t = T_{w}$ and finish at $t = T$ Same schedule. Confirm the cosine value at $t = T_{w} = 1000$ and at $t = T = 11000$ , and argue the ramp meets the cosine smoothly.

Forecast: the handoff must not jump. What value should cosine give right at $t = T_{w}$ ?

At $t = T_{w} = 1000$ : $x = \frac{1000 - 1000}{10000} = 0$ , so $η = \frac{1 0 ^{- 3}}{2} (1 + cos 0) = \frac{1 0 ^{- 3}}{2} (2) = 1 0^{- 3} = η_{peak}$ . Why this step? The warmup ramp also ends at $η_{peak}$ (Ex 2). Both branches agree at the seam → no jump. That's why the piecewise function is continuous.
At $t = T = 11000$ : $x = 1$ , $η = \frac{1 0 ^{- 3}}{2} (1 + cos π) = \frac{1 0 ^{- 3}}{2} (1 - 1) = 0$ . Why this step? Training ends at LR $0$ — the model stops moving, settling into its minimum.
Slope at the seam: the ramp's slope is $η_{peak} / T_{w} = 1 0^{- 6}$ per step; the cosine's slope at $x = 0$ is $- \frac{η _{peak} π}{2 ( T - T _{w} )} sin 0 = 0$ . Why this step? Cosine is designed with zero slope at both ends, so the LR flattens at the top before falling — a soft landing, not a corner. (There IS a tiny kink in slope: the ramp arrives with slope $+ 1 0^{- 6}$ and cosine leaves with slope $0$ , but the value is continuous, which is what matters for stability.)

Verify: value at seam from both sides $= 1 0^{- 3}$ ✓; value at end $= 0$ ✓; cosine end-slope $= 0$ ✓.

Cell G — degenerate inputs

Worked example Ex 7 ·

$T_{w} = 0$ , and querying past the end $t > T$ (a) A config sets $T_{w} = 0$ . What does $η (t) = η_{peak} \cdot t / T_{w}$ give? (b) A logger asks for $η$ at $t = 12000$ when $T = 11000$ .

Forecast: both of these are "trap" inputs a real trainer must handle without crashing.

(a) $T_{w} = 0$ makes $η_{peak} \cdot t /0$ → division by zero: undefined. Why this step? $T_{w} = 0$ means "no warmup"; the correct behaviour is to skip the warmup branch entirely and start on the main schedule at $η_{peak}$ . Real code guards with if T_w > 0.
(a) So the sensible defined value is $η (0) = η_{peak} = 1 0^{- 3}$ (jump straight to peak). Why this step? With zero warmup steps there is nothing to ramp; you are immediately "warmed up."
(b) Past the end: cosine progress $x = \frac{12000 - 1000}{10000} = 1.1 > 1$ . Plugging in gives $cos (1.1 π) = cos (198°) \approx - 0.951$ , so $η = \frac{1 0 ^{- 3}}{2} (1 - 0.951) = 2.45 \times 1 0^{- 5}$ — a negative-slope overshoot back upward, which is nonsense for a finished schedule. Why this step? The formula is only valid on $x \in [0, 1]$ ; beyond $T$ the cosine curls back up. Real code clamps: $η (t) = 0$ for $t \geq T$ .
(b) Correct answer: $η = 0$ for any $t \geq T$ . Why this step? Training is over; the LR is held at its final value.

Verify: (a) $T_{w} = 0$ → guard to $η_{peak} = 1 0^{- 3}$ ✓. (b) unclamped formula at $x = 1.1$ gives $\approx 2.45 \times 1 0^{- 5}$ (demonstrating the overshoot), clamped answer $= 0$ ✓.

Cell H — real word problem (large batch)

Worked example Ex 8 · pick

$η_{peak}$ and count steps You trained ResNet at batch $B_{0} = 256$ with a stable peak $η_{0} = 0.1$ . You scale to batch $B = 2048$ using the linear scaling rule. Your dataset is $1, 280, 000$ images and you warm up for $5$ epochs. Give the scaled peak LR and the warmup length $T_{w}$ in steps.

Forecast: batch went up $8 \times$ ; guess what happens to LR and to steps-per-epoch.

Linear scaling: $η_{peak} = η_{0} \cdot \frac{B}{B _{0}} = 0.1 \cdot \frac{2048}{256} = 0.1 \cdot 8 = 0.8$ . Why this step? Each step now averages $8 \times$ more examples, so a bigger, more reliable gradient can safely take an $8 \times$ larger step (the linear scaling rule). Warmup is exactly what makes this large $0.8$ usable early.
Steps per epoch $= \frac{images}{B} = \frac{1 , 280 , 000}{2048} = 625$ . Why this step? One optimizer step consumes one batch, so an epoch is (dataset $\div$ batch) steps. Bigger batch → fewer steps per epoch.
Warmup length: $T_{w} = 5 \times 625 = 3125$ steps. Why this step? " $5$ epochs of warmup" must be converted to steps because the schedule is defined in steps (a parent-note mistake warns about this exactly).

Verify: LR scaled by the same factor as batch: $0.8/0.1 = 8 = 2048/256$ ✓. Steps/epoch $\times$ batch $= 625 \times 2048 = 1, 280, 000 =$ dataset size ✓. $T_{w} = 3125$ ✓. Note: at old batch $256$ the same 5 epochs would be $5 \times 5000 = 25000$ steps — so changing batch silently changed warmup length, the trap from the parent note.

Cell I — exam twist: invert the schedule

Worked example Ex 9 · solve for the step

$t$ With $η_{peak} = 1 0^{- 3}$ , $T_{w} = 1000$ , $T = 11000$ : at which step during cosine decay does the LR first drop to $2.5 \times 1 0^{- 4}$ ?

Forecast: $2.5 \times 1 0^{- 4}$ is a quarter of the peak — that's below the midpoint value, so we expect $t$ past the halfway point of decay.

Set the cosine equal to the target: $\frac{1 0 ^{- 3}}{2} (1 + cos (π x)) = 2.5 \times 1 0^{- 4}$ . Why this step? "Find $t$ given $η$ " means solving the schedule backwards — invert it.
Divide out: $1 + cos (π x) = \frac{2 \cdot 2.5 \times 1 0 ^{- 4}}{1 0 ^{- 3}} = 0.5$ , so $cos (π x) = - 0.5$ . Why this step? Isolate the cosine so we can apply its inverse.
$π x = arccos (- 0.5) = \frac{2 π}{3}$ , so $x = \frac{2}{3} \approx 0.6667$ . Why this step? $arccos$ answers "which angle has this cosine?" — the tool for undoing cosine, valid because $x \in [0, 1]$ keeps $π x \in [0, π]$ where $arccos$ is single-valued.
Convert back to a step: $t = T_{w} + x (T - T_{w}) = 1000 + 0.6667 \times 10000 = 1000 + 6667 = 7667$ . Why this step? Undo the normalization $x = (t - T_{w}) / (T - T_{w})$ to recover the raw step.

Verify: plug $t = 7667$ back: $x = (7667 - 1000) /10000 = 0.6667$ , $η = \frac{1 0 ^{- 3}}{2} (1 + cos (0.6667 π)) = \frac{1 0 ^{- 3}}{2} (1 - 0.5) = 2.5 \times 1 0^{- 4}$ ✓. And $0.667 > 0.5$ , matching our forecast that it's past the midpoint. ✓

Recall One-line reflex for each cell

$t \leq T_{w}$ , zero start ::: $η = η_{peak} t / T_{w}$
$t \leq T_{w}$ , non-zero start ::: $η = η_{start} + (η_{peak} - η_{start}) t / T_{w}$
$t = T_{w}$ handoff ::: value equals $η_{peak}$ from BOTH branches (continuous)
cosine midpoint $x = 0.5$ ::: $η = η_{peak} /2$
$t \geq T$ ::: clamp to $0$ (never trust raw cosine)
$T_{w} = 0$ ::: skip warmup, start at $η_{peak}$
large batch ::: $η_{peak} = η_{0} B / B_{0}$ ; convert epochs→steps

Recall Decision flow

graph TD
  A["given step t"] --> B{"t less than or equal Tw"}
  B -->|yes| C["linear ramp"]
  B -->|no| D{"t less than T"}
  D -->|yes| E["cosine decay"]
  D -->|no| F["clamp to zero"]

Related: Adam optimizer · Gradient clipping · Weight initialization · Transformer training recipe · 3.2.07 Learning rate warmup (Hinglish)

Non-zero-start warmup value at t=500, start 1e-5, peak 4e-4, Tw 2000?

1.075 \times 1 0^{- 4}

Cosine value at progress x=0.25 with peak 1e-3?

8.5355 \times 1 0^{- 4}

Step where cosine LR hits 2.5e-4 (peak 1e-3, Tw 1000, T 11000)?

step

7667

(progress

x = 2/3

Scaled peak LR for batch 2048 from 0.1 at batch 256?

0.8

(linear scaling, factor 8).

What is

η

for any

t \geq T

0

— clamp; never use the raw cosine past

x = 1