Learning rate warmup: Question bank — AI-ML deep dive

The core object of the whole topic is the learning rate $η$ (Greek letter "eta") — the multiplier on each gradient step $θ_{t + 1} = θ_{t} - η g_{t}$ , where $g_{t}$ is the gradient (the "downhill direction") at step $t$ , and $T_{w}$ is the number of warmup steps over which $η$ climbs to its peak $η_{peak}$ . Every question below leans on those symbols; nothing new is introduced without saying so.

True or false — justify

Warmup makes the model converge to a better final loss than a good schedule without warmup.

False in general. Warmup mostly buys stability at the start; if training was already stable it may not improve the final loss at all. Its guaranteed benefit is avoiding early divergence, not a better optimum.

A permanently tiny learning rate is a safe substitute for warmup.

False. A tiny

η

is stable but trains slowly and often settles in a worse minimum because it can't move far enough late in training. Warmup gives stability early and speed later — see Learning rate schedules.

Warmup and Adam's bias correction solve the same problem, so you only need one.

False. Bias correction rescales the moving averages

m_{t}, v_{t}

toward their true expectation; it does not shrink the overall step size. Warmup shrinks

η

itself early. They are complementary — Adam optimizer needs both.

If you use warmup you no longer need a decay schedule.

False. Warmup only shapes the first

T_{w}

steps. Holding

η_{peak}

forever prevents fine convergence; warmup is almost always paired with a decay like Cosine annealing.

Warmup is only relevant for Transformers.

False. It became famous with Transformers but helps any setup with noisy early gradients: large batches, high peak LR, adaptive optimizers, very deep nets. See Transformer training recipe for why it's especially load-bearing there.

Linear warmup means the loss decreases linearly during warmup.

False. Linear refers to

η (t)

rising as a straight line; the loss curve during warmup is usually still messy and non-monotonic because the model is barely trained.

With

η_{start} = 0

, the very first optimizer step does nothing.

True. At

t = 0

η (0) = η_{peak} \cdot 0/ T_{w} = 0

, so

θ_{1} = θ_{0}

. The step exists but has zero size — harmless, and the ramp takes over immediately after.

Doubling the batch size while keeping everything else fixed leaves your warmup length unchanged in effect.

False.

T_{w}

is counted in optimizer steps; doubling batch halves steps-per-epoch, so "warm up for 1 epoch" silently halves your step count. Also, larger batches often need a larger peak LR (Linear scaling rule (large batch training)), pushing you toward longer warmup.

RAdam removes the need for manual warmup by rectifying the early variance estimate.

True (largely). RAdam mathematically switches off adaptivity until the second-moment estimate is trustworthy, reproducing warmup's effect automatically. It's the theory-driven version of the same fix.

Spot the error

"I set

T_{w} = 500

epochs so the network warms up for a long time."

The unit is wrong: warmup is measured in optimizer steps, not epochs. 500 epochs of warmup would keep

η

tiny for most of training. Intended was likely 500 steps.

"During warmup I disable gradient clipping because the small LR already keeps steps safe."

Risky. A small

η

shrinks the update, but a single huge gradient can still spike; Gradient clipping guards the gradient magnitude directly and is complementary to warmup, not redundant.

"Warmup fixes bad weight initialization, so init choice doesn't matter much."

Overreach. Warmup tolerates noisy early gradients but cannot rescue a genuinely broken init (e.g. exploding activations). Good Weight initialization and warmup address overlapping-but-different failure modes.

"My peak LR diverges, so I'll just make warmup longer."

Longer warmup delays the danger but doesn't cure a peak that's fundamentally too high — you'll still blow up when you reach it. The right fix is lowering

η_{peak}

, possibly alongside longer warmup.

"I warmed up then held

η

constant at peak; loss plateaus, so warmup failed."

The plateau is from the missing decay, not warmup. Constant peak LR bounces around the minimum; add a decay schedule and the plateau usually resolves.

"After warmup I restart the LR at zero for the cosine phase."

Error: cosine decay should start at

η_{peak}

(the value warmup just reached) and fall to

0

. Restarting at zero throws away the warmup and stalls learning. The handoff must be continuous at

t = T_{w}

Why questions

Why does warmup help adaptive optimizers specifically?

They divide

η

v_{t}

, the running second-moment estimate. With few early samples

v_{t}

is a poor estimate, so the effective step

η / v_{t}

swings wildly; a small

η

keeps that product tame until

v_{t}

settles.

Why is the variance of early gradients, not just their magnitude, the danger?

High variance means the direction

g_{t}

points very differently batch-to-batch. A large

η

amplifies those swings, so one unlucky batch can hurl the weights into a bad basin you never leave.

Why choose the half-cosine shape

\frac{1}{2} (1 + cos (π x))

for the decay after warmup?

It has zero slope at both ends: flat where it meets

η_{peak}

(smooth handoff from warmup) and flat as it reaches

0

(gentle finish). A straight-line decay would kink at the handoff.

Why a straight line for the warmup ramp specifically?

It's the simplest continuous function pinned to two points

(0, η_{start})

and

(T_{w}, η_{peak})

; there's no evidence a fancier ramp helps, so the minimal choice wins.

Why does the linear scaling rule require warmup to be usable?

Scaling batch size scales

η

up, and a large

η

from step 0 is exactly what warmup was invented to survive. Warmup is the mechanism that makes the big scaled LR safe in the "ImageNet in 1 hour" recipe.

Why can warmup let you reach a better minimum than a constant tiny LR, despite starting slow?

Late in training the loss surface is smoother, so a large

η

can jump across shallow traps toward flatter, better minima. A permanently tiny

η

gets stuck in whatever basin it lands in first.

Edge cases

What happens at exactly

t = T_{w}

under linear warmup with zero start?

η (T_{w}) = η_{peak} \cdot T_{w} / T_{w} = η_{peak}

— the ramp hits the peak precisely at the boundary, then the main schedule takes over. Continuous, no jump.

What is

η

at the midpoint of cosine decay (progress

x = 0.5

Exactly

η_{peak} /2

, because

cos (π /2) = 0

gives

\frac{η _{peak}}{2} (1 + 0)

. A handy sanity check for any implementation.

What if

T_{w} = 0

(warmup switched off)?

The linear formula

η_{peak} t / T_{w}

divides by zero. In practice

T_{w} = 0

is a special-case meaning "no warmup" — jump straight to

η_{peak}

at step 0, the very behaviour warmup was meant to avoid.

What if

T_{w} \geq T

(warmup longer than total training)?

You never reach

η_{peak}

and never decay — training ends mid-ramp at some

η < η_{peak}

. Effectively a slow linear LR increase with no fine-tuning phase; almost never what you want.

What does

η_{start} = η_{peak}

produce?

A flat line — no ramp at all, just constant

η_{peak}

during "warmup." It's warmup in name only and offers none of the early stability.

Under warmup + cosine, what is

η

at the final step

t = T

Progress

x = (T - T_{w}) / (T - T_{w}) = 1

, and

cos (π) = - 1

, so

η = \frac{η _{peak}}{2} (1 - 1) = 0

— training finishes with a vanishing step, as intended.

Recall One-line summary of every trap above

Warmup buys early stability, not a magic better optimum; it's measured in steps, always paired with decay, complements (never replaces) bias correction, clipping, and good init, and its formulas behave cleanly at every boundary — $η = 0$ at start, $η_{peak}$ at $T_{w}$ , and $0$ at $T$ .