Question bank — Learning rate scheduling

False — the steady-state variance $V\propto \eta$ is fixed by $\eta$, not by time; more steps just keep jittering inside the same band. Only $\eta \to 0$ shrinks the band to zero.

False — warmup ramps $\eta$ up for the first $T_w$ steps, and warm restarts jump it back up periodically. Decay is the typical late-phase behaviour, not a universal rule. See Warm Restarts (SGDR).

False — cosine is flat near $t=0$ and $t=T$ (its slope $\to 0$ there) and drops fastest in the middle; only the middle looks straight. That is exactly why it settles gently at the end.

False — $\sum \eta (t)=\infty$ is about being able to travel arbitrarily far to reach the minimum; only $\sum \eta (t{)}^2<\infty$ is about the noise dying out. You need both, and they pull in opposite directions.

False — $\sum 1/\sqrt{t}=\infty$ ✓ but $\sum (1/\sqrt{t}{)}^2=\sum 1/t=\infty$ ✗, so the squared-sum condition fails; the noise never provably dies. The exact schedule is ${\eta }_0/(1+\lambda t)$, the $1/t$ form.

False — warmup protects the unstable early phase (random weights, tiny/unreliable Adam statistics); decay handles the late fine-tuning phase. They act at opposite ends of training, so modern schedules use both.

False — it travels faster early but leaves a larger noise floor $V\propto \eta$, so final loss is worse; you trade speed for a worse plateau. The point of scheduling is to get speed early and precision late.

True (for small $\eta$) — since $V\approx \eta {\sigma }^2/(2a)$ is linear in $\eta$, halving $\eta$ halves $V$. This is the numerical backbone of "smaller LR = smaller settling jitter."

The error: at step 0 Adam's second-moment estimate $\hat{v}\approx 0$, so effective steps $\eta /\sqrt{\hat{v}}$ blow up and the loss diverges. Decay theory assumes stable gradients; you need warmup first. See Adam and Adaptive Optimizers.

The error: mixing units. If an epoch is 500 steps, "$s=30$ steps" makes $\eta$ collapse ~500× too fast. Always confirm whether $t$ (and hence $s$) is in steps or epochs.

The error framing: the near-zero LR at the end is deliberate fine-tuning — it lets the parameters settle into the bottom of the basin rather than jitter. The last epochs polish the minimum, they don't waste time. See Loss Landscapes and Minima.

The error: larger batches have less gradient noise (smaller ${\sigma }^2$), so the effective good learning rate scales up (roughly linearly). Keeping the old $\eta$ under-utilizes the cleaner gradient. See Batch Size and Learning Rate scaling.

The error: warm restarts don't violate convergence — each restart is a fresh decay phase whose purpose is to jump the ball out of one basin and let it settle into a possibly better one. It's an exploration strategy layered on top of decay, not a contradiction of it.

The error: it's the theoretically sufficient schedule, but it decays too aggressively early — LR shrinks before the model has traveled far, so training stalls. Practice prefers gentler shapes (cosine, step) that keep $\eta$ high longer.

Because $V$ is a steady state of $V^{\prime }=(1-\eta a{)}^{2}V+{\eta }^2{\sigma }^2$: each step's decay of old jitter, $(1-\eta a{)}^{2}V$, exactly balances new injected noise ${\eta }^2{\sigma }^2$, fixing $V$ regardless of step count. Time can't help once you're at equilibrium.

Its flat-high start lets the model travel fast while the landscape is still far from the minimum, and its flat-low end gives gentle settling — a linear ramp does neither well, cutting speed early and slamming the LR down abruptly late.

Exponential decay multiplies by a fixed factor every step continuously, so like radioactive decay it has a constant half-life set by the decay rate $\lambda$. Step decay holds $\eta$ flat then drops by $\gamma$ every $s$ epochs, so there's no smooth constant-ratio halving time.

Training at a fixed LR lets SGD explore a basin until progress stalls; the abrupt $\times \gamma$ drop then quenches the jitter and lets the model fall into a lower sub-basin it was previously bouncing over. See Stochastic Gradient Descent.

Transformers rely on Adam, whose early second-moment estimate $\hat{v}$ is unreliable and makes effective steps explode; plain SGD has no such adaptive denominator, so its early steps are already bounded and warmup matters less.

$\sum \eta =\infty$ wants LRs large/persistent enough to travel forever, while $\sum {\eta }^2<\infty$ wants them shrinking fast enough that squared steps sum finitely — the schedule must decay slowly enough for the first yet fast enough for the second. See Robbins-Monro Stochastic Approximation.

At $t=0$, $\cos 0=1$ gives $\eta ={\eta }_{\max }$; at $t=T$, $\cos \pi =-1$ gives $\eta ={\eta }_{\min }$. The endpoints are exact plateaus where the slope is zero, so $\eta$ barely moves right at the boundaries.

With ${\eta }_{\min }=0$ the model fully stops updating at $t=T$ (pure fine-tune-then-freeze); a small positive ${\eta }_{\min }$ keeps a floor of exploration, useful when you plan warm restarts or continued training.

It degenerates into a constant learning rate — every "drop" multiplies by 1, so nothing changes. You're back to the constant-$\eta$ noise floor with no scheduling benefit.

If designed correctly, both pieces equal ${\eta }_{\max }$ at $t=T_w$, so the curve is continuous — a mismatch there causes a visible LR jump that can destabilize training right at the transition.

There is no ramp — you start at full ${\eta }_{\max }$ immediately, reintroducing the exact early-instability (exploding Adam steps) that warmup was meant to prevent. It's equivalent to "no warmup."

It approaches zero but never equals it — ${\eta }_0/(1+\lambda t)\to 0^{+}$. That's exactly what Robbins–Monro wants: infinitely small yet with divergent sum, so the model can still creep the last distance to the minimum.

The factor $(1-\eta a)$ flips sign, so each step overshoots to the other side of the minimum; if $|1-\eta a|>1$ the oscillations grow and training diverges. This is the "ball bouncing across the valley, never settling" picture made precise. See Hyperparameter Tuning.