Learning rate scheduling: Exercises — AI-ML deep dive

This page is your self-test for Learning Rate Scheduling. Every problem states its difficulty level (L1 → L5). Try each one first, then open the collapsible solution. After every level there is one trap callout — the exact mistake most people make there.

Notation reminder (every symbol used on this page, defined here so you never need the parent open):

$η$ = learning rate, the size of each parameter step.
$η_{0}$ = starting learning rate; $η (t)$ = the learning rate at training step or epoch $t$ .
$η_{m a x}$ = peak learning rate a schedule reaches; $η_{m i n}$ = floor value a schedule decays down to.
$γ$ = step-decay multiplier, the factor ( $0 < γ < 1$ ) by which $η$ is multiplied at each drop.
$s$ = drop interval for step decay: the number of epochs between drops.
$λ$ = decay rate for exponential ( $e^{- λ t}$ ) and inverse ( $1/ (1 + λ t)$ ) schedules; bigger $λ$ = faster decay.
$T$ = period (total length) of a cosine cycle; $T_{w}$ = warmup length in steps.
Linear warmup schedule: for the first $T_{w}$ steps the learning rate ramps up straight from $0$ to $η_{m a x}$ : $η (t) = η_{m a x} \cdot \frac{t}{T _{w}}$ for $t \leq T_{w}$ . Why: at step 0 the weights are random and Adam's variance estimate is unreliable, so tiny early steps prevent blow-up.
$⌊ x ⌋$ = the floor: the biggest whole number $\leq x$ (e.g. $⌊ 2.7 ⌋ = 2$ ).
$V$ = the steady-state variance of the parameter around a minimum (how much it "buzzes"); $σ^{2}$ = gradient-noise variance; $a$ = curvature of the local quadratic loss.

Prerequisite topics you may want open: Stochastic Gradient Descent, Adam and Adaptive Optimizers, Robbins-Monro Stochastic Approximation, Warm Restarts (SGDR).

Level 1 — Recognition

Recall Solution 1.1

(a) Step decay → 3 (the floor $⌊ t / s ⌋$ counts how many drops happened).
(b) Exponential decay → 1 (smooth $e^{- λ t}$ curve).
(c) $1/ t$ decay → 4 (denominator grows linearly in $t$ ).
(d) Cosine annealing → 2 (the $cos$ term).

Why the floor gives a staircase: $⌊ t / s ⌋$ jumps up by 1 only when $t$ crosses a multiple of $s$ , so $η$ is flat between drops and then multiplies by $γ$ — a staircase.

Figure s01 — the step-decay staircase. Below, the red curve is $η (t) = η_{0} γ^{⌊ t / s ⌋}$ with $η_{0} = 0.2$ , $γ = 0.5$ , $s = 20$ . Notice three things: (1) between the dotted vertical lines the red curve is perfectly flat — $η$ does not change mid-block; (2) at each multiple of $s = 20$ epochs it drops by a factor $γ = 0.5$ (each arrow marks a halving); (3) the drops get smaller in absolute size because they multiply an ever-smaller number. This is the "train at a plateau, then suddenly drop to a lower basin" behaviour.

Recall Solution 1.2

At $t = 0$ : $cos 0 = 1$ , so $η = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + 1) = η_{m i n} + (η_{m a x} - η_{m i n}) = η_{m a x}$ .
At $t = T$ : $cos π = - 1$ , so $η = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 - 1) = η_{m i n}$ .

Why memorise the endpoints: they are the two anchor points — everything else is a smooth ride between them.

Recall Solution 1.3

The inverse / $1/ t$ decay, $η (t) = η_{0} / (1 + λ t)$ . Why: it behaves like $η_{0} / t$ for large $t$ ; the harmonic series $\sum 1/ t = \infty$ (you can still travel arbitrarily far) while $\sum 1/ t^{2} < \infty$ (the noise dies out).

Level 2 — Application

Recall Solution 2.1

WHAT: count drops. $⌊ 45/20 ⌋ = ⌊ 2.25 ⌋ = 2$ .
WHY: two full blocks of 20 epochs have passed (epochs 0–19, 20–39), and we are in the third block (40–59) which has had 2 drops applied.
$η = 0.2 \cdot 0. 1^{2} = 0.2 \cdot 0.01 = 0.002$ .

Recall Solution 2.2

WHY use the half-life: after each 10 epochs $η$ halves. $30$ epochs $= 3$ half-lives.
$η = 0.05 \cdot e^{- λ \cdot 30} = 0.05 \cdot e^{- 3 l n 2} = 0.05 \cdot 2^{- 3} = 0.05 \cdot \frac{1}{8} = 0.00625$ .
Check algebraically: $e^{- λ t} = (e^{- λ \cdot 10})^{3} = (1/2)^{3}$ . ✔

Recall Solution 2.3

Since $750 < 3000$ we are inside warmup, so we use the linear warmup formula $η = η_{m a x} \cdot \frac{t}{T _{w}}$ .
$η = 6 \times 1 0^{- 4} \cdot \frac{750}{3000} = 6 \times 1 0^{- 4} \cdot 0.25 = 1.5 \times 1 0^{- 4}$ .
Why linear up: early on Adam's second-moment estimate is tiny, so effective steps can blow up; ramping keeps them small until statistics settle.

Recall Solution 2.4

Halfway: $cos (π \cdot 200/400) = cos (π /2) = 0$ .
$η = 0.002 + \frac{1}{2} (0.02 - 0.002) (1 + 0) = 0.002 + \frac{1}{2} (0.018) = 0.002 + 0.009 = 0.011$ .
This is exactly the average $(η_{m a x} + η_{m i n}) /2 = 0.011$ — but only because $t = T /2$ where $cos = 0$ .

Level 3 — Analysis

Recall Solution 3.1

$V = \frac{η σ ^{2}}{2 a} = \frac{0.02 \cdot 2}{2 \cdot 4} = \frac{0.04}{8} = 0.005$ .
Halve $η$ to $0.01$ : $V = \frac{0.01 \cdot 2}{8} = 0.0025$ — exactly half.
WHY it halves cleanly: in the small- $η$ approximation $V \propto η$ linearly, so scaling $η$ by a factor scales $V$ by the same factor. See Robbins-Monro Stochastic Approximation for the full derivation the parent note gives.

Recall Solution 3.2

$η a = 0.02 \cdot 4 = 0.08$ , so $2 - η a = 1.92$ .
$V = \frac{0.02 \cdot 2}{4 \cdot 1.92} = \frac{0.04}{7.68} = 0.005208 \dots$ .
The exact value is slightly larger than the approximation $0.005$ .
WHY: the approximation drops the $- η a$ term, replacing $2 - η a = 1.92$ by $2$ . Dividing by a bigger number ( $2$ ) gives a smaller $V$ . So the approximation always underestimates $V$ (for $0 < η a < 2$ ). The gap grows as $η a$ grows.

Recall Solution 3.3

The formula expects a drop every 30 epochs $= 30 \times 500 = 15000$ steps.
Feeding raw steps means it drops every 30 steps instead.
Ratio $= 15000/30 = 500 \times$ faster decay.
WHY this matters: after just 30 steps $η$ has already halved; after 300 steps it is $0. 5^{10} \approx 0.001$ of $η_{0}$ — training effectively freezes almost immediately. Always confirm the time unit $t$ .

Level 4 — Synthesis

Recall Solution 4.1

(a) Step 1000 — inside warmup ( $1000 < 2000$ ): $η = η_{m a x} \cdot \frac{t}{T _{w}} = 1 \times 1 0^{- 3} \cdot \frac{1000}{2000} = 5 \times 1 0^{- 4} .$

(b) Step 7000 — past warmup, so we are in the cosine phase. The cosine "clock" must be reset to start at the end of warmup. Define local time $τ = t - T_{w}$ and a local cosine horizon $T_{c o s} = T_{total} - T_{w}$ . (Here $T_{c o s}$ plays the role of the period $T$ from the notation reminder, but measured only over the post-warmup portion — we rename it to avoid confusing it with the total run length.) So $T_{c o s} = 12000 - 2000 = 10000$ .

$τ = 7000 - 2000 = 5000$ . That is exactly halfway ( $T_{c o s} /2 = 5000$ ).
$cos (π τ / T_{c o s}) = cos (π \cdot 5000/10000) = cos (π /2) = 0$ .
$η = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + 0) = 0 + \frac{1}{2} (1 \times 1 0^{- 3}) (1) = 5 \times 1 0^{- 4}$ .

WHY reset the cosine clock: if you used raw $t$ in $cos (π t / T_{total})$ , then at handoff (step 2000) the LR would jump down discontinuously instead of starting smoothly at $η_{m a x}$ . Shifting by $T_{w}$ makes the two pieces meet cleanly at $η_{m a x}$ . Look at Figure s02 — the red curve is continuous at the join.

Recall Solution 4.2

Linear scaling rule: scale $η$ by the same factor as the batch size. New $η = 1 \times 1 0^{- 3} \cdot 4 = 4 \times 1 0^{- 3}$ .
WHY: a $4 \times$ bigger batch averages over $4 \times$ more samples, so the gradient noise variance drops by $\approx 4 \times$ ; you can afford a $4 \times$ larger step to keep the same effective progress per example. See Batch Size and Learning Rate scaling.
Add: a warmup. The now-larger $4 \times 1 0^{- 3}$ can destabilise early training, so ramp up over the first few thousand steps before applying it fully.

Level 5 — Mastery

Recall Solution 5.1

A note on indexing: all sums below run over $t = 1, 2, 3, \dots$ (i.e. from $t = 1$ to $\infty$ ), matching the harmonic series (which starts at $t = 1$ since $1/0$ is undefined).

Condition 1 ( $\sum_{t} η (t) = \infty$ , "can travel arbitrarily far"):

For $η_{0} / t$ : $\sum_{t = 1}^{\infty} \frac{η _{0}}{t} = η_{0} \sum_{t = 1}^{\infty} \frac{1}{t}$ — the harmonic series, which diverges to $\infty$ . ✔
For $η_{0} r^{t}$ : $\sum_{t = 1}^{\infty} η_{0} r^{t} = η_{0} \cdot \frac{r}{1 - r}$ — a finite geometric sum (since $0 < r < 1$ ). ✘ The geometric schedule fails Condition 1: its total step budget is bounded.

Condition 2 ( $\sum_{t} η (t)^{2} < \infty$ , "noise dies out"):

For $η_{0} / t$ : $\sum_{t = 1}^{\infty} \frac{η _{0}^{2}}{t ^{2}} = η_{0}^{2} \sum_{t = 1}^{\infty} \frac{1}{t ^{2}}$ — a $p$ -series with $p = 2 > 1$ , which converges (to $η_{0}^{2} π^{2} /6$ ). ✔ So $1/ t$ decay satisfies both conditions.
(The geometric schedule is already disqualified by Condition 1, so its Condition-2 status is moot — though for completeness $\sum r^{2 t}$ also converges, that cannot rescue it.)

Conclusion: $1/ t$ decay passes both Robbins–Monro conditions; the geometric schedule fails the first.

Practical consequence (one sentence): because a geometric/exponential schedule can only move the parameters a bounded total distance, if the minimum lies farther than that finite budget the training stalls short of it, whereas $1/ t$ decay can always reach any distance while still driving the residual noise to zero.

Recall Solution 5.2 (sample design + justification)

Design: linear warmup over $T_{w} = 1000$ steps ( $10%$ of budget) to a peak $η_{m a x} = 1 \times 1 0^{- 3}$ , then cosine anneal to $η_{m i n} = 0$ over the remaining $9000$ steps.

Warmup length $10%$ : long enough for Adam's second-moment estimate $\overset{v}{^}$ to stabilise, short enough not to waste budget. Why warmup at all: at step 0, $\overset{v}{^} \approx 0$ so effective step $η / \overset{v}{^}$ can explode.
Cosine, not step: smooth, spends most time high (fast travel) then flattens near the end (gentle settling) — see the parent note's endpoint check.
$η_{m i n} = 0$ : we have a fixed budget and want the tightest final fit; the noise-floor result $V \propto η$ says driving $η \to 0$ minimises residual jitter.

Values (local cosine time $τ = t - T_{w}$ , horizon $T_{c o s} = T_{total} - T_{w} = 9000$ ):

Step 500 (warmup, since $500 < 1000$ ): $η = η_{m a x} \cdot \frac{500}{1000} = 1 \times 1 0^{- 3} \cdot 0.5 = 5 \times 1 0^{- 4}$ .
Step 5500 (cosine, $τ = 5500 - 1000 = 4500 = T_{c o s} /2$ ): $cos (π \cdot 4500/9000) = cos (π /2) = 0$ , so $η = \frac{1}{2} (1 \times 1 0^{- 3}) (1 + 0) = 5 \times 1 0^{- 4}$ .
Step 10000 (cosine end, $τ = 10000 - 1000 = 9000 = T_{c o s}$ ): $cos (π \cdot 9000/9000) = cos π = - 1$ , so $η = \frac{1}{2} (1 \times 1 0^{- 3}) (1 - 1) = 0$ .

Sanity: the schedule starts at $0$ , climbs to $η_{m a x}$ at step 1000, and decays smoothly to $0$ at step 10000 — continuous at every seam.

Recall Solution 5.3

Cycle lengths: $T_{0} = 100$ , $T_{1} = 200$ , $T_{2} = 400$ , $T_{3} = 800$ .

Restart 1 (start of cycle 1) at step $T_{0} = 100$ .
Restart 2 (start of cycle 2) at step $T_{0} + T_{1} = 100 + 200 = 300$ .
Restart 3 (start of cycle 3) at step $T_{0} + T_{1} + T_{2} = 100 + 200 + 400 = 700$ .
WHY restarts help: each jump back to $η_{m a x}$ lets the ball hop out of a narrow basin and possibly find a wider, flatter minimum that generalises better. See Warm Restarts (SGDR).