Learning rate scheduling: Worked examples — AI-ML deep dive

This page is the exhaustive worked-example companion to the parent note. There, we derived why the learning rate $η$ must shrink. Here, we hit every kind of question a schedule can throw at you — every phase, every degenerate input, every limiting value — one fully worked example per case.

Prerequisites we lean on: Stochastic Gradient Descent, Adam and Adaptive Optimizers, Robbins-Monro Stochastic Approximation, Loss Landscapes and Minima.

The scenario matrix

Every schedule question falls into one of these case classes. The whole point of this page is to leave no cell untouched.

Cell	Case class	What makes it tricky	Covered by
A	Discrete drop, mid-interval	floor function jumps	Ex 1
B	Discrete drop, exact boundary	is $t = s$ before or after the drop?	Ex 2
C	Smooth decay, endpoints ( $t = 0$ , $t = T$ )	must return $η_{m a x}$ , $η_{m i n}$ exactly	Ex 3
D	Smooth decay, non-symmetric interior point	the "not linear" trap	Ex 4
E	Warmup phase ( $t \leq T_{w}$ )	ramp up, not down	Ex 5
F	Warmup → decay handoff (the join $t = T_{w}$ )	continuity: two formulas must agree	Ex 6
G	Degenerate / zero inputs ( $η_{m i n} = 0$ , $t = 0$ , $γ = 1$ )	limits and "no-op" schedules	Ex 7
H	Limiting behaviour ( $t \to \infty$ )	does $\sum η$ diverge? does $η \to 0$ ?	Ex 8
I	Real-world word problem (units: epoch vs step)	the unit-conversion trap	Ex 9
J	Exam twist (solve for $t$ , not for $η$ )	invert the schedule	Ex 10

We use these symbols throughout — all defined in the parent, restated here so nothing is assumed:

The full schedule we return to again and again:

That figure shows the warmup + cosine shape. Follow the coral warmup line rising, the join at $t = T_{w}$ , then the lavender cosine curve settling. Every example below lives somewhere on a curve like this.

Cell A — discrete drop, mid-interval

Cell B — discrete drop, exact boundary

Cell C — smooth decay, endpoints

Worked example Ex 3 · Cosine annealing at both ends

$η_{m a x} = 0.01$ , $η_{m i n} = 0$ , $T = 1000$ . Find $η$ at $t = 0$ and at $t = T = 1000$ .

Forecast: an ideal schedule should start hot and end cold. Predict the two endpoint values before computing.

Recall the cosine formula (parent note): $η (t) = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + cos \frac{π t}{T})$ .

At $t = 0$ : $cos (0) = 1$ , so $η = 0 + \frac{1}{2} (0.01) (1 + 1) = \frac{1}{2} (0.01) (2) = 0.01$ . Why this step? Plugging $t = 0$ makes the cosine argument $0$ ; $cos 0 = 1$ pushes the bracket to its maximum value $2$ , recovering $η_{m a x}$ .
At $t = T = 1000$ : $cos (π) = - 1$ , so $η = 0 + \frac{1}{2} (0.01) (1 - 1) = 0$ . Why this step? At the end the argument is $π$ ; $cos π = - 1$ collapses the bracket to $0$ , recovering $η_{m i n}$ .

Verify: the schedule spans exactly $[η_{m i n}, η_{m a x}] = [0, 0.01]$ as $t$ goes $0 \to T$ . Both endpoints land on their intended targets. ✔ Answers: $0.01$ and $0$ .

Cell D — smooth decay, non-symmetric interior

Worked example Ex 4 · Cosine at a quarter (the "not linear" trap)

$η_{m a x} = 0.1$ , $η_{m i n} = 0$ , $T = 100$ . Find $η$ at $t = 25$ . Is it above or below $0.05$ ?

Forecast: a linear schedule would give $0.075$ at the quarter point (dropped 25% of the way). A midpoint-guess person says $0.05$ . Cosine says…?

Compute the argument: $\frac{π t}{T} = \frac{π \cdot 25}{100} = \frac{π}{4}$ . Why this step? The cosine's input is the fraction of the journey $t / T = 0.25$ times $π$ .
$cos (π /4) = \frac{2}{2} \approx 0.7071$ .
$η = 0 + \frac{1}{2} (0.1) (1 + 0.7071) = 0.05 \times 1.7071 = 0.08536$ . Why this step? Because $cos$ is still high early on, the bracket is well above $1$ , keeping $η$ high.

Verify: $0.08536 > 0.075$ (linear) $> 0.05$ (midpoint guess). Cosine stays high early, exactly as its flat start promises. Look at the landscape: this keeps the ball travelling fast through the early basin. ✔ Answer: $0.0854$ (above $0.05$ ).

The figure overlays the cosine curve (lavender) against the linear guess (dashed slate). At $t = 25$ the vertical gap is the mistake you'd make by assuming linearity.

Cell E — warmup phase

Worked example Ex 5 · Inside warmup

Transformer: $η_{m a x} = 5 \times 1 0^{- 4}$ , warmup $T_{w} = 4000$ steps (linear). Find $η$ at step $t = 1000$ .

Forecast: we are one-quarter of the way up the warmup ramp. Guess before computing.

Confirm the phase: $1000 < 4000$ , so we are in warmup, using $η (t) = η_{m a x} \cdot \frac{t}{T _{w}}$ . Why this step? Warmup and decay use different formulas; you must first decide which regime $t$ lives in.
$η = 5 \times 1 0^{- 4} \cdot \frac{1000}{4000} = 5 \times 1 0^{- 4} \times 0.25 = 1.25 \times 1 0^{- 4}$ . Why this step? The ramp is a straight line from $0$ at $t = 0$ to $η_{m a x}$ at $t = T_{w}$ ; at a quarter of the way, we're at a quarter of the peak.

Verify: $0 < 1.25 \times 1 0^{- 4} < η_{m a x} = 5 \times 1 0^{- 4}$ . ✔ Why warmup up, not down? At step 0 Adam's second-moment estimate $\overset{v}{^} \approx 0$ , so effective steps $η / \overset{v}{^}$ could explode; a small $η$ keeps them tame. Answer: $1.25 \times 1 0^{- 4}$ .

Cell F — warmup → decay handoff

Worked example Ex 6 · The join must be continuous

Warmup for $T_{w} = 4000$ up to $η_{m a x} = 5 \times 1 0^{- 4}$ , then $1/ t$ -style decay $η (t) = η_{m a x} T_{w} / t$ afterwards. Evaluate at $t = 4000$ from both formulas. Do they agree?

Forecast: if the schedule has a jump at the join, training gets a jolt. A well-built schedule is continuous. Predict whether the two sides match.

Warmup formula at $t = T_{w}$ : $η = η_{m a x} \cdot \frac{4000}{4000} = η_{m a x} = 5 \times 1 0^{- 4}$ . Why this step? At the top of the ramp the fraction $t / T_{w} = 1$ , giving the full peak.
Decay formula at $t = T_{w}$ : $η = η_{m a x} 4000/4000 = η_{m a x} 1 = 5 \times 1 0^{- 4}$ . Why this step? The decay is deliberately written so $T_{w} / t = 1$ exactly at the handoff.

Verify: both give $5 \times 1 0^{- 4}$ — the schedule is continuous at $t = T_{w}$ ; no jump. This is the famous Transformer schedule (warmup then inverse-square-root). ✔ Answer: $5 \times 1 0^{- 4}$ from both sides.

The figure zooms on the join: coral warmup and mint decay meet at one point — no discontinuity. See Warm Restarts (SGDR) for schedules that deliberately jump instead.

Cell G — degenerate / zero inputs

Worked example Ex 7 · Three degenerate schedules

Evaluate each edge case. (a) Step decay with $γ = 1$ : what is $η (t)$ for all $t$ ? (b) Cosine with $η_{m a x} = η_{m i n} = 0.003$ : what is $η (t)$ ? (c) Warmup at exactly $t = 0$ : what is $η (0)$ ?

Forecast: each of these "breaks" a knob. Guess which ones flatten into a constant schedule.

(a) $η (t) = η_{0} \cdot 1^{⌊ t / s ⌋} = η_{0} \cdot 1 = η_{0}$ for every $t$ . Why? $γ = 1$ means "drop by a factor of 1" = don't drop at all. Step decay degenerates to a constant schedule — and by the parent's noise-floor result, leaves residual variance $V \propto η_{0} > 0$ .
(b) $η = η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + cos \dots) = 0.003 + \frac{1}{2} (0) (\dots) = 0.003$ . Why? When max = min, the amplitude $(η_{m a x} - η_{m i n}) = 0$ kills the cosine term. Flat line at $0.003$ .
(c) $η (0) = η_{m a x} \cdot \frac{0}{T _{w}} = 0$ . Why? The warmup ramp genuinely starts from zero — the very first step takes essentially no step. That's the whole point.

Verify: (a) and (b) both collapse to constants ( $η_{0}$ and $0.003$ ); (c) gives $0$ . All three are consistent with the formulas at their degenerate inputs. ✔ Answers: $η_{0}$ , $0.003$ , $0$ .

Cell H — limiting behaviour ( $t \to \infty$ )

Worked example Ex 8 · Does

$1/ t$ decay satisfy Robbins–Monro? Schedule $η (t) = \frac{η _{0}}{1 + λ t}$ with $η_{0} = 0.1$ , $λ = 0.01$ . (i) What is $lim_{t \to \infty} η (t)$ ? (ii) Does $\sum_{t} η (t)$ diverge? (iii) Does $\sum_{t} η (t)^{2}$ converge?

Forecast: the Robbins–Monro conditions demand $\sum η = \infty$ and $\sum η^{2} < \infty$ . Predict whether this schedule passes both before computing.

(i) As $t \to \infty$ , the denominator $1 + λ t \to \infty$ , so $η (t) \to 0$ . Why this step? We must send $η \to 0$ to kill the noise floor $V \propto η$ (parent derivation). This schedule does.
(ii) For large $t$ , $η (t) \approx \frac{η _{0}}{λ t} \propto \frac{1}{t}$ . The harmonic series $\sum 1/ t$ diverges. Why this step? Divergence means the total distance travellable is infinite — the optimizer can reach any minimum, however far. ✔
(iii) $η (t)^{2} \approx \frac{( η _{0} / λ ) ^{2}}{t ^{2}} \propto \frac{1}{t ^{2}}$ . The series $\sum 1/ t^{2} = π^{2} /6$ converges. Why this step? Convergence means accumulated noise is finite — the jitter dies out. ✔

Verify: $η \to 0$ ✔, $\sum η = \infty$ ✔, $\sum η^{2} < \infty$ ✔. This is exactly why $1/ t$ is the classical Robbins–Monro schedule. (By contrast, constant $η$ fails: $\sum η^{2} = \infty$ .) $passes both$ .

Cell I — real-world word problem (unit trap)

Worked example Ex 9 · Epochs vs steps

A dataset has $50, 000$ images, batch size $250$ , so $200$ mini-batch steps per epoch. You want step decay that halves $η$ every $30$ epochs ( $η_{0} = 0.1$ ). Your framework schedules in steps, not epochs. What $s_{steps}$ do you pass? And what is $η$ at step $t = 12000$ ?

Forecast: if you naively pass $s = 30$ (thinking steps), you'll decay $200 \times$ too fast. Predict the correct $s$ and where step 12000 lands.

Convert the drop period: $s_{steps} = 30 epochs \times 200 \frac{steps}{epoch} = 6000$ steps. Why this step? The framework counts $t$ in steps, so the drop interval must be expressed in the same unit. Mixing units is the classic bug the parent warns about.
Step 12000 in epochs: $12000/200 = 60$ epochs — the second drop.
Drops so far: $⌊ 12000/6000 ⌋ = 2$ , so $η = 0.1 \times 0. 5^{2} = 0.025$ . Why this step? Same floor logic as Ex 1, now in step-units.

Verify: cross-check in epoch-units: $⌊ 60/30 ⌋ = 2 \Rightarrow 0.1 \times 0.25 = 0.025$ . Both unit systems agree because we converted correctly. Had we wrongly used $s = 30$ steps, we'd get $⌊ 12000/30 ⌋ = 400$ drops → $η = 0.1 \times 0. 5^{400} \approx 0$ — absurd. ✔ Answers: $s_{steps} = 6000$ , $η = 0.025$ . See Batch Size and Learning Rate scaling for how batch size also affects $η_{0}$ itself.

Cell J — exam twist (invert the schedule)

Worked example Ex 10 · Solve

for the step, not for $η$ Exponential decay $η (t) = η_{0} e^{- λ t}$ with $η_{0} = 0.1$ , $λ = 0.002$ . At which step has $η$ fallen to one-tenth of its start, $η = 0.01$ ?

Forecast: this is the half-life idea generalized to a "tenth-life". Guess the order of magnitude of $t$ (hundreds? thousands?).

Set up the equation: $0.01 = 0.1 e^{- 0.002 t}$ . Why this step? We're told $η$ and asked for $t$ — invert the schedule.
Divide by $0.1$ : $e^{- 0.002 t} = 0.1$ . Why this step? Isolate the exponential.
Take natural log (the tool that undoes $e^{\cdot}$ ): $- 0.002 t = ln (0.1) = - ln 10 \approx - 2.3026$ . Why this tool? $ln$ is precisely the inverse of $e^{x}$ ; it's the only clean way to bring $t$ down from the exponent.
Solve: $t = \frac{2.3026}{0.002} = 1151.3$ , round up to $t = 1152$ . Why round up? Steps are integers; at $t = 1151$ we're a hair above $0.01$ , at $t = 1152$ just below.

Verify: plug back $η (1151.3) = 0.1 e^{- 0.002 \times 1151.3} = 0.1 e^{- 2.3026} = 0.1 \times 0.1 = 0.01$ ✔. (Half-life sanity: $t_{1/2} = ln 2/ λ = 0.6931/0.002 \approx 347$ steps; a "tenth-life" is $lo g_{2} 10 \approx 3.32$ half-lives $\approx 1152$ steps ✔.) Answer: $t \approx 1151$ (step $1152$ first below).

Recall

Recall Which cells collapse to a constant schedule, and why?

Step decay with $γ = 1$ (Ex 7a) and cosine with $η_{m a x} = η_{m i n}$ (Ex 7b). ::: Both zero out the "amount of change" — $γ^{k} = 1$ for all $k$ , and amplitude $(η_{m a x} - η_{m i n}) = 0$ — so $η$ never moves. By the parent's noise-floor result these leave $V \propto η > 0$ .

Recall At a step-decay boundary

$t = s$ , has the drop happened? Yes. ::: $⌊ s / s ⌋ = 1$ , so $η = η_{0} γ^{1}$ ; plateaus are half-open $[k s, (k + 1) s)$ and the boundary belongs to the new, lower plateau.

Recall Why does the cosine value at the quarter-point exceed the linear guess?

Because cosine stays flat/high early. ::: $cos (π /4) \approx 0.707$ , so the bracket $(1 + cos) \approx 1.707$ keeps $η$ near $85%$ of range, versus $75%$ for a straight line.

Recall Which tool inverts an exponential-decay schedule to solve for

$t$ ? The natural logarithm $ln$ . ::: $ln$ is the exact inverse of $e^{x}$ , bringing $t$ down out of the exponent.

For tuning $η_{0}$ , $T$ , $T_{w}$ themselves, see Hyperparameter Tuning.