Visual walkthrough — Learning rate scheduling

---

Step 1 — What "a step downhill" even means

WHAT. Training a network means adjusting a knob (call it $\theta$) to make a badness-score (the loss $L$) as small as possible. Think of $\theta$ as your horizontal position and $L(\theta )$ as the height of the ground under you. You want to walk to the lowest point.

WHY a picture first. Before any algebra, you must see that "training" is just "walk downhill." The whole rest of the page is measuring how big each footstep should be.

PICTURE. Look at the burnt-orange bowl below. The dot is your current position ${\theta }_t$ (the subscript $t$ = "at time-step number $t$", i.e. after $t$ footsteps). The arrow shows one step.

---

Step 2 — The ground is noisy: real gradients wobble

WHAT. In real training we never measure the exact slope. We estimate it from a small random handful of data (a mini-batch). So the slope we read is the true slope plus a random error.

WHY this matters. This single fact — the wobble — is the villain of the whole story. Without it, a constant step would eventually land exactly at the bottom. With it, we will prove you can only jitter nearby. So we must put the wobble into the math right now.

PICTURE. The teal true-slope arrow points cleanly downhill. The faint plum arrows are the noisy slopes we actually read on different mini-batches — scattered around the true one.

So the real step rule is: ${\theta }_{t+1}={\theta }_t-\eta (\nabla L({\theta }_t)+{\xi }_t).$

---

Step 3 — Zoom into the bottom: the valley is a parabola

WHAT. We only care whether we can settle at the minimum, so we zoom right into it. Near any smooth minimum the ground looks like a simple parabola (a symmetric bowl): $L(\theta )=\frac{1}{2}a{\theta }^2.$

WHY a parabola, and why is that allowed? Any smooth valley, viewed close enough, is bowl-shaped — the parabola is the simplest curve with a single lowest point. Choosing coordinates so the minimum sits at $\theta =0$ costs us nothing and makes the algebra clean. This is the same local-minimum picture used everywhere.

PICTURE. The plum parabola with its lowest point at $\theta =0$. The number $a$ is its steepness (curvature): a big $a$ is a narrow steep bowl, a small $a$ a wide gentle one.

Why the derivative here? We need the slope to feed the step rule, and $a\theta$ is the exact slope of this bowl. That is the only reason it enters.

---

Step 4 — One clean update equation

WHAT. Substitute the parabola's slope $a\theta$ into the noisy step rule.

WHY. We now have everything — position, true pull-back, and noise — in one line we can analyse.

${\theta }_{t+1}={\theta }_t-\eta (a{\theta }_t+{\xi }_t)=(1-\eta a){\theta }_t-\eta {\xi }_t.$

PICTURE. The update splits into two moves, drawn as two arrows from ${\theta }_t$:

---

Step 5 — Measuring the jitter: introduce variance

WHAT. "How far from the bottom do we typically end up?" is a question about spread, not a single position. The tool that measures spread is the variance.

WHY variance and not just position? Because the kick is random, ${\theta }_t$ is random too — asking "what is ${\theta }_t$?" has no fixed answer. The honest question is "how wide is the cloud of positions?" Variance answers exactly that: it is the average of ${\theta }^2$, the typical squared distance from the centre. That is why this tool, not another.

PICTURE. A cloud of landing dots around $\theta =0$; the width of the cloud is $V$, the variance.

Now take the variance of both sides of the Step-4 equation. Two facts do the work:

• Multiplying a random quantity by a constant $c$ multiplies its variance by $c^2$ (spread scales with the square of a stretch). So $(1-\eta a){\theta }_t$ has variance $(1-\eta a{)}^{2}V$.
• The kick $-\eta {\xi }_t$ is independent of where you stand, so its variance ${\eta }^2{\sigma }^2$ simply adds (independent spreads add).

$\text{Var}({\theta }_{t+1})=(1-\eta a{)}^2\text{Var}({\theta }_t)+{\eta }^2{\sigma }^2.$

---

Step 6 — Steady state: solve for the noise floor

WHAT. After long training the cloud stops growing or shrinking — it reaches a stable width. Call that steady width $V$, meaning $\text{Var}({\theta }_{t+1})=\text{Var}({\theta }_t)=V$.

WHY. A stable width is the "resting jitter" — the residual error you are stuck with. Finding it is the whole point.

$V=(1-\eta a{)}^{2}V+{\eta }^2{\sigma }^2.$

Now solve for $V$. Move the first term over: $V(1-(1-\eta a{)}^2)={\eta }^2{\sigma }^2.$ Expand the bracket — this is where the shrink factor pays off: $1-(1-\eta a{)}^2=2\eta a-{\eta }^2{a}^2=\eta a(2-\eta a).$ Divide: $\boxed{V=\frac{\eta {\sigma }^2}{a(2-\eta a)}\approx \frac{\eta {\sigma }^2}{2a}\text{ for small }\eta .}$

PICTURE. $V$ plotted against $\eta$: a line through the origin. Shrink $\eta$, the floor drops toward zero; only $\eta =0$ touches the bottom.

---

Step 7 — Edge and degenerate cases (never leave a gap)

WHAT / WHY. A derivation you can trust must survive its extremes. We check each corner.

PICTURE. Four mini-panels, one per case.

---

Step 8 — From "decay" to how to decay (Robbins–Monro)

WHAT. We proved $\eta$ must shrink. But shrink how fast? Too fast and you freeze before reaching the bottom; too slow and the noise never dies. The Robbins–Monro conditions pin the sweet spot.

WHY these two sums. Each condition guards against one failure mode we just met.

PICTURE. Two schedule curves: one that stalls (decays too fast) and one that satisfies both conditions.

---

The one-picture summary

Everything compressed: a ball descending a noisy bowl, its jitter-cloud shrinking as $\eta$ decays step by step, ending pinned at the minimum.

Recall Feynman retelling — say it in plain words

Training is walking downhill on a bumpy score. Because we only estimate the slope from random data, every step carries a little random kick. Near the bottom, two forces fight each step: the valley pulls you in by a factor $(1-\eta a)$, and the noise kicks you out by $\eta {\xi }_t$. Balancing them, the width of your resting jitter comes out to $V\approx \eta {\sigma }^2/2a$ — proportional to the step size $\eta$. So with a constant step you're stuck buzzing above the true minimum forever; the only way to press the jitter to zero is to shrink $\eta$ toward zero as you go. But you can't shrink it any old way: shrink too fast and you freeze short of the goal, too slow and the noise never quiets. Robbins–Monro names the balance — the step sizes must add up to infinity (so you can still travel any distance) yet their squares must add up to something finite (so the noise finally dies). That is the whole reason learning-rate schedules exist, and why the good ones look the way they do.

Recall

In one sentence, why can't a constant learning rate reach the true minimum? ::: The stochastic-gradient noise leaves a steady-state jitter of variance $V\propto \eta$, which is only zero when $\eta$ itself is zero. What is the stability ceiling on the step size in the parabola model? ::: $\eta <2/a$; at $\eta a\ge 2$ the variance recursion grows and the ball diverges. If gradients had zero noise (${\sigma }^2=0$), would decay still be required? ::: No — then $V=0$ for any $\eta$; the need to decay comes entirely from gradient noise.

---

See also: Hyperparameter Tuning · Batch Size and Learning Rate scaling · 3.2.06 Learning rate scheduling (Hinglish).