Worked examples — Learning rate effects on convergence

This page is the practice ground for Learning rate effects on convergence. The parent note showed you why the learning rate $\eta$ (the small number that scales each step of Gradient Descent) controls whether training crawls, cruises, or explodes. Here we hammer through every case that can happen, one worked example per case, so you never meet a situation you haven't already seen.

Before we start, one reminder of the only rule we use over and over — the update step:

For a bowl-shaped loss $\mathcal{L}(w)=\frac{1}{2}\lambda w^2$ the slope is $g=\lambda w$, so one step becomes $w_{t+1}=w_t-\eta \lambda w_t=(1-\eta \lambda )w_t.$ The whole story of convergence lives in that one number in brackets. Call it the step multiplier: $r=1-\eta \lambda .$ Every iteration multiplies your distance-from-the-minimum by $r$. So $w_t=r^t{w}_0$. Keep this in your pocket — most examples below just compute $r$ and read off its meaning.

---

The scenario matrix

Here is every case class a single-variable learning-rate problem can throw at you. Each of the 8 examples below is tagged with the cell it fills.

#	Case class	What makes it special	$\eta$ vs stability edge ${\eta }_{\text{crit}}=2/\lambda$	Example
A	Tiny $\eta$	$0<r<1$ near 1, glacial	$\eta \ll {\eta }_{\text{crit}}$	Ex 1
B	"Sweet-spot" $\eta$	$0<r<1$ small, fast	$\eta <{\eta }_{\text{crit}}$	Ex 2
C	Perfect $\eta =1/\lambda$	$r=0$, one-step	$\eta =\frac{1}{2}{\eta }_{\text{crit}}$	Ex 3
D	Overshoot but converging	$-1<r<0$, zig-zag inward	$\eta <{\eta }_{\text{crit}}$	Ex 4
E	Exact boundary	$r=-1$, permanent oscillation	$\eta ={\eta }_{\text{crit}}$	Ex 5
F	Divergence	$r<-1$, explosion	$\eta >{\eta }_{\text{crit}}$	Ex 6
G	Degenerate / zero inputs	flat loss ($\lambda =0$), or start at minimum ($w_0=0$)	any $\eta$	Ex 7
H	Word problem + exam twist	pick $\eta$ from an ill-conditioned 2-D bowl (two $\lambda$s)	must satisfy the worst $\lambda$	Ex 8

The figure above plots the step multiplier $r=1-\eta \lambda$ against $\eta$ and colours the five behaviour zones. Every example is one dot on this line. Notice the two fences: $\eta =1/\lambda$ (where $r=0$, best possible) and $\eta =2/\lambda$ (where $r=-1$, the cliff edge).

---

Example 1 — Case A: tiny learning rate (glacial)

1. Find the step multiplier. $r=1-\eta \lambda =1-0.01\cdot 2=0.98$. Why this step? $r$ tells us the whole trajectory: $w_t=r^t{w}_0$. No need to simulate one row at a time.
2. Read off the sign and size. $0<r<1$, and $r$ is very close to 1. Why this step? Close to 1 means each step barely shrinks the gap — the hallmark of "too small."
3. Compute $w_{10}$. $w_{10}=0.98^{10}\cdot 1=0.8171$. Why this step? We wanted a concrete number to see how little progress happened: after 10 steps we've only closed from $1$ to $\approx 0.82$.

Verify: Started at gap $1$, ended at $0.817$ — still $82\%$ of the way out. Monotonic (same sign, never overshoots), which matches $0<r<1$. Symptom = slow but steady, exactly Case A. ✓

---

Example 2 — Case B: sweet-spot learning rate

1. Step multiplier. $r=1-0.3\cdot 2=0.4$. Why this step? Same trick: $0<r<1$ but comfortably below 1 → fast geometric decay.
2. Three iterations. $w_1=0.4$, $w_2=0.16$, $w_3=0.064$. Why this step? Each step multiplies by $0.4$; we list them so you can literally watch the gap shrink by $60\%$ each time.
3. Compare. After only 3 steps we're at $0.064$ — far closer than Example 1 reached in 10. Why this step? This is the punchline: a bigger (but still safe) $\eta$ converges much faster.

Verify: $w_3=0.4^3=0.064$. ✓ Same sign throughout (no overshoot), fast. This is the "just right" no-schedule-needed regime — Case B.

---

Example 3 — Case C: the perfect one-step rate

1. Set the target. We want $w_1=r{w}_0=0$, and $w_0=1\ne 0$, so we need $r=0$. Why this step? $r$ is the only thing multiplying $w_0$; the only way to kill the gap in one step is $r=0$.
2. Solve $r=0$. $1-\eta \lambda =0\Rightarrow \eta =1/\lambda =1/2=0.5$. Why this step? This is exactly Newton's method here: stepping by $1/\lambda$ (the inverse curvature) hits the bottom of a quadratic in one shot. See the curvature view.
3. Take the step. $w_1=1-0.5\cdot g_0$, with $g_0=\lambda w_0=2$, so $w_1=1-0.5\cdot 2=0$. Why this step? Confirms the algebra with the raw update rule, not just the $r$ shortcut.

Verify: $\eta \lambda =0.5\cdot 2=1\Rightarrow r=0\Rightarrow w_1=0$. ✓ Note $\eta =0.5$ is exactly half the stability edge $2/\lambda =1$. Case C.

---

Example 4 — Case D: overshoot that still converges

1. Step multiplier. $r=1-0.7\cdot 2=-0.4$. Why this step? Now $r$ is negative but its size $|r|=0.4<1$ — the gap shrinks yet flips side each step.
2. Iterate. $w_1=-0.4$, $w_2=0.16$, $w_3=-0.064$. Why this step? We overshot past $0$ to the negative side (that's Case D), then overshot back, each time landing closer to $0$.
3. Interpret the picture. In the figure the point hops left-of-zero, right-of-zero, left again — a shrinking spiral toward the bottom. Why this step? Negative $r$ = geometry of crossing the valley; $|r|<1$ = still winning.

Verify: $w_3=(-0.4{)}^3=-0.064$; $|w_3|=0.064$ matches Example 2's magnitude but with alternating signs. Still converging because $\eta =0.7<2/\lambda =1$. ✓ Case D.

---

Example 5 — Case E: exactly on the boundary (endless oscillation)

1. Find the edge. Stability edge is ${\eta }_{\text{crit}}=2/\lambda =2/2=1$. Use $\eta =1$. Why this step? This is the $r=-1$ case — the exact frontier from the parent's condition $\eta <2/L$.
2. Compute $r$ and iterate. $r=1-1\cdot 2=-1$. So $w_1=-1$, $w_2=1$, $w_3=-1,\dots$ Why this step? $|r|=1$ means the gap never shrinks and never grows — it just flips sign: a perpetual bounce between $+1$ and $-1$.
3. Conclude. Loss $\mathcal{L}=w^2=1$ at every step — flat forever, no progress. Why this step? Shows that "not diverging" is not the same as "converging." The edge is useless in practice.

Verify: $w_t=(-1{)}^t$, so $|w_t|=1$ for all $t$. Loss constant at $1$. ✓ Case E — the reason we use $\eta <2/L$ strictly, and in practice $\eta \approx 1/L$ for margin.

---

Example 6 — Case F: divergence (explosion)

1. Step multiplier. $r=1-1.5\cdot 2=-2$. Why this step? $|r|=2>1$: every step doubles the gap while flipping its sign → runaway.
2. Iterate. $w_1=-2$, $w_2=4$, $w_3=-8$, $w_4=16$. Why this step? Concrete explosion, matching the parent note's "large $\eta$" example.
3. Closed form. $|w_t|=2^t$, so $w_t\to \pm \infty$. Why this step? Confirms exponential divergence; in real training this shows up as loss = NaN.

Verify: $|w_3|=2^3=8$, sign $(-1{)}^3=-1\Rightarrow w_3=-8$. ✓ $\eta =1.5>2/\lambda =1$, so divergence is guaranteed — Case F. The cure is a smaller $\eta$, or Momentum/adaptive rates that damp this.

---

Example 7 — Case G: degenerate & zero inputs

1. (a) gradient of a constant. $g=\nabla \mathcal{L}=0$ everywhere. Why this step? The slope of a flat function is zero, so there's no downhill direction.
2. (a) apply the update. $w_1=w_0-\eta \cdot 0=5$. We never move, for any $\eta$. Why this step? Shows the update is powered by the gradient — kill the gradient and learning stalls regardless of $\eta$. (Equivalently $r=1-\eta \cdot 0=1$: gap unchanged.)
3. (b) start exactly at the bottom. $g_0=\lambda w_0=2\cdot 0=0$, so $w_1=0-0.3\cdot 0=0$. Why this step? We're already at the minimum; the gradient is zero, so we correctly stay put — a fixed point of the update.

Verify: (a) $w_t=5$ for all $t$ (no motion). (b) $w_t=0$ for all $t$ (already solved). ✓ Both are degenerate/zero-input cells — Case G. Lesson: a zero gradient (flat region, saddle, or exact optimum) freezes vanilla gradient descent; that's why Momentum exists to push through flat spots.

---

Example 8 — Case H: word problem + exam twist (one $\eta$, two curvatures)

1. Each coordinate updates independently. For coordinate $i$: $w_{i,t+1}=(1-\eta {\lambda }_i)w_{i,t}$. Why this step? The loss splits into two separate 1-D bowls, so our $r$-analysis applies to each with its own ${\lambda }_i$.
2. Stability must hold for BOTH. Need $|1-\eta {\lambda }_i|<1$ for $i=1,2$. The tightest is the largest $\lambda$: $\eta <\frac{2}{{\lambda }_{\max }}=\frac{2}{20}=\mathrm{0.1.}$ Why this step? If the steep direction (${\lambda }_2=20$) diverges, the whole run diverges — the big curvature sets the ceiling. This is smoothness $L={\lambda }_{\max }$ in action ($\eta <2/L$).
3. Now the flat direction pays the price. Take $\eta =0.1$ (the limit). Its multiplier is $r_1=1-\eta {\lambda }_1=1-0.1\cdot 1=\mathrm{0.9.}$ Why this step? $r_1=0.9$ is painfully close to 1 → $w_1$ crawls (like Case A) even though we chose the biggest legal $\eta$.
4. Name the culprit (exam twist). The ratio $\kappa ={\lambda }_{\max }/{\lambda }_{\min }=20/1=20$ is the condition number. Why this step? Big $\kappa$ = one direction forces a small $\eta$ while another then converges slowly — the classic ill-conditioning bottleneck. Fixes: Momentum, Adam Optimizer, or preconditioning with the Hessian Matrix.

Verify: ${\eta }_{\max }=2/20=0.1$; at that $\eta$, $r_1=0.9$ (slow) and $r_2=1-0.1\cdot 20=-1$ (edge — barely stable). $\kappa =20$. ✓ Case H — this is precisely why constant single-$\eta$ descent struggles and adaptive methods win.

---

Recall Quick self-test

The step multiplier for a 1-D bowl is r ::: r = 1 - η·λ Gap after t steps ::: w_t = r^t · w_0 One-step-to-minimum happens when ::: η = 1/λ (so r = 0) Permanent oscillation (never settles) happens at ::: η = 2/λ (r = -1) Divergence happens when ::: η > 2/λ (|r| > 1) In a multi-dimensional bowl, the safe η is capped by the ::: largest curvature λ_max, giving η < 2/λ_max The number that predicts slow convergence despite a legal η is the ::: condition number κ = λ_max/λ_min