Early stopping: Worked examples — AI-ML deep dive

This is the drill-ground for Early stopping. The parent note built the idea; here we hammer every possible shape of question it can throw at you — noisy curves, flat plateaus, the exact math linking stopping-time to weight decay, degenerate validation sets, and an exam twist. Guess each answer before reading the steps.

The scenario matrix

Every early-stopping problem lands in one of these cells. The worked examples that follow are tagged with the cell they cover.

#	Case class	What makes it tricky	Example
A	Clean U-curve	textbook: clear min, clear rise	Ex 1
B	Noisy curve, blip below patience	a false uptick you must ignore	Ex 2
C	min-delta matters	tiny "improvements" that don't count	Ex 3
D	Degenerate: monotone-down curve	val loss never rises — when do you stop?	Ex 4
E	High-curvature direction ( $λ$ large)	fits almost fully, fast	Ex 5
F	Low-curvature direction ( $λ$ small)	barely moves — effectively pruned	Ex 5
G	Solve for $α$ (steps ⇄ weight decay)	invert the matching formula	Ex 6
H	Solve for $t$ (how many steps for a target shrink)	logs appear	Ex 7
I	Limiting behaviour ( $t \to \infty$ , $λ \to 0$ )	edges of the formula	Ex 8
J	Real-world word problem	noisy tiny val set, choose knobs	Ex 9
K	Exam twist	learning rate too big → divergence	Ex 10

Prerequisites you may want open: Gradient Descent, Hessian and Curvature, L2 Regularization (Weight Decay), Overfitting and Generalization, Validation and Cross-Validation, Learning Rate Schedules.

Part 1 — Reading validation curves (cells A–D)

Before any math, fix the two counters we carry epoch to epoch.

Look at the figure: the cyan curve dips to a bottom (the amber dot = best), then climbs. Patience is the horizontal amber bracket after the bottom — we only quit once that bracket fills up with non-improving epochs.

Worked example Ex 3 — Cell C: min-delta filters fake improvements

Epoch → val loss: 1:0.500, 2:0.499, 3:0.4985, 4:0.498, 5:0.60. Patience $p = 2$ , min-delta $= 0.005$ (an improvement only counts if loss drops by at least $0.005$ ). Forecast: with min-delta on, is epoch 2 an "improvement"?

best = 0.500 (epoch 1). Why? Starting reference.
Epoch 2: drop is $0.500 - 0.499 = 0.001 < 0.005$ → not counted as improvement, counter = 1. Why this step? min-delta says tiny wiggles are noise, not learning.
Epoch 3: compare against best = 0.500: drop $0.500 - 0.4985 = 0.0015 < 0.005$ → still no improvement, counter = 2 = p → stop at epoch 3.
Restore epoch 1 ( $0.500$ ) — the last qualifying best. Why? Nothing beat it by min-delta.

Verify: cumulative drop by epoch 3 is $0.0015 < 0.005$ , so no qualifying improvement occurred; stop at epoch $1 + p = 3$ . If min-delta were $0$ , we'd instead keep going to epoch 4's $0.498$ . ✓ (This shows why min-delta too large ⇒ stops too soon — the table knob.)

Part 2 — Steps as regularization strength (cells E–I)

Now the math engine. Recall from the parent, in one eigen-direction with eigenvalue $λ$ (curvature of the loss bowl, from the Hessian), learning rate $ε$ , starting from $w^{(0)} = 0$ :

The figure plots $fit (t)$ for a high-curvature direction (steep cyan, snaps to 1) and a low-curvature direction (flat amber, crawls). Early stopping = drawing a vertical line at your stop-time $t$ : it reads off "keep the steep ones, suppress the flat ones."

Worked example Ex 5 — Cells E & F: high- vs low-curvature after

$t = 10$ steps Let $ε = 1$ . High-curvature direction $λ_{H} = 0.5$ (so $ε λ_{H} = 0.5$ ). Low-curvature direction $λ_{L} = 0.02$ (so $ε λ_{L} = 0.02$ ). Stop at $t = 10$ . Forecast: which direction is nearly fully fit, which is barely touched?

High: $fit = 1 - (1 - 0.5)^{10} = 1 - 0. 5^{10} = 1 - 0.000977 = 0.99902$ . Why? $(0.5)^{10}$ is tiny — this direction converged in a handful of steps.
Low: $fit = 1 - (1 - 0.02)^{10} = 1 - 0.9 8^{10} = 1 - 0.8171 = 0.1829$ . Why? $0.98$ shrinks slowly; after 10 steps it's crawled less than 1/5 of the way.
Interpretation: stopping at $t = 10$ keeps the high-curvature signal (fit $\approx 1$ ) and effectively prunes the low-curvature direction (fit $\approx 0.18$ ) — exactly what $L_{2}$ does. Why this matters? Noise-fitting typically lives in the low-curvature, slow directions.

Verify: $1 - 0. 5^{10} = 0.999023...$ and $1 - 0.9 8^{10} = 0.182927...$ . High $≫$ Low. ✓

Worked example Ex 8 — Cell I: limiting behaviour

Two limits, same formula. (a) Let $t \to \infty$ with $0 < ε λ < 2$ . (b) Let $λ \to 0$ at fixed $t$ . Forecast: what does $fit$ approach, and what does that mean for regularization?

(a) If $∣1 - ε λ ∣ < 1$ then $(1 - ε λ)^{t} \to 0$ , so $fit \to 1$ . Why? A number of magnitude $< 1$ raised to a growing power vanishes. Then matching gives $\frac{α}{λ + α} \to 0 \Rightarrow α \to 0$ : train forever ⇒ no regularization. This is the parent's "more steps ⇔ weaker regularization."
(b) As $λ \to 0$ : expand $(1 - ε λ)^{t} \approx 1 - ε λ t$ for small $λ$ , so $fit \approx ε λ t \to 0$ . Why this step? First-order (linear) approximation of a power for tiny argument. A perfectly flat direction ( $λ = 0$ ) is never fit — fit $= 0$ for all finite $t$ . Why it matters? Degenerate flat directions carry no signal and are automatically suppressed forever.

Verify: with $ε λ = 0.1$ , at $t = 200$ , $1 - 0. 9^{200} = 1 - 7.06 \times 1 0^{- 10} \approx 1.0$ (limit a). With $ε = 1, t = 5$ : at $λ = 0.001$ , $fit = 1 - 0.99 9^{5} = 0.004990 \approx ε λ t = 0.005$ (limit b). ✓

Part 3 — Word problem and exam twist (cells J, K)

Worked example Ex 9 — Cell J: real-world, noisy tiny validation set

You have 10,000 samples; you carved off a 200-sample validation set. Val loss jitters by $\pm 0.03$ epoch to epoch. Your true best is at epoch 40, but a $+ 0.03$ noise spike appears at epochs 41–42 before recovery. Current patience $p = 2$ , min-delta $= 0$ . Forecast: with $p = 2$ , do you stop prematurely? What do you change?

At epoch 40 best is set. Epochs 41, 42 spike up (noise) → counter = 1, 2 → stop at 42, missing the recovery. Why this step? Noise band $0.03$ can create 2 consecutive fake rises, exhausting a small patience.
Fix option A: raise patience so it exceeds the noise-run length, e.g. $p = 10$ . Why? Validation noise of $\pm 0.03$ rarely produces 10 fake rises in a row.
Fix option B: set min-delta $= 0.03$ so noise-sized wiggles don't reset/increment on real improvements; or enlarge the val set (200 → 1000) to shrink the jitter. Why? Standard error of a mean scales like $1/ n$ , so $5 \times$ more samples cuts noise by $5 \approx 2.24$ .

Verify: noise reduction factor from 200→1000 samples: $1000/200 = 5 = 2.236$ ; new jitter $\approx 0.03/2.236 = 0.0134$ . With $p = 2$ and a 2-epoch noise run you stop at epoch $40 + 2 = 42$ (premature). ✓

Worked example Ex 10 — Cell K: exam twist — learning rate too big

The shrinkage formula assumed the factor $(1 - ε λ)$ has magnitude $< 1$ . Take $λ = 1$ but a reckless learning rate $ε = 2.5$ , so $ε λ = 2.5$ . Compute $fit (t)$ for $t = 1, 2, 3$ . Forecast: does the weight converge toward $w^{*}$ , or blow up?

Factor $= 1 - ε λ = 1 - 2.5 = - 1.5$ , magnitude $1.5 > 1$ . Why check this? The whole geometric-series/convergence story needs $∣1 - ε λ ∣ < 1$ .
$fit (1) = 1 - (- 1.5)^{1} = 1 - (- 1.5) = 2.5$ . $fit (2) = 1 - (- 1.5)^{2} = 1 - 2.25 = - 1.25$ . $fit (3) = 1 - (- 1.5)^{3} = 1 - (- 3.375) = 4.375$ . Why? $(- 1.5)^{t}$ grows and flips sign — the iterate oscillates with growing amplitude.
Conclusion: the weight diverges (overshoots past $w^{*}$ , alternating sides, growing). Early stopping can't save a diverging run — you must first fix the learning rate to satisfy $ε < 2/ λ$ (here $ε < 2$ ). Why it matters? The stopping-as-regularization equivalence is only valid inside the stable regime.

Verify: with $ε λ = 2.5$ : fit(1)=2.5, fit(2)=−1.25, fit(3)=4.375; magnitudes grow ⇒ divergence. Stability requires $ε λ < 2$ , i.e. $ε < 2$ for $λ = 1$ . ✓

Recall Self-test: name the cell, then solve

Curve 1:0.7, 2:0.5, 3:0.52, 4:0.48, 5:0.49, 6:0.50, 7:0.51, $p = 3$ . Which epoch restored & when do we stop? ::: best = 0.48 at epoch 4; epochs 5,6,7 fail → counter 1,2,3 → stop at epoch 7, restore epoch 4. (Cell B, noisy.) $ε λ = 0.2$ , $t = 5$ . Fit? ::: $1 - 0. 8^{5} = 1 - 0.32768 = 0.67232$ . (Cell E/F.) Want fit $= 0.99$ with $ε λ = 0.2$ . Steps? ::: $(0.8)^{t} = 0.01 \Rightarrow t = ln 0.01/ ln 0.8 = 20.6 \Rightarrow 21$ steps. (Cell H.) $ε λ = 0.2$ , $t = 5$ : equivalent $α$ ? ::: $α / λ = 0.32768/ (1 - 0.32768) = 0.4874$ , so $α \approx 0.49 λ$ . (Cell G.)