Visual walkthrough — Early stopping

---

Step 1 — What "the loss" even is, as a landscape

WHAT. Picture the network's job as walking around a hilly surface. Every point on the ground is a possible setting of the weights $w$ (all the tunable numbers of the network, stacked into one long list). The height of the ground at that point is the loss $J(w)$ — how badly the network does on the training data. Low ground = good.

WHY a landscape? Because training is just walking downhill. If we can understand the shape of the ground near the bottom, we understand where training ends up.

PICTURE. Below, the ground dips into a bowl. The very bottom is the point $w^{\ast }$ — the weights that fit the training data best. Notice the bowl is stretched: steep along one direction, gentle along another. That lopsidedness is the whole plot of this movie.

---

Step 2 — Approximating the bowl with a clean equation

WHAT. Real loss surfaces are bumpy. But near the bottom, almost any smooth valley looks like a simple, symmetric bowl. We replace the messy surface with its clean stand-in:

$J(w)\approx \underset{\text{height of the floor}}{\underbrace{J(w^{\ast })}}+\frac{1}{2}\underset{\text{how far we are, as a row}}{\underbrace{(w-w^{\ast }{)}^{\top }}}\underset{\text{shape of the bowl}}{\underbrace{H}}\underset{\text{how far we are, as a column}}{\underbrace{(w-w^{\ast })}}$

Term by term:

• $w-w^{\ast }$ — the arrow from the bottom to where we stand. If it's zero, we're at the best fit.
• $H$ — the Hessian, a table of numbers describing curvature: how fast the ground steepens as you walk in each direction (see Hessian and Curvature).
• $\frac{1}{2}(w-w^{\ast }{)}^{\top }H(w-w^{\ast })$ — reads as "distance, bent by the bowl's shape, distance" — a number: how high above the floor we sit.

WHY this tool and not another? Why a second-order (squared-distance) approximation, not just a straight-line one? With the slope gone, a straight line would be perfectly flat — it can't have a bottom that curves back up. The squared term is the simplest shape that can curve up into a valley on all sides. It's the least we can write down that still has a lowest point. That's exactly why the Hessian (the second-derivative object) shows up and nothing simpler will do.

PICTURE. The real wobbly valley (pink) and its smooth parabola stand-in (yellow) lying almost on top of each other near the bottom — with the tangent at $w^{\ast }$ drawn flat (slope zero).

---

Step 3 — Straightening the bowl (eigen-directions)

WHAT. A tilted, stretched bowl is annoying because moving one weight changes several curvatures at once. So we rotate our map to line the axes up with the bowl's natural stretch directions. This is the job of writing

$H=Q\Lambda Q^{\top }.$

• $Q$ — a rotation: its columns are the bowl's natural axes (the eigenvectors, the "principal directions" of the valley).
• $\Lambda$ — a diagonal list of curvatures ${\lambda }_1,{\lambda }_2,\dots$ (the eigenvalues), one per natural axis. Each ${\lambda }_i$ is "how steep the bowl is along axis $i$."
• $Q^{\top }w$ — the same point $w$, but described in the rotated map's coordinates.

WHY do this? Because in the rotated map the directions stop interacting. Each axis becomes its own independent 1-D bowl of steepness ${\lambda }_i$. A hard multi-dimensional problem becomes many easy one-dimensional ones we can solve separately. This is the reason eigen-coordinates are worth the trouble.

PICTURE. Left: the tilted bowl's contour ovals. Right: after rotating by $Q$, the ovals become axis-aligned — one steep axis (big $\lambda$), one shallow axis (small $\lambda$).

---

Step 4 — One step of gradient descent, in this map

WHAT. Gradient Descent walks downhill: from where you stand, subtract a small nudge in the steepest-uphill direction. For our bowl the steepest-uphill arrow works out to $\nabla J(w)=H(w-w^{\ast }),$ "curvature $\times$ how far from the bottom" — the farther you stand from $w^{\ast }$ and the steeper the bowl, the stronger the uphill pull. Plug it into the update rule $w^{(t)}=w^{(t-1)}-\epsilon \nabla J$ and measure everything from the bottom:

$w^{(t)}-w^{\ast }=\underset{\text{shrink factor}}{\underbrace{(I-\epsilon H)}}\underset{\text{last step’s arrow from bottom}}{\underbrace{(w^{(t-1)}-w^{\ast })}}$

• $\epsilon$ — the learning rate: how big a step we take (see Learning Rate Schedules).
• $I$ — "do nothing" (the identity).
• $I-\epsilon H$ — each step, the arrow-from-bottom gets multiplied by this: it shrinks us toward $w^{\ast }$.

WHY subtract the gradient? The gradient points the way up. To go down, step the opposite way. Small $\epsilon$ keeps us from overshooting the bowl's far wall.

PICTURE. An arrow from the current point to $w^{\ast }$, and the next arrow after one update — shorter, pointing more toward the bottom.

---

Step 5 — Unrolling $t$ steps along one axis

WHAT. In the rotated map each axis is independent, so pick one axis $i$ and track its single coordinate $[Q^{\top }w{]}_i$ (the "pick out axis $i$" number from Step 3). Start training from tiny weights, which we idealize as $w^{(0)}=0$, so on this axis the arrow-from-bottom starts at $0-[Q^{\top }{w}^{\ast }{]}_i=-[Q^{\top }{w}^{\ast }{]}_i$.

Because the axes don't interact, one update just multiplies that arrow by the single number $(1-\epsilon {\lambda }_i)$. So after $t$ updates the arrow-from-bottom is that number applied $t$ times: $[Q^{\top }{w}^{(t)}{]}_i-[Q^{\top }{w}^{\ast }{]}_i=(1-\epsilon {\lambda }_i{)}^t(0-[Q^{\top }{w}^{\ast }{]}_i)=-(1-\epsilon {\lambda }_i{)}^t[Q^{\top }{w}^{\ast }{]}_i.$

Now move the $[Q^{\top }{w}^{\ast }{]}_i$ term to the other side to read off the weight itself:

$[Q^{\top }{w}^{(t)}{]}_i=\underset{\text{how much of the target we’ve reached}}{\underbrace{(1-(1-\epsilon {\lambda }_i{)}^t)}}\underset{\text{the full best-fit value on this axis}}{\underbrace{[Q^{\top }{w}^{\ast }{]}_i}}$

Read the front factor as a progress dial from $0$ to $1$:

• $t=0$: factor $=0$ → we're still at the start (fully "regularized," weights near zero).
• $t\to \infty$: $(1-\epsilon {\lambda }_i{)}^t\to 0$ → factor $\to 1$ → we reach the full best-fit value $[Q^{\top }{w}^{\ast }{]}_i$ (no regularization left).

WHY does the axis matter? The dial climbs at speed set by ${\lambda }_i$:

• Steep axis (big ${\lambda }_i$): $(1-\epsilon {\lambda }_i)$ is small → dies fast → dial hits $1$ in a few steps. Fit quickly.
• Shallow axis (small ${\lambda }_i$): $(1-\epsilon {\lambda }_i)\approx 1$ → dies slowly → dial still near $0$ after $t$ steps. Barely fit.

PICTURE. Two progress curves versus step number $t$: the steep-axis dial shoots up to $1$; the shallow-axis dial crawls. A vertical "we stop here" line at some $t$ cuts the shallow axis while it's still low.

---

Step 6 — What $L_2$ regularization does, for comparison

WHAT. L2 Regularization (Weight Decay) doesn't stop early — it changes the destination. It adds a penalty $\frac{\alpha }{2}\Vert w{\Vert }^2$ that dislikes large weights, with strength ==$\alpha$== ("weight-decay"). Its new best point, per axis, is:

$[Q^{\top }\tilde{w}{]}_i=\underset{\text{shrink dial}}{\underbrace{\frac{{\lambda }_i}{{\lambda }_i+\alpha }}}[Q^{\top }{w}^{\ast }{]}_i$

Read the dial $\frac{{\lambda }_i}{{\lambda }_i+\alpha }$:

• Steep axis (${\lambda }_i\gg \alpha$): dial $\approx 1$ → keep the full fit.
• Shallow axis (${\lambda }_i\ll \alpha$): dial $\approx 0$ → crush the weight toward zero.

WHY the same pattern? Look: it keeps steep axes and kills shallow ones — exactly what stopping early did in Step 5, but for a completely different reason. One method ran out of time; the other paid a penalty. Same fingerprint.

PICTURE. Two dials side by side, both plotted against curvature ${\lambda }_i$: the $L_2$ dial and the early-stopping dial trace the same S-shape from $0$ (shallow) to $1$ (steep).

---

Step 7 — Snapping the two dials together

WHAT. Two dials do the same thing, so set them equal and see what $t$ and $\alpha$ must satisfy. Equal shrink means equal un-fit amounts:

$\boxed{(1-\epsilon {\lambda }_i{)}^t\approx \frac{\alpha }{{\lambda }_i+\alpha }}$

The left side is early stopping's leftover; the right side is $L_2$'s leftover. Now take the natural logarithm of both sides (the tool that turns a power $t$ into a plain multiplier — that's exactly what $\log$ is for):

$t\log (1-\epsilon {\lambda }_i)=\log \frac{\alpha }{{\lambda }_i+\alpha }.$

For small steps ($\epsilon {\lambda }_i\ll 1$) use $\log (1-\epsilon {\lambda }_i)\approx -\epsilon {\lambda }_i$ (a straight line hugging the log near $1$). Divide through and flip signs:

$\boxed{t\approx \frac{1}{\epsilon {\lambda }_i}\log \frac{{\lambda }_i+\alpha }{\alpha }}$

• $t$ — number of steps taken (our stopping choice).
• $\frac{1}{\epsilon {\lambda }_i}$ — the natural time-scale of axis $i$: steep axes (big ${\lambda }_i$) have a short time-scale, so they finish fast; this is where the ${\lambda }_i$-dependence lives and it does not drop out.
• $\log \frac{{\lambda }_i+\alpha }{\alpha }$ — how many time-scales you must wait to shrink axis $i$ down to the target $L_2$ dial. Bigger $\alpha$ → smaller log → stop sooner.

WHY the log, and where does $t\approx 1/(\epsilon \alpha )$ come from? The relation above is the honest one: $t$ genuinely depends on both ${\lambda }_i$ and $\alpha$. The often-quoted shortcut $t\approx 1/(\epsilon \alpha )$ is only the special case of a shallow axis with $\alpha \ll {\lambda }_i$: then $\log \frac{{\lambda }_i+\alpha }{\alpha }\approx \log \frac{{\lambda }_i}{\alpha }$ is large while ${\lambda }_i$ is small, and for the barely-moved directions ($\epsilon {\lambda }_{i}t\ll 1$) the leftover $(1-\epsilon {\lambda }_i{)}^t\approx 1-\epsilon {\lambda }_{i}t$ matches $\frac{\alpha }{{\lambda }_i+\alpha }\approx 1-\frac{{\lambda }_i}{\alpha }\cdot \frac{\alpha }{{\lambda }_i}$... more simply, equating $\epsilon {\lambda }_{i}t\approx {\lambda }_i/\alpha$ cancels the ${\lambda }_i$ and leaves $t\approx 1/(\epsilon \alpha )$. So the clean reciprocal is a linearized snapshot for the weakly-fit directions, not the full law.

WHY care? Either way the punchline stands: more steps $t$ = weaker effective regularization = smaller $\alpha$. Choosing when to stop is choosing a regularization strength. The stopwatch and the penalty knob are the same knob wearing two costumes.

PICTURE. A see-saw: "steps $t$" on one side, "decay $\alpha$" on the other, hinged at the axis time-scale $1/(\epsilon {\lambda }_i)$ — push $t$ up and the matched $\alpha$ goes down.

---

Step 8 — Edge & degenerate cases (never leave a gap)

WHAT. We must check the corners, or the reader will hit a scenario we never drew.

• A perfectly flat direction, ${\lambda }_i=0$. The dial $\frac{{\lambda }_i}{{\lambda }_i+\alpha }=0$ (for $L_2$) and $(1-\epsilon \cdot 0{)}^t=1$ so early-stopping's progress $1-1=0$. Both keep this weight at zero forever. Good: a direction the loss doesn't care about never gets a large weight. Consistent.
• A very steep direction, ${\lambda }_i$ large. Dial $\to 1$ both ways: fit fully, no shrink. Consistent.
• Learning rate too big, $\epsilon {\lambda }_i>2$. Then $|1-\epsilon {\lambda }_i|>1$: the shrink factor grows, the arrow-from-bottom explodes — training diverges. This is why the whole "shrink toward $w^{\ast }$" story needs $0<\epsilon {\lambda }_i<2$. Outside that, no equivalence — you just blow up.
• Zero steps, $t=0$. Progress dial $=0$: weights stay at their tiny init. This is infinite regularization ($\alpha \to \infty$) — an untrained, maximally simple model. Consistent with $\alpha \approx 1/(\epsilon t)$ as $t\to 0$.

WHY show all four? Because the boxed equivalence only holds inside the stable, small-step regime. A reader must know exactly where the map is trustworthy and where it tears: outside $0<\epsilon {\lambda }_i<2$ the "shrink toward $w^{\ast }$" picture is simply false, and the flat/steep corners confirm that both methods agree at the extremes. Showing every corner is what makes the "early stopping $=$ $L_2$" claim safe to rely on rather than a lucky coincidence in the middle.

PICTURE. The shrink factor $(1-\epsilon {\lambda }_i)$ plotted against $\epsilon {\lambda }_i$: safe green band $0$ to $2$, danger zones outside where the arrow grows.

---

The one-picture summary

WHAT. Everything at once: the stretched bowl, the training path curving in from the origin, and the two dials it walks past — reaching steep axes fast, shallow axes never. Stop the walk anywhere = pick an $L_2$ strength.

Recall Feynman: retell the walkthrough with no symbols

Imagine a bowl that's been squashed — steep on one side, shallow on the other. Right at the very bottom the ground is flat (no downhill left — that's what "bottom" means), which is why the simple squared-bowl picture is all you need there. You roll a marble in from the corner; it dives quickly toward the bottom along the steep direction but dawdles along the shallow one. If a friend taps your shoulder and stops the marble early, its position along the shallow direction is still small — you've automatically kept the "important" weights and left the "unimportant" ones tiny. Now here's the twist: there's a totally different method that adds a spring pulling every weight toward zero, harder on the weights the loss doesn't care about. When you compare where the marble stopped to where the spring settles it — they land in the same spot. So the moment you stop the clock is the same as choosing how strong the spring is. Stop sooner = stronger spring = simpler model; run forever = no spring = memorize everything.

Recall Check yourself

Why do shallow (small-${\lambda }_i$) directions stay small when we stop early? ::: Their per-step shrink factor $(1-\epsilon {\lambda }_i)$ is near $1$, so the progress dial $1-(1-\epsilon {\lambda }_i{)}^t$ barely rises in $t$ steps — the weight hasn't traveled far from zero. Why does the linear term of the Taylor expansion drop out at $w^{\ast }$? ::: Because $w^{\ast }$ is the minimum: the slope there is zero, $\nabla J(w^{\ast })=0$, so the term $\nabla J(w^{\ast }{)}^{\top }(w-w^{\ast })$ is zero times something — it vanishes. What is the honest per-axis link between stopping time and weight decay? ::: $t\approx \frac{1}{\epsilon {\lambda }_i}\log \frac{{\lambda }_i+\alpha }{\alpha }$; the reciprocal $t\approx 1/(\epsilon \alpha )$ is only the weakly-fit special case. What breaks the equivalence entirely? ::: A learning rate so big that $\epsilon {\lambda }_i>2$: the shrink factor exceeds $1$ in magnitude and training diverges instead of settling.

Related: Overfitting and Generalization · Validation and Cross-Validation · 3.2.11 Early stopping (Hinglish)