Intuition The one core ideaWhen you train a neural network, it first learns the real patterns, then starts memorizing the noise — so its performance on unseen data first improves, then gets worse. Early stopping simply watches that unseen-data score and halts training at the exact moment it stops improving.
This page assumes nothing . Before you can read the parent note Early stopping , you need to know what a weight , a gradient , a loss , a Hessian , and an eigen-direction actually are — as pictures, not letters. We build every symbol here, in the order the parent uses them.
w weight vector w
Plain words: every tunable number in the network, stacked into a single column.The picture: a point in space. If the network had 2 knobs, w ( w 1 , w 2 )
Why the topic needs it: early stopping is a rule about where we let this dot travel to . To talk about "stopping the dot early", we first need the dot.
Intuition Why "vector" and not just "a number"?A real network has millions of knobs, not one. A single number could never describe all of them at once. A vector — an ordered list — lets us name the whole configuration with one symbol w moving a point .
J ( w ) how wrong the network is on some data. Big J J
Plain words: the size of the network's mistakes, added up.The picture: a height . Above every point w J ( w )
Why the letter J It is just tradition (from "objective"). It could be any letter; the parent note uses J training cost.
Why the topic needs it: "training" means walking downhill on this landscape — finding weights where the mistakes are small. The lowest points are the good models. See Gradient Descent for the walking rule and Overfitting and Generalization for why a low training J
Common mistake Low training loss is NOT automatically goodFeels right: small mistakes on training data = good model.
Wrong because: the network can lower J training examples that don't appear elsewhere. That's overfitting — the whole reason early stopping exists.
∇ J ( w ) arrow that points in the direction of steepest uphill , and whose length says how steep .
Plain words: "which way is up, and how sharply?"The picture: stand on a hillside; the gradient arrow points straight up the slope. Walk opposite the arrow to go down.
Why the symbol ∇ It is a special "slope operator." ∇ J J w
Why the topic needs it: the training rule is step opposite the gradient . Early stopping is about how many such steps we take.
WHAT this line does: moves the dot a little bit downhill. WHY: repeating it many times slides the dot toward a valley (low mistakes). WHAT IT LOOKS LIKE: figure s03 — the dot hops down the slope, each hop of length set by ε
Common mistake A stride that's too big overshoots — the stability limitFeels right: bigger ε Wrong because: if the stride is longer than the far wall of the valley, each hop lands higher up the opposite side . The dot bounces back and forth, growing instead of settling. For a bowl of curvature λ i ( 1 − ε λ i ) ∣1 − ε λ i ∣ < 1 0 < ε λ i < 2 0 < ε λ i < 1 Learning Rate Schedules .
w ∗ ∗ best weights: the point where J
Picture: the lowest dip in the landscape.
w ( 0 ) before any training. We start them small and random — a dot near the origin ( 0 , 0 )
Why the topic needs both: early stopping is the statement "we stop the dot somewhere between w ( 0 ) w ∗ Fewer steps = the dot stays closer to the small-and-random start = smaller weights = a simpler function. That "distance travelled from the origin" is the hidden regularization.
Intuition Why "small weights ≈ simple model"Weights near zero mean the network barely bends the input — it's almost a straight, gentle function. Big weights let it make sharp wiggles that can trace every noisy data point. Starting near zero and not travelling far keeps the function gentle. Compare L2 Regularization (Weight Decay) , which pulls weights toward zero directly.
Near the bottom w ∗ bowl . But bowls differ: some are round, some are long thin valleys. The object that records that shape is the Hessian.
H changes as you move — i.e. the curvature of the loss bowl in every direction.
Plain words: "how sharply does the ground bend here?"The picture: a steep, narrow ravine has high curvature ; a wide, shallow dish has low curvature .
Why the topic needs it: the parent's whole proof approximates the bowl by its curvature H J ( w ) ≈ J ( w ∗ ) + 2 1 ( w − w ∗ ) ⊤ H ( w − w ∗ ) "height above the bottom = (something) × (how far from the bottom)² weighted by the curvature H The ⊤ Hessian and Curvature .
Common mistake This bowl picture is only LOCAL — read the fine printFeels right: treat J ( w ) ≈ J ( w ∗ ) + 2 1 ( w − w ∗ ) ⊤ H ( w − w ∗ ) Wrong because: it is a Taylor approximation — it only holds near w ∗
H positive-definite (all λ i > 0 upward so w ∗ λ i ≤ 0 The argument assumes a single, convex bowl . Real deep networks are non-convex : many valleys, saddles, and flat plateaus (see the edge-case note in §5). The clean early-stopping ↔ L 2 guiding intuition , applied locally around whichever minimum training happens to approach.
A tilted bowl is confusing. But every bowl has special axes along which it is a clean one-dimensional curve — no cross-talk between directions.
Definition Eigen-directions and eigenvaluesλ i eigen-directions are the natural axes of the bowl (its long axis and short axis). Along each one, a single number λ i i = 1 , 2 , … in that direction alone .
Big λ i = steep wall along axis i Small λ i = nearly flat floor along axis i
Q Λ H = Q Λ Q ⊤ list the axes and curvatures you already met:
Q columns are the eigen-directions (the red and green axes in figure s05). It rotates from your everyday w 1 , w 2 Λ eigenvalues λ 1 , λ 2 , … and zeros elsewhere — the list of curvatures, one per axis.Q ⊤ w w is the whole content.
Why we need them: in these special axes, the messy many-knob update becomes many independent one-knob updates. Each direction shrinks on its own by a factor ( 1 − ε λ i ) t fit fraction in direction i = 1 − ( 1 − ε λ i ) t only in the stable regime 0 < ε λ i < 1 ( 1 − ε λ i ) 1 0 1 ε λ i > 2
Intuition Reading that fit fraction from zeroStart at 0 remaining gap shrinks by ( 1 − ε λ i )
Steep axis (big λ i ( 1 − ε λ i ) → 1 Flat axis (small λ i ( 1 − ε λ i ) 1 0
Common mistake Edge case — real networks are NOT one convex bowlFeels right: treat the loss as the single clean bowl of figure s05.
Wrong because: deep networks are non-convex — the landscape has many minima, saddle points (curving up one way, down another, some λ i < 0 w ∗ L 2 locally around whichever minimum gradient descent is currently sliding into, and it explains why early stopping regularizes there — but it is an approximation, not a global truth. That's exactly why in practice we still rely on the empirical U-shaped validation curve below rather than trusting the formula blindly.
Definition Validation lossL val not allowed to learn from — a fresh mini-exam. We watch it to guess how the model does on the real world .
Picture: two curves against epoch number. Training loss slides down forever; validation loss dips, bottoms out, then climbs — the famous U-shape .
Why the topic needs it: the bottom of that U is exactly where early stopping fires. The training curve alone can't tell you when to stop, because it never turns back up. See Validation and Cross-Validation .
Definition Epoch, patiencep
Epoch: one full pass over the training data. We check L val Patience p how many epochs of no improvement we tolerate before quitting. It exists because the validation curve is jittery, and one bad epoch is often a fluke, not real overfitting.
Loss J = landscape height
Gradient descent step size epsilon
Stability need epsilon lambda below 1
Distance travelled = complexity
Hessian H = local bowl curvature
Eigen-directions and lambda
Fast axes fit slow axes suppressed
Warning non convex many bowls
Test yourself — you are ready for the parent note when you can answer each without peeking.
What does the symbol w The weight vector — every tunable number in the network stacked into one list; pictured as a single point (dot) in space.
What is the loss J ( w ) The height of a landscape above the point w
What does the gradient ∇ J It points in the direction of steepest uphill ; we step opposite it (downhill) to reduce loss.
In w ( t ) ( t ) A label — it counts which training step we're on, not exponentiation.
What does the learning rate ε The size of each downhill stride; per axis we need 0 < ε λ i < 2 < 1
Why does starting at a small random w ( 0 ) Short travel keeps weights near zero, which means a gentle/simple function — less able to trace noise.
What does the Hessian H The local curvature (shape) of the loss bowl near its bottom; it is only a Taylor approximation valid near w ∗ H
What are Q Λ H = Q Λ Q ⊤ Q Λ λ i
What does a large eigenvalue λ i A steep-walled direction; gradient descent fits it fully and fast.
What does a small λ i It barely moves, so it stays near zero — effectively suppressed (this is the regularization).
Why is the clean single-bowl / L 2 Deep networks are non-convex — many minima, saddles and plateaus — so the story holds only locally around whichever minimum training approaches.
What shape does the validation-loss curve take, and what is patience p A U-shape (down, minimum, up); patience is how many non-improving epochs we tolerate before stopping.
