Noesis
3.2.11Training Deep Networks

Early stopping

1,912 words9 min readdifficulty · medium4 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHY does early stopping exist?

WHAT problem does it solve? Overfitting — the gap between low training error and high test error.

HOW is it a form of regularization? Fewer effective training steps means the weights travel a shorter distance from their small random initialization. Small weights ≈ a simpler function. So limiting training time limits model complexity — just like an L2L_2 penalty does (we prove this below).

Figure — Early stopping

The mechanism, step by step

HOW it runs (the algorithm):

  1. Split data into train / validation (validation never touches the gradients).
  2. After each epoch, compute validation loss LvalL_{\text{val}}.
  3. If LvalL_{\text{val}} improved → save these weights (this is the "best" checkpoint) and reset a counter.
  4. If it did not improve → increment the counter.
  5. When counter == patience pp → stop, and restore the saved best weights.

Deriving why early stopping ≈ L2L_2 regularization

We show, from first principles, that stopping early is (approximately) equivalent to weight decay. This is the "deep" reason it regularizes.

Gradient descent update. With learning rate ε\varepsilon and gradient ∇J=H(w−w∗)\nabla J = H(w-w^*): w(t)−w∗=(I−εH) (w(t−1)−w∗)w^{(t)} - w^* = (I - \varepsilon H)\,(w^{(t-1)} - w^*) Why? Substitute the gradient into w(t)=w(t−1)−ε∇J(w(t−1))w^{(t)} = w^{(t-1)} - \varepsilon \nabla J(w^{(t-1)}) and subtract w∗w^*.

Diagonalize. Write H=QΛQ⊤H = Q\Lambda Q^\top with eigenvalues λi\lambda_i. In eigen-coordinates [Q⊤w]i[Q^\top w]_i, starting from w(0)=0w^{(0)}=0, unrolling tt steps gives: [Q⊤w(t)]i=(1−(1−ελi)t) [Q⊤w∗]i[Q^\top w^{(t)}]_i = \big(1-(1-\varepsilon\lambda_i)^t\big)\,[Q^\top w^*]_i Why? Each eigen-direction shrinks independently by factor (1−ελi)(1-\varepsilon\lambda_i) per step; geometric series.

Worked examples

Practical knobs

Knob WHAT it controls Failure if wrong
Patience pp how long to tolerate no improvement too small → stop early (underfit); too large → wasted compute, mild overfit
min-delta minimum change counted as "improvement" too large → stops too soon
Monitored metric val loss vs val accuracy accuracy is flatter/noisier; loss usually smoother
Restore best weights roll back to best checkpoint if off, you keep the last (worse) weights

Flashcards

What shape does the validation-loss curve take that early stopping exploits?
A U-shape: it decreases, reaches a minimum (best generalization), then rises as the model overfits.
Why must the validation set never influence gradient updates?
So it stays an unbiased estimate of generalization; if it drove training, its loss would also become optimistically biased.
Define "patience" in early stopping.
The number of consecutive epochs with no validation improvement that we tolerate before stopping.
Why restore the best checkpoint rather than the last weights?
The last epochs after the minimum are more overfit; the best checkpoint had the lowest validation loss = best generalization.
In the quadratic analysis, what plays the role of 1/α1/\alpha (inverse weight-decay)?
The number of training steps tt — more steps = weaker effective regularization = larger effective weights.
Early stopping keeps which eigen-directions and suppresses which?
Keeps high-curvature (large λi\lambda_i) directions that fit quickly; suppresses low-curvature (small λi\lambda_i) directions that barely move in tt steps.
Why is stopping at the first uptick of validation loss a mistake?
Validation loss is noisy; a single blip may be followed by real improvement. Use patience >1>1.
Matching condition between early stopping and L2L_2?
(1−ελi)t≈αλi+α(1-\varepsilon\lambda_i)^t \approx \dfrac{\alpha}{\lambda_i+\alpha}.
Recall Feynman: explain it to a 12-year-old

Imagine studying for a test. At first, learning helps — you understand the ideas. But if you keep cramming the exact practice questions all night, you start memorizing those specific questions instead of the ideas, and you do worse on the real (different) test. Early stopping is a friend who watches your practice-test scores and taps you on the shoulder: "You peaked — stop now, go take the real exam." We even keep a photo (checkpoint) of your best brain-state and hand it back to you.

Connections

  • Overfitting and Generalization — early stopping is a cure for the train/test gap.
  • L2 Regularization (Weight Decay) — provably near-equivalent to early stopping under a quadratic loss.
  • Gradient Descent — the update rule whose trajectory we truncate.
  • Hessian and Curvature — eigenvalues λi\lambda_i decide per-direction shrinkage.
  • Validation and Cross-Validation — supplies the signal early stopping watches.
  • Learning Rate Schedules — interacts: ελi\varepsilon\lambda_i sets how fast each direction is learned.

Concept Map

memorizes

causes

U-shaped minimum

halts at

solves

monitors

uses

avoids stopping on

restores

limits

shorter weight travel

approx equivalent to

derived via

Training too long

Fits noise not signal

Overfitting

Validation loss curve

Best generalization point

Early stopping

Patience p

Noisy val blips

Best checkpoint weights

Fewer training steps

Small weights simpler function

L2 weight decay

Quadratic loss approximation with Hessian H

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, jab hum ek deep network ko train karte hain, shuru mein model asli patterns seekhta hai — general cheezein jo test data pe bhi kaam karti hain. Isliye training loss aur validation loss dono girte hain. Lekin ek point ke baad model ke paas naye general patterns nahi bachte, to woh training data ka noise ya rat-tu memorization karne lagta hai. Us moment se validation loss badhne lagta hai, jabki training loss neeche hi jaata rahta hai. Yeh U-shape (smile) hi early stopping ka pura khel hai.

Early stopping ka matlab: har epoch ke baad validation loss dekho, jahan woh sabse kam ho wahan ke weights save kar lo, aur jab kaafi epochs tak improvement na ho (isko patience kehte hain), tab training rok do aur us best wale weights ko wapas load kar lo. Ek galti jo sab karte hain — validation loss pehli baar badha to turant stop kar dena. Yeh galat hai kyunki curve noisy hota hai, ek chhota blip real improvement ke pehle bhi aa sakta hai. Isliye patience >1>1 rakho aur hamesha best checkpoint restore karo, last wala nahi.

Sabse gehri baat: early stopping actually L2L_2 regularization (weight decay) jaisa hi kaam karta hai. Jitne zyada steps train karoge, weights utne bade ho jaayenge (zyada complex model); jitne kam steps, utne chhote weights (simple model). Maths se prove hota hai ki "kitne steps tt" chalana, weight-decay strength α\alpha ke inverse jaisa hai. High-curvature directions jaldi fit ho jaati hain, low-curvature waali tt steps mein bahut kam hilti hain — to woh apne aap suppress ho jaati hain. Isliye early stopping free aur powerful regularizer hai — bas ek achha validation set chahiye.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,630 wordsD2Visual walkthroughThe derivation in pictures · 3,004 wordsD3Worked examplesEvery scenario covered · 2,527 wordsD4ExercisesGraded, full solutions · 3,503 wordsD5Question bankConcept traps, quick reveals · 1,502 words

Test yourself — Training Deep Networks

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Gradient descent and variants — convergence analysisMaths · 4.10.20Cross-validation — k-foldCoding · 5.6.5Learning rate schedulingAI-ML · 3.2.6Train - validation - test splittingAI-ML · 2.1.12Gradient Boosting MachinesAI-ML · 2.3.12Grid search and random searchAI-ML · 2.6.13Cross-validation pitfalls and nested CVAI-ML · 2.6.15