Early stopping — Hinglish notes

3.2.11 · AI-ML › Training Deep Networks

Early stopping exist kyun karta hai?

Yeh kaun si problem solve karta hai? Overfitting — low training error aur high test error ke beech ka gap.

Yeh regularization ka ek form kaise hai? Kam effective training steps ka matlab hai ki weights apne chhote random initialization se chhothi distance travel karte hain. Chhote weights ≈ ek simpler function. Toh training time limit karna model complexity ko limit karta hai — bilkul waise hi jaise ek $L_{2}$ penalty karta hai (isko hum neeche prove karte hain).

Mechanism, step by step

Yeh kaise run karta hai (algorithm):

Data ko train / validation mein split karo (validation gradients ko kabhi touch nahi karta).
Har epoch ke baad, validation loss $L_{val}$ compute karo.
Agar $L_{val}$ improve hua → yeh weights save karo (yeh "best" checkpoint hai) aur ek counter reset karo.
Agar yeh improve nahi hua → counter increment karo.
Jab counter $=$ patience $p$ → stop, aur saved best weights restore karo.

Derive karna — early stopping ≈ $L_{2}$ regularization kyun hai

Hum first principles se dikhate hain ki early stopping (approximately) weight decay ke equivalent hai. Yahi iska "deep" regularization karne ka reason hai.

Gradient descent update. Learning rate $ε$ aur gradient $\nabla J = H (w - w^{*})$ ke saath: $w^{(t)} - w^{*} = (I - ε H) (w^{(t - 1)} - w^{*})$ Kyun? Gradient ko $w^{(t)} = w^{(t - 1)} - ε \nabla J (w^{(t - 1)})$ mein substitute karo aur $w^{*}$ subtract karo.

Diagonalize karo. $H = Q Λ Q^{⊤}$ likho eigenvalues $λ_{i}$ ke saath. Eigen-coordinates $[Q^{⊤} w]_{i}$ mein, $w^{(0)} = 0$ se shuru karke, $t$ steps unroll karne par milta hai: $[Q^{⊤} w^{(t)}]_{i} = (1 - (1 - ε λ_{i})^{t}) [Q^{⊤} w^{*}]_{i}$ Kyun? Har eigen-direction independently $(1 - ε λ_{i})$ factor se shrink hoti hai har step mein; geometric series.

Worked examples

Practical knobs

Knob	WHAT control karta hai	Galat hone par failure
Patience $p$	kitni der tak bina improvement ke tolerate karein	bahut chhota → jaldi rok (underfit); bahut bada → wasted compute, mild overfit
min-delta	minimum change jo "improvement" mani jaaye	bahut bada → bahut jaldi rok deta hai
Monitored metric	val loss vs val accuracy	accuracy flatter/noisier hai; loss usually smoother
Restore best weights	best checkpoint par roll back karo	off hone par, last (worse) weights rakh lete hain

Flashcards

Validation-loss curve ki kya shape hoti hai jo early stopping exploit karta hai?

U-shape: yeh decrease hoti hai, ek minimum tak pahunchti hai (best generalization), phir model ke overfit hone par rise kati hai.

Validation set gradient updates ko kabhi influence kyun nahi karna chahiye?

Taaki yeh generalization ka unbiased estimate bana rahe; agar yeh training drive karti, toh iska loss bhi optimistically biased ho jaata.

Early stopping mein "patience" define karo.

Consecutive epochs ki woh sankhya jisme koi validation improvement nahi hoti, jise hum stop karne se pehle tolerate karte hain.

Best checkpoint restore kyun karein, last weights kyun nahi?

Minimum ke baad ke last epochs zyada overfit hote hain; best checkpoint mein sabse kam validation loss tha = best generalization.

Quadratic analysis mein

1/ α

(inverse weight-decay) ka role kya play karta hai?

Training steps ki sankhya

t

— zyada steps = kamzor effective regularization = zyada bade effective weights.

Early stopping kaun si eigen-directions rakhta hai aur kaun si suppress karta hai?

High-curvature (bade

λ_{i}

) directions rakhta hai jo jaldi fit hoti hain; low-curvature (chhote

λ_{i}

) directions suppress karta hai jo

t

steps mein barely move karti hain.

Validation loss ki pehli uptick par rukna galat kyun hai?

Validation loss noisy hoti hai; ek akela blip real improvement ke baad aa sakta hai. Patience

> 1

use karo.

Early stopping aur

L_{2}

ke beech matching condition kya hai?

(1 - ε λ_{i})^{t} \approx \frac{α}{λ _{i} + α}

Recall Feynman: ek 12-saal-ke bachche ko samjhao

Socho tum test ki tayari kar rahe ho. Pehle pehle, padhna help karta hai — tum ideas samajhte ho. Lekin agar tum raat bhar exact practice questions ratta lete rehte ho, toh tum un specific questions ko memorize karne lagte ho ideas ki jagah, aur real (different) test par worse karte ho. Early stopping ek dost ki tarah hai jo tumhare practice-test scores dekh raha hai aur tumhare kandhe par thapki deta hai: "Tum peak par the — ab ruko, real exam do." Hum tumhare best brain-state ka ek photo (checkpoint) bhi rakhte hain aur woh tumhe wapas dete hain.

Connections

Overfitting and Generalization — early stopping train/test gap ka ilaaj hai.
L2 Regularization (Weight Decay) — quadratic loss ke under early stopping ke provably near-equivalent.
Gradient Descent — woh update rule jiska trajectory hum truncate karte hain.
Hessian and Curvature — eigenvalues $λ_{i}$ per-direction shrinkage decide karte hain.
Validation and Cross-Validation — woh signal provide karta hai jo early stopping monitor karta hai.
Learning Rate Schedules — interact karta hai: $ε λ_{i}$ set karta hai ki har direction kitni tezi se seekhi jaati hai.