Question bank — Early stopping

False. Training loss keeps falling (the model can always fit its own data better); only validation loss makes the U-turn. That widening gap is exactly overfitting.

True. It regularizes implicitly by limiting how far weights travel from their small initialization; fewer steps = smaller effective weights = simpler function, mirroring L2 Regularization (Weight Decay).

False. They are only approximately equivalent, and only under a quadratic approximation of the loss near a minimum. The match is $(1-\epsilon {\lambda }_i{)}^t\approx \frac{\alpha }{{\lambda }_i+\alpha }$, not an exact identity.

False, it's the reverse. More steps play the role of $1/\alpha$ growing, so $\alpha$ (the effective decay) shrinks toward zero — more steps means weaker regularization and larger weights.

False. Validation loss is noisy (mini-batch stochasticity, small val set), so a single uptick is often a blip. That's why we wait $p>1$ epochs before trusting the rise.

False. Effective regularization also depends on the learning rate $\epsilon$ and the curvature ${\lambda }_i$ per direction. The matching condition involves the product $\epsilon {\lambda }_i$ and the step count $t$ together, not $t$ alone.

False. Accuracy is flatter and noisier (it counts discrete correct/wrong), so it can plateau or jitter differently, triggering stops at other epochs than a smooth loss would.

False, it's the low-curvature ones. Large-${\lambda }_i$ directions fit almost fully and fast; small-${\lambda }_i$ directions barely move in $t$ steps and stay near their small init — that's what gets suppressed.

False. We selected the stopping epoch using validation loss, so it's no longer unbiased — it became part of the optimization. You need a separate untouched test set (see Validation and Cross-Validation).

Error: deploy the best checkpoint, not the last. Epochs after the minimum are more overfit; you keep the lowest-validation-loss weights, which came earlier.

Error: stopping on the first rise ignores noise. With patience $p=3$ you'd have kept going and often found a better minimum at epoch 5.

Error: the equivalence depends on the learning rate — $\epsilon$ sits inside $(1-\epsilon {\lambda }_i{)}^t$. A bad $\epsilon$ breaks the approximation and can destabilize Gradient Descent entirely.

Error: a tiny val set makes $L_{\text{val}}$ so jittery it can swamp real improvements, causing premature stops or missed minima. Enlarge the set or raise patience.

Error: too-large min-delta treats genuine small gains as "no improvement," so the counter fills up early and you stop too soon (underfit).

Error: past the minimum, every extra epoch drifts toward overfitting, so "last" is systematically worse than "best," not equivalent.

Error: the proof assumes the loss looks like a bowl near a single minimum $w^{\ast }$. Far from a minimum, or with many minima, the clean $\frac{\alpha }{{\lambda }_i+\alpha }$ correspondence need not hold.

Broad patterns reduce loss the most and fastest — they lie along high-curvature directions the optimizer descends first. Noise-fitting is what's left once the easy gains are used up.

If it drove the weights, its loss would become optimistically biased and stop measuring generalization — it would just become more training data with no honest test signal left.

Fewer steps means weights stay closer to their small random init, and small weights encode a smoother, simpler function. Time budget becomes a complexity budget.

A single epoch's validation loss is a noisy sample; patience averages out blips by requiring a streak of no improvement before believing the peak has passed.

In the matched formula, large $t$ drives $(1-\epsilon {\lambda }_i{)}^t\to 0$, which forces $\frac{\alpha }{{\lambda }_i+\alpha }\to 0$, i.e. $\alpha \to 0$. Small $t$ does the opposite. So $t$ and $\alpha$ move inversely.

A decaying learning rate slows how fast each eigen-direction fits, so the same stopping epoch corresponds to a different effective regularization. The schedule reshapes what "stop at step $t$" means.

Because the weights themselves are what you deploy; knowing "epoch 5 was best" is useless if those exact parameters were overwritten by later, worse epochs.

Early stopping never triggers; you either train to the budget's end or need more epochs. The U-shape's right arm simply hasn't appeared yet — you may be underfitting still.

The best checkpoint stays at epoch 1 and every later epoch fails to beat it, so you'll stop after $p$ epochs and restore epoch-1 weights — a sign the model overfits almost immediately (too flexible or too little data).

The shrink factor $(1-\epsilon \cdot 0{)}^t=1$, so $[Q^{\top }{w}^{(t)}{]}_i$ stays at $0$ forever — that direction is fully suppressed, matching infinitely strong decay along a flat valley.

Then that eigen-direction grows instead of shrinking — gradient descent diverges there and the neat regularization interpretation collapses. Stability needs $0<\epsilon {\lambda }_i<2$.

No. At the shared $t$, the steep (${\lambda }_i$ large) direction is nearly fully fit while the flat one is barely moved, so early stopping applies different effective decay per direction — just like L2 Regularization (Weight Decay) does.

Not cleanly by this method — you'd need cross-validation folds (see Validation and Cross-Validation) to estimate the U-curve, since without held-out data there's no honest signal for when generalization peaks.