Underfitting vs overfitting diagnosis: Worked examples

This page is the drill hall. The parent note gave you the ideas; here we grind through every kind of number you can be handed and turn it into a diagnosis + an action. Before any symbol appears we say what it means, in plain words.

The scenario matrix

Every diagnosis problem you will ever meet is one of the cells below. The examples that follow are labelled with the cell they cover, so you can check none is missing.

#	Cell (case class)	What makes it special	Covered by
C1	High bias, low variance	big avoidable bias, tiny gap	Ex 1
C2	Low bias, high variance	tiny avoidable bias, big gap	Ex 2
C3	High bias AND high variance	both gaps big (worst case)	Ex 3
C4	Just right	both gaps small	Ex 4
C5	Degenerate: $J_{t r ain} = 0$	perfect memorisation, zero training error	Ex 5
C6	Sign trap: $J_{c v} < J_{t r ain}$	negative "variance" — what does it mean?	Ex 6
C7	*Limiting: $ε^{}$ unknown / assumed 0**	wrong baseline flips the diagnosis	Ex 7
C8	Real-world word problem	you must extract the numbers yourself	Ex 8
C9	Exam twist: iteration sequence	diagnosis changes as you act	Ex 9
C10	Learning-curve shape read	diagnose from a curve, not a table	Ex 10 (figure)

The worked examples

Worked example Ex 1 — Cell C1: High bias, low variance

A spam filter reports training error 24%, validation error 26%. Human checkers make $ε^{*} = 3%$ errors.

Forecast: guess the diagnosis before reading on. Both errors high, both close — which disease?

Compute avoidable bias $= J_{t r ain} - ε^{*} = 24% - 3% = 21%$ . Why this step? It tells us how much of the error is pure model weakness, above the finish line.
Compute variance $= J_{c v} - J_{t r ain} = 26% - 24% = 2%$ . Why this step? The small gap says the model behaves almost the same on new data — it is not memorising.
Compare: $21% ≫ 2%$ , so bias dominates. Diagnosis: underfitting (high bias). Why this step? The 80/20 rule — attack the bigger number first.
Action: raise capacity — richer features, bigger model, train longer. See 3.2.01-Neural-network-capacity.

Verify: avoidable bias + variance $= 21% + 2% = 23%$ , and total avoidable error $= J_{c v} - ε^{*} = 26% - 3% = 23%$ . ✓ The two pieces add back to the whole.

Worked example Ex 3 — Cell C3: High bias AND high variance (the double whammy)

A model: training error 15%, validation error 30%, $ε^{*} = 2%$ .

Forecast: which single word — underfit or overfit? (Trap: it's neither alone.)

Avoidable bias $= 15% - 2% = 13%$ . Why? Big — the model is weak even on practice.
Variance $= 30% - 15% = 15%$ . Why? Also big — and it still falls apart on new data.
Both large and comparable $\Rightarrow$ both diseases at once. Why this step? Real models can be simultaneously too weak in some regions and too wiggly in others.
Action: this needs both cures in sequence — first raise capacity (kills bias), then regularize (kills variance). Do not do both blindly at once, or you'll cancel your own progress.

Verify: total avoidable $= 30% - 2% = 28% = 13% + 15%$ . ✓ Neither component is negligible.

Worked example Ex 5 — Cell C5 (degenerate):

$J_{t r ain} = 0$ A degree-15 polynomial on 10 points: training error 0, validation error 156, and the true noise floor $ε^{*} \approx 1$ .

Forecast: zero training error — is that good?

Avoidable bias $= 0 - 1 = - 1$ . Why the negative? Training error dropped below the true noise floor. That is impossible for honest generalisation — it means the model fit the noise itself, not the signal. Interpretation: a negative avoidable bias is a red flag, not a bonus. Clamp it: effective bias $= 0$ .
Variance $= 156 - 0 = 156$ . Why? Enormous — the classic overfitting fingerprint.
Diagnosis: extreme overfitting. Zero training error is a symptom, never a goal.
Action: fewer parameters, more data, regularization. As data grows (see 2.6.03-Learning-curves-interpretation) the model is forced to abandon the noise.

Verify: total avoidable $= J_{c v} - ε^{*} = 156 - 1 = 155$ , and clamped-bias $+$ variance $= 0 + 156 = 156$ ; the $1$ discrepancy is exactly the model dipping below the noise floor. ✓ (The mismatch itself is the tell-tale.)

Worked example Ex 6 — Cell C6 (sign trap):

$J_{c v} < J_{t r ain}$ A run reports training error 12%, validation error 9%, $ε^{*} = 4%$ .

Forecast: the quiz score beat the practice score. Bug, or luck?

Variance $= J_{c v} - J_{t r ain} = 9% - 12% = - 3%$ . Why this step? Negative variance is not physical for a fixed model — the surprise set should be at least as hard as practice on average.
Diagnose the cause, not the model. Common reasons:
- the validation set is tiny (a lucky small sample), or
- leakage — easy examples leaked into validation, or
- dropout/augmentation was left ON during training (raising $J_{t r ain}$ artificially) but OFF at validation. Why this step? A negative gap is a process signal: fix the measurement before trusting any diagnosis.
Once fixed, re-measure. Only the bias part is meaningful now: $12% - 4% = 8%$ suggests mild underfitting.

Verify: treat variance as $max (0, J_{c v} - J_{t r ain}) = max (0, - 3%) = 0%$ . Avoidable bias $8%$ then dominates $\Rightarrow$ bias-side action once the split is trustworthy. ✓

Worked example Ex 7 — Cell C7 (limiting / wrong baseline): assuming

$ε^{*} = 0$ Same numbers as Ex 1 (train 24%, val 26%), but a beginner assumes $ε^{*} = 0$ instead of the true $3%$ .

Forecast: does the wrong finish line change the diagnosis?

With $ε^{*} = 0$ : avoidable bias $= 24% - 0 = 24%$ ; variance $= 2%$ . Why this step? Only the bias number moves; variance never uses $ε^{*}$ .
Compare with the correct $ε^{*} = 3%$ (Ex 1): bias was $21%$ . Why this step? The direction of the diagnosis (high bias) is unchanged — bias still dominates.
When does the baseline flip the answer? Only when true $ε^{*}$ is large. E.g. train $6%$ , val $8%$ : with $ε^{*} = 0$ bias looks like $6%$ (attack bias!), but with true $ε^{*} = 5%$ bias is only $1%$ (attack variance!). That is why the finish line matters.

Verify: wrong-baseline bias $24%$ vs correct $21%$ differ by exactly the assumed baseline error $3%$ . ✓ And the flip case: $6% - 0 = 6% > 2%$ variance, but $6% - 5% = 1% < 2%$ variance — opposite winners. ✓

Worked example Ex 8 — Cell C8 (real-world word problem)

"Our recommender got 8,200 of 10,000 training clicks right and 7,100 of 10,000 validation clicks right. Marketing analysts hit a ceiling around 1 in 10 wrong. Where do we invest?"

Forecast: extract three numbers, then diagnose.

$J_{t r ain} = 1 - 8200/10000 = 18%$ ; $J_{c v} = 1 - 7100/10000 = 29%$ ; $ε^{*} = 10%$ . Why this step? Word problems hide the numbers in prose — convert to error fractions first.
Avoidable bias $= 18% - 10% = 8%$ . Variance $= 29% - 18% = 11%$ . Why this step? Now the two rulers apply exactly as before.
Variance ( $11%$ ) $>$ bias ( $8%$ ), both substantial $\Rightarrow$ overfitting is the bigger leak, with real bias too.
Invest order: regularize / more data first (biggest win), keep an eye on the $8%$ bias after.

Verify: total avoidable $= 29% - 10% = 19% = 8% + 11%$ . ✓

Worked example Ex 9 — Cell C9 (exam twist: the diagnosis moves)

An exam gives three iterations and asks "what changed?"

Iter A: train $15%$ , val $18%$ , $ε^{*} = 2%$
Iter B (bigger model): train $4%$ , val $12%$
Iter C (added dropout + augmentation): train $3%$ , val $4%$

Forecast: track which gap shrinks at each step.

Iter A: bias $= 13%$ , variance $= 3%$ $\Rightarrow$ underfitting. Why? Bias $≫$ variance.
Iter B: bias $= 2%$ , variance $= 8%$ $\Rightarrow$ now overfitting. Why? Growing capacity killed bias but exposed variance — the classic swap.
Iter C: bias $= 1%$ , variance $= 1%$ $\Rightarrow$ balanced. Why? Regularization and augmentation closed the gap without re-inflating bias.
Answer to the exam: capacity fixed bias, regularization fixed variance — you attack one disease per move.

Verify: totals $= 16%, 10%, 2%$ respectively, each equal to bias+variance ( $13 + 3$ , $2 + 8$ , $1 + 1$ ). ✓ Monotone drop confirms real progress.

Worked example Ex 10 — Cell C10 (read the learning curve, not a table)

Below are three sketched learning curves ( $J$ vs training-set size $m$ ). Diagnose each from shape alone.

Forecast: for each panel, ask: do the two lines meet, and how high?

Left (high bias): both curves rise/settle to a high plateau, touching. Why? More data can't help a model too weak to fit even the training set — gap stays tiny, level stays high.
Middle (high variance): training stays low, validation stays high, a wide gap that closes only slowly. Why? The model nails training points but needs lots of data before it's forced to generalise.
Right (good fit): both converge to a low value with a small gap. Why? Capacity matches the problem; extra data gives diminishing returns near $ε^{*}$ .

Verify: the vertical gap at large $m$ equals the variance ruler — wide in the middle panel, thin on the sides — matching the tables in Ex 1–4. ✓

Recall Self-test

A model has train error 5%, val error 20%, human error 4%. Diagnosis? ::: Bias $= 1%$ , variance $= 15%$ $\Rightarrow$ overfitting (high variance). Why is $J_{t r ain} = 0$ a warning sign, not a trophy? ::: It usually means the model memorised noise (dipped below the noise floor $ε^{*}$ ), giving huge variance. $J_{c v} < J_{t r ain}$ — first thing to suspect? ::: A measurement problem: tiny/leaky validation set, or dropout left on at training but off at validation. Which ruler ignores $ε^{*}$ entirely? ::: Variance $= J_{c v} - J_{t r ain}$ — it never uses the baseline.

See also: 2.6.04-Cross-validation-techniques for trustworthy $J_{c v}$ , and the Hinglish version.