Underfitting vs overfitting diagnosis: Visual walkthrough

This page rebuilds the whole diagnosis idea from the ground up — one picture at a time. By the end you will be able to look at two numbers and say "underfitting" or "overfitting" and know why. Parent: Underfitting vs overfitting diagnosis.

We assume you have seen nothing. Every symbol below is earned before it is used.

Step 1 — What is a "model", really? A curve trying to pass through dots

WHAT. We plot data as points and a model as a curve.

WHY. Before any error, bias, or variance can mean anything, you must see that machine learning is "draw a curve near the dots." Everything later is a statement about how well the curve hugs the dots.

PICTURE. The black dots are the data. The cyan curve is one candidate model.

Figure — Underfitting vs overfitting diagnosis

Here $h$ is chosen once and for all to name "the curve." $h (x)$ means "the curve's height at horizontal position $x$ ."

Step 2 — Measuring "how far off": the error

WHAT. For each dot we draw a vertical stick from the dot to the curve, then average the squared stick-lengths.

WHY. We need ONE number that says "how bad is this curve?" A single average lets us compare curves. We square (rather than take absolute value) because squaring is smooth — later tools that adjust the curve rely on that smoothness — and because it punishes one huge miss more than many tiny ones.

PICTURE. Amber sticks are the misses. Long sticks = bad fit.

The letter $J$ is just a traditional name for "cost." Nothing mysterious.

Step 3 — Two different dot-piles: train and test

WHAT. Split the dots into two groups. Fit the curve using only the left group; measure error on both.

WHY. The only score that matters in the real world is performance on data you have not seen — that is generalization. Training error alone can be faked by memorising. We need both numbers to diagnose anything.

PICTURE. Left panel = training dots the curve was fitted to. Right panel = fresh test dots the same curve now faces.

Everything from here is: where do these two numbers sit, and how far apart are they?

Step 4 — Underfitting: a curve too stiff to bend

WHAT. Fit a straight line to curved data. Read off both errors.

WHY. We want the fingerprint of a too-simple model. Because the line is wrong everywhere in the same way, it misses the training dots AND the test dots by roughly the same large amount.

PICTURE. The stiff cyan line leaves long amber sticks on both train and test — high error, small gap.

Step 5 — Overfitting: a curve so wobbly it traces the noise

WHAT. Fit a very wiggly curve to the same training dots, then confront it with fresh test dots.

WHY. We want the opposite fingerprint. The curve fits noise, which is different in every pile, so training looks perfect while test blows up — a huge gap.

PICTURE. Left: the wiggly cyan curve threads every training dot (tiny sticks). Right: the same frozen curve badly misses fresh dots (long amber sticks).

Step 6 — The Goldilocks curve: capacity that matches the pattern

WHAT. Fit a curve whose flexibility matches the true shape.

WHY. This is the target of the whole diagnosis. Both errors are low; the gap is small at a low height.

PICTURE. The cyan curve follows the hill's spine; short sticks on both piles.

Step 7 — The degenerate edges you must not forget

WHAT. Walk the three degenerate scenarios.

WHY. The contract: cover every case, including the broken ones, so no reading surprises you.

PICTURE. Three mini-panels of the traps below.

Edge A — one training dot ( $m = 1$ ). Any curve can pass exactly through one point, so $J_{train} = 0$ automatically. This is not good fit; the number is meaningless. Need enough dots before trusting $J_{train}$ .
Edge B — empty test pile ( $m_{test} = 0$ ). The gap is undefined (dividing by zero examples). You cannot diagnose generalization with no fresh data — see cross-validation to reuse data safely.
Edge C — pure noise, no pattern. If $y$ is random with no relationship to $x$ , the best possible $ϵ$ is high. Then high $J_{train}$ with small gap can look like underfitting, but no model can do better. Compare to the noise floor $ϵ$ , never to zero.

The one-picture summary

Everything compresses into one map: height (how big the errors are) versus gap (how far train and test sit apart). Each corner is a diagnosis.

Recall Feynman retelling — say it back in plain words

We started with dots on paper and a curve trying to hug them. To score a curve we measured how tall each miss is, squared it so misses always count as bad, and averaged — that single number is $J$ . Then we split the dots into a practice pile and a fresh exam pile, because only the exam tells us if we actually learned. A too-stiff curve (a ruler) misses everything the same way: both scores bad, tiny gap — that's underfitting. A too-wobbly curve threads every practice dot including the random noise: practice perfect, exam awful, huge gap — that's overfitting. The right curve bends just as much as the true pattern: both scores low and close. Watch out for tricks — one lonely dot gives free zero error, an empty exam pile gives no gap at all, and pure noise sets a high floor $ϵ$ you must measure against instead of zero. So the entire diagnosis is two glances: how high? and how far apart?

Recall

Underfitting fingerprint ::: high $J_{train}$ , small gap (both errors high and equal) Overfitting fingerprint ::: low $J_{train}$ , large gap ( $J_{test} ≫ J_{train}$ ) Why square the miss instead of absolute value ::: squaring removes the sign, is smooth for optimisation, and punishes big misses harder What must you compare errors against, not zero ::: the irreducible noise floor $ϵ$ Why $m = 1$ gives a misleading $J_{train} = 0$ ::: any curve passes exactly through a single point, so zero training error is automatic and meaningless

Next: watch these errors move as data grows in learning curves interpretation.