Learning curves analysis: Worked examples — AI-ML deep dive

This page is a case-by-case drill room for Learning curves analysis. The parent note taught you the two curves — training error and validation error — and the three shapes they make. Here we hunt down every situation those curves can be in and solve each one from scratch.

First, a reminder of the two quantities, because every example uses them and we must never use a symbol before it is earned.

The single thing we read off a learning curve is the gap and the height:

Before any example, we must earn the three symbols the parent note used in its error laws. Each is a plain-English quantity with a picture.

Definition The three error ingredients (define once, use everywhere)

$ϵ_{bia s}$ = the best error your model shape can ever reach, even with unlimited data. Picture a straight ruler trying to trace a curve: no matter how you slide it, it can't hug the bend — the leftover mismatch is $ϵ_{bia s}$ . Bigger $ϵ_{bia s}$ = model too simple.
$ϵ_{v a r ian ce}$ = the extra error from the model wobbling to fit the particular examples it happened to see (memorising quirks). Picture two different small samples giving two very different fitted curves — that jumpiness is $ϵ_{v a r ian ce}$ .
$σ^{2}$ = the noise variance: how much the labels $y$ scatter for reasons no model can explain (measurement error, randomness). Read $σ$ (sigma) as the typical size of that unavoidable scatter, and $σ^{2}$ as its square. This is the irreducible floor: even a perfect model cannot beat it.

The scenario matrix

Every learning-curve situation you will meet falls into one of these cells. Read it as a quadrant: the horizontal axis is the gap (variance ruler), the vertical axis is the height (bias ruler). The figure below draws that quadrant; the table lists the extra edge-cases (D–H) that live outside it.

#	Edge-case cell	Signature on the curve	What it demands
A	High bias (underfit)	both curves high, tiny gap, flat	more capacity (Model Complexity, Feature Engineering, less Regularization)
B	High variance (overfit)	train low, val high, big gap still closing	more data / more Regularization
C	Good fit	both low, gap small, both flat	ship it
D	Degenerate: $m$ tiny	train error $\approx 0$ , val error huge	don't trust it yet — too few points
E	Limiting case: $m \to \infty$	curves reach their floors	predict the floor before spending money
F	Sign twist: "more data" verdict	is the val curve still sloping down?	decide yes/no on collecting data (Sample Size Determination)
G	Real-world word problem	messy numbers, a business decision	translate to bias/variance, act
H	*Exam twist: bias and* variance both bad**	high height and large gap	fix bias first, then variance

We now walk every cell.

Cell A — High bias (underfitting)

Worked example A · Linear model on curvy data

A house-price model is linear. Errors are in $(\$ 1000)^2$.

m       : [10, 30, 50, 70, 100]
J_train : [45, 48, 50, 51, 52]
J_val   : [50, 52, 53, 53, 54]

Diagnose the cell, and predict $J_{v a l}$ if we collected 10 000 examples.

Forecast: guess before reading — do both curves sit high? Is the gap small? What happens with 100× more data?

Compute the gap at the largest $m$ . $Gap (100) = 54 - 52 = 2$ . Why this step? A small gap rules out variance — the model is not memorising, since it's almost as bad on its own homework as on the quiz.
Look at the height. $J_{v a l} (100) = 54$ , and it barely moved from $50$ at $m = 10$ . Why this step? A high, flat height is the fingerprint of a bad ceiling — the model simply cannot bend to the data.
Extrapolate the floor. Use the corrected training law $J_{t r ain} (m) \approx ϵ_{bia s} + σ^{2} (1 - c / m)$ . Because the $- c / m$ piece climbs to zero as $m$ grows, $J_{t r ain}$ rises from $45$ toward its plateau $ϵ_{bia s} + σ^{2} \approx 52$ — exactly what the numbers show ( $45 \to 52$ ). At $m = 10000$ the $1/ m$ piece is negligible, so both curves sit at their plateau and $J_{v a l} \approx 54$ still. Why this step? This is the whole point of the cell — collecting data changes nothing once you are on the plateau.

Verify: height $\approx 54$ (high), gap $= 2$ (tiny), both flat → Cell A, high bias. Predicted $J_{v a l} (10000) \approx 54$ . Fix: add polynomial features / reduce regularization, not more data.

Cell B — High variance (overfitting)

Worked example B · Degree-10 polynomial

Same house data, degree-10 model:

m       : [10, 30, 50, 70, 100]
J_train : [0.1, 2, 5, 8, 10]
J_val   : [80, 60, 45, 35, 28]

Diagnose, and estimate the gap at $m = 200$ using the parent's gap law.

Forecast: is the training error near zero? Is the val curve still diving? Guess whether more data helps.

Height and gap at $m = 100$ . $J_{t r ain} = 10$ (near-floor), $J_{v a l} = 28$ , $Gap (100) = 18$ . Why this step? Big gap + tiny train error = memorisation. The model aces homework, flunks quizzes.
Is the val curve flat? From $80 \to 28$ it is still falling steeply. Why this step? A still-falling val curve is the green light that more data will keep helping (unlike Cell A).
Fit the gap law $Gap (m) \approx ϵ_{v a r ian ce} \cdot k / m$ . Recall from the definitions: $ϵ_{v a r ian ce}$ is the wobble error and $k$ is the positive shape constant that says how fast the wobble fades with data. We can't split them apart from one measurement, so we lump them into a single fitted constant $K = ϵ_{v a r ian ce} k$ . At $m = 100$ , gap $= 18$ , so $K = 18 \times 100 = 1800$ . Then $Gap (200) \approx K /200 = 1800/200 = 9$ . Why this step? The parent gap law is $Gap (m) \propto 1/ m$ ; we calibrate its single constant $K$ on the point we have, then read off the future.

Verify: gap halves when $m$ doubles ( $18 \to 9$ ), consistent with $1/ m$ . Cell B, high variance. Fix: more data or more Regularization.

Cell C — Good fit

Worked example C · Degree-3 polynomial

m       : [10, 30, 50, 70, 100]
J_train : [5, 12, 15, 16, 17]
J_val   : [15, 18, 19, 18, 18]

Confirm this is the sweet spot.

Forecast: where do the two curves end up relative to each other?

Gap at largest $m$ : $18 - 17 = 1$ . Why? Tiny gap → not overfitting.
Height: $J_{v a l} = 18$ , far below the linear model's $54$ . Why? Low ceiling → not underfitting.
Flatness: both curves settle around $17$ – $19$ after $m = 50$ . Why? Convergence means the model learned the true pattern, not noise.

Verify: low height ( $18$ ) and small gap ( $1$ ) → Cell C, good fit. Ship it.

Cell D — Degenerate: $m$ is tiny

Forecast: can any model avoid $J_{t r ain} = 0$ when it has only 3 points to fit?

Why $J_{t r ain} = 0$ here is meaningless. With $m$ points and enough parameters, almost any flexible model passes exactly through all of them. Zero training error at $m = 3$ is expected, not diagnostic. Why this step? A degenerate input (too few points) makes the training curve uninformative — you cannot read bias/variance yet.
What to do. Slide $m$ up. Only once the curves have several points and start to flatten can you name a cell. Why this step? The whole diagnosis lives in the shape, and a shape needs more than one point.

Verify: at $m = 3$ the verdict is "inconclusive — gather more $m$ ", not "overfit". This is the boundary case every reader must know.

Cell E — Limiting case: $m \to \infty$

Forecast: which term survives when $m$ is enormous, the constant or the $1/ m$ part?

Take the limit of training error. As $m \to \infty$ , $\frac{12}{m} \to 0$ , so $J_{t r ain} \to 4$ . Why this step? The tool is the limit: it strips away every term that shrinks with data and leaves the irreducible floor $ϵ_{bia s} + σ^{2}$ . We use a limit — not just plugging a big number — because we want the exact asymptote, the value collecting infinite data can never beat.
Take the limit of validation error. $J_{v a l} = 4 + \frac{312}{m} \to 4$ . Why this step? Same limit; here the whole gap term $312/ m$ vanishes, so the curves meet at $4$ .
Read the gap in the limit. $Gap (m) = \frac{300}{m} \to 0$ . Why this step? A gap that limits to zero means the variance is fully erasable by data.

Verify: both floors $= 4$ ; gap $\to 0$ . So the irreducible floor is $4$ ; everything above it was variance that data erases. This example is a pure-variance model at its clean limiting behaviour — its own cell (E), showing how any curve's floor is found, not a relabelling of Cell B.

Cell F — Sign twist: does the val curve still slope down?

Forecast: the sign of the slope of the val curve is the whole answer — guess it for each.

Slope of P's val curve. $Δ J_{v a l} = 54 - 52 = + 2$ over the range → essentially flat / slightly up. Why this step? A flat or rising val curve means data has stopped paying off → this is Cell A, do not collect. Answer: No (fix bias).
Slope of Q's val curve. $Δ J_{v a l} = 28 - 80 = - 52$ , steeply down. Why this step? A still-falling val curve means each new example is still buying accuracy → Cell B, Yes, collect more (see Sample Size Determination).

Verify: P slope $\geq 0$ → No; Q slope $< 0$ → Yes. The sign of the val-curve slope is the switch.

Cell G — Real-world word problem

Forecast: did the gap shrink when data grew 10×? Extrapolate.

Convert to gaps (in accuracy points). At $500$ : gap $= 99 - 65 = 34$ . At $5000$ : gap $= 95 - 88 = 7$ . Why this step? Accuracy gap is the variance ruler; we watch whether $10 \times$ data closed it.
The gap fell $34 \to 7$ — huge improvement, classic Cell B being cured by data. Why this step? Confirms more data is the right lever (not just regularization).
Extrapolate val accuracy. Val went $65 \to 88$ ( $+ 23$ ) for $10 \times$ data. Diminishing returns mean the next $10 \times$ won't add another $23$ , but a gain past $88$ toward $\sim 90 - 92%$ is plausible while the gap is still $7 > 0$ . Why this step? A positive gap that is still closing justifies some more data; we temper the estimate because returns diminish.

Verify: gap $34 \to 7$ (closing), val still climbing → recommend collecting more data, but budget for diminishing returns; consider light Regularization to squeeze the last $7%$ gap cheaply.

Cell H — Exam twist: bias AND variance both bad

Forecast: if you only had one move, would you add capacity or add data/regularization?

Read both rulers. Height $= J_{v a l} = 55$ (high → bias present). Gap $= 55 - 30 = 25$ (large → variance present). Why this step? Both fingerprints appear together; this is the cell people misdiagnose.
Fix bias first. Even if you closed the entire gap by adding data, $J_{v a l}$ could only fall toward $J_{t r ain} = 30$ — still high. Why this step? The training floor ( $30$ ) is your best possible val error once the gap closes; if that floor is already too high, no amount of variance-fixing saves you. So lower the floor first (more Model Complexity / Feature Engineering), then attack the gap.
Order of operations. ① reduce bias → new lower $J_{t r ain}$ ; ② then add data / Regularization to close the remaining gap. Why this step? Fixing variance on a high-bias model wastes the data budget.

Verify: best achievable val after closing the gap $= J_{t r ain} = 30 >$ acceptable, so bias is the binding constraint → fix capacity first. Correct order: bias, then variance.

Recall Self-test — cover the right side

A flat, high pair of curves with a tiny gap is which cell? ::: Cell A, high bias — add capacity, not data. Training error near zero, big val gap still closing — which cell? ::: Cell B, high variance — more data / regularization. At $m = 3$ , $J_{t r ain} = 0$ — what's the verdict? ::: Inconclusive (Cell D): too few points to diagnose. The one number that answers "collect more data?" ::: the sign of the val-curve slope (still falling = yes). Height high AND gap large — fix which first? ::: bias first (lower the training floor), then variance. What does $σ^{2}$ mean in the error laws? ::: noise variance — the irreducible scatter in labels that no model can explain.