Learning curves analysis: Visual walkthrough — AI-ML deep dive

This page rebuilds the two curves of a learning curve from absolute zero. We will not assume you know what "error", "bias", or "variance" mean as numbers. We will watch them appear, one picture at a time, until the famous shapes — one rising, one falling — fall out on their own.

Step 1 — Draw the two things we are measuring

WHAT. A model is a machine that takes an input $x$ and guesses an output. Call its guess $h (x)$ (read " $h$ of $x$ " — the letter $h$ stands for hypothesis, our fancy word for "the model's current guess-rule"). The true answer is $y$ .

WHY. Before we can plot error, we need a single number that says "how wrong was this one guess?". That number is called a loss, written $L$ . We will use the most common one, squared error — you take the gap between guess and truth and square it:

$L (model’s guess h (x), true answer y) = (h (x) - y)^{2}$

$h (x) - y$ ::: the plain gap (stick length), positive if we guessed too high, negative if too low
$(\dots)^{2}$ ::: squaring — kills the sign (a stick below counts the same as one above) and punishes big misses harder

If the guess equals the truth, the gap is $0$ so $L = 0$ . Why squaring and not, say, absolute value? Because the whole variance story later (Step 5) is about averaging squared gaps — and the tidy " $σ^{2} / m$ " law only holds for squares. Squaring is the tool that makes noise add up cleanly.

PICTURE. Below, the cyan dots are true answers; the amber line is the model's guess-rule. Each vertical stick is one gap $h (x) - y$ ; the loss $L$ is that stick's length squared.

Step 2 — Two different sets: the one it studied and the one it hasn't seen

WHAT. We split our data into two piles (this is the Train-Test Split idea). The training set is what the model studies from. The validation set is a separate pile we never let it study — we only use it to test.

WHY. A student who has memorised the exam answers looks perfect on those exact questions but may know nothing. To catch that, you must test on fresh questions. Two piles ⇒ two errors:

$J_{t r ain} = error on the pile it studied J_{v a l} = error on the fresh pile$

PICTURE. Left box = training pile (the model bent itself to fit these). Right box = validation pile (never touched during learning).

Step 3 — Slide the data slider: start with almost no data

WHAT. Now the key move. We let $m$ (the training-set size) grow, and re-train the model at each size. Start tiny: $m = 2$ or $3$ points.

WHY. A learning curve is a graph versus $m$ . So $m$ is our slider. We must understand the two extremes — tiny $m$ and huge $m$ — because the curve just connects them.

PICTURE. With only 3 points and plenty of knobs, the model can wiggle right through every one. Zero sticks ⇒ $J_{t r ain} \approx 0$ . But that same wiggly rule is nonsense elsewhere, so on the fresh box its sticks are huge.

Step 4 — Slide to lots of data: the model can no longer please everyone

WHAT. Now $m$ is large — hundreds of points, far more than the model's knobs.

WHY. We want the right edge of the curve. With many points crowding in (more points than knobs), the model cannot thread through all of them at once; it must compromise.

PICTURE. Many cyan dots, one amber rule. It can't hit them all, so training sticks appear (training error rises from ~0). But because it now respects the real trend, fresh points land close too — validation sticks shrink.

Step 5 — Where do they level off? Named ingredients of error

WHAT. As $m$ keeps growing, both curves flatten toward limits. To predict where, we split every error into named parts, and we will use one consistent symbol per part for the rest of the page.

Definition The ingredients (fixed vocabulary — introduced here, used everywhere after)

$ϵ_{bia s}$ ::: the floor from being too simple — the wrongness the model still has even with unlimited data because its shape can't match the pattern. Depends on the model, not on $m$ .
$σ^{2}$ ::: the variance of the random noise on a single data point — how much one $y$ jiggles above/below its true value, measured as an average squared gap. This is the raw, per-point noise number.
$ϵ_{i r r}$ ::: the irreducible error — the floor of wrongness no model can ever beat, because the target itself is noisy. For squared-error loss these two are the same number: $ϵ_{i r r} = σ^{2}$ . We keep two names only for their two roles — $σ^{2}$ is "noise on one point going in", $ϵ_{i r r}$ is "unbeatable error floor coming out" — but numerically $ϵ_{i r r} \equiv σ^{2}$ .
$k$ ::: the number of free knobs the model can turn (its complexity). More knobs = more flexible shape. (First met informally in Step 3; here it gets its symbol.)

WHY these parts. Any wrongness is either (1) a mismatch the model can never fix ( $ϵ_{bia s}$ ), (2) noise nobody can fix ( $ϵ_{i r r} = σ^{2}$ ), or (3) the model being thrown off by the particular noisy sample it happened to see — an amount built from $σ^{2}$ and $k$ . Only part (3) depends on how much data you have — so only part (3) moves as the slider slides.

PICTURE. A stacked bar: the two bottom blocks (bias + irreducible) are fixed height; the top block (the sample-jiggle penalty) melts down as the arrow "more data" pushes right.

Earning the $\frac{σ ^{2}}{m}$ law — watch the jiggle cancel

WHY a picture, not a sentence. We keep claiming "averaging noise cancels it as $σ^{2} / m$ ". That is the hinge of the whole page, so we show it rather than assert it.

WHAT. Take one point's true value and add random noise of typical size $σ$ (so its variance, the average squared wobble, is $σ^{2}$ ). Now take $m$ such points and average them. The average's own wobble is smaller — and by exactly how much is the law.

Why it works, step by term. Each noisy point is (truth) + (its own jiggle). Averaging $m$ of them:

$average = truth + leftover noise \frac{1}{m} i = 1 \sum m (jiggle_{i})$

$jiggle_{i}$ ::: the random wobble on point $i$ , variance $σ^{2}$ each, independent
$\frac{1}{m} \sum$ ::: we average the jiggles too

When you add $m$ independent wobbles, their squared sizes add (the ups and downs partly cancel), giving total variance $m σ^{2}$ ; the $\frac{1}{m}$ out front squares to $\frac{1}{m ^{2}}$ :

$variance of the average = \frac{1}{m ^{2}} \cdot (m σ^{2}) = \frac{σ ^{2}}{m}$

$\frac{1}{m ^{2}}$ ::: comes from the $\frac{1}{m}$ being squared (variance squares constants)
$m σ^{2}$ ::: $m$ independent variances add up
result $\frac{σ ^{2}}{m}$ ::: twice the data → half the leftover jiggle

So the piece of error caused by "the sample's noise leaking into the fitted knobs" must carry a factor $\frac{1}{m}$ . That is why the moving term is $\propto \frac{1}{m}$ and not $\frac{1}{m ^{2}}$ or constant — it is literally this variance-of-an-average law. The squared-error loss from Step 1 is what let us add variances so cleanly.

Formula The plateau formulas, every symbol earned

Each knob is one more thing the sample noise can wiggle, so the leaked-noise term is (jiggle per knob) × (number of knobs) $= \frac{σ ^{2}}{m} \cdot k = \frac{σ ^{2} k}{m}$ .

Training error (measured on the same points the knobs were tuned on): $J_{t r ain} (m) \approx too-simple floor ϵ_{bia s} + noise floor = σ^{2} ϵ_{i r r} - σ^{2} = per-pt noise k = knobs, / m = averaging fits own noise, looks too good \frac{σ ^{2} k}{m}$ On the training pile the model has bent toward the sample noise, so it looks better than truth by $\frac{σ ^{2} k}{m}$ → the term is subtracted, and it starts near $0$ for tiny $m$ then rises as the discount shrinks. Validation error (measured on fresh points the knobs never saw): $J_{v a l} (m) \approx same floor ϵ_{bia s} + = σ^{2} ϵ_{i r r} + same leaked noise, now hurts \frac{σ ^{2} k}{m}$ On fresh points the fitted-in noise hurts instead of helping, so the same $\frac{σ ^{2} k}{m}$ is added.

Here the tool "divide by $m$ " was chosen deliberately: it is the variance-of-an-average law made visible. Bias and irreducible noise have no $m$ — that is the whole reason more data can't cure them, only the $\frac{σ ^{2} k}{m}$ term.

Step 6 — Case A: high bias (the model is too simple)

WHAT. Make $ϵ_{bia s}$ large (e.g. a straight-line model on a curvy pattern), with few knobs (small $k$ ).

WHY. We must cover every case the reader could meet. This is the "both errors high" case.

PICTURE. Both curves climb/fall and kiss at a high level. The gap between them is tiny because both share the same big $ϵ_{bia s} + ϵ_{i r r}$ floor.

$lim_{m \to \infty} J_{t r ain} = ϵ_{bia s} + ϵ_{i r r}, lim_{m \to \infty} J_{v a l} = ϵ_{bia s} + ϵ_{i r r} + gone \frac{σ ^{2} k}{m} \to 0 \Rightarrow J_{v a l} \to ϵ_{bia s} + ϵ_{i r r}$

Step 7 — Case B: high variance (the model is too complex)

WHAT. Make $k$ large (a very wiggly model with little data).

WHY. The opposite failure: this is the "big gap" case, and the one where the gap is your alarm bell.

PICTURE. Training curve sits near the floor $ϵ_{i r r}$ (bias is tiny for a flexible model); validation curve floats far above it. The vertical amber gap is exactly $\frac{2 σ ^{2} k}{m}$ — the sum of the discount below and the penalty above.

$Gap (m) = J_{v a l} (m) - J_{t r ain} (m) = (\frac{σ ^{2} k}{m}) - (- \frac{σ ^{2} k}{m}) = \frac{2 σ ^{2} k}{m}$

$σ^{2} k$ ::: total leaked sample-noise (per-point jiggle × number of knobs)
$\frac{1}{m}$ ::: the shrink-with-data factor
factor $2$ ::: the same leaked noise counts twice — a discount on train plus a penalty on val

Step 8 — The degenerate edges (never leave a scenario unshown)

WHAT & WHY. Three corner cases the formulas must survive:

$m < k$ from the start (too few points to pin the knobs). With fewer points than free knobs, the model is under-determined — many rules fit equally, so no unique fit and no single error to plot. The curve must start at $m \geq$ (enough constraints). This is the flip side of Step 3's interpolation regime: there $k \geq m$ let us hit every point; here $m$ is simply too small to define a fit at all. At $m = 0$ the average $\frac{1}{m}$ is literally division by zero.
Perfect noiseless data, right-sized model. Then $ϵ_{bia s} = 0$ and $σ^{2} = 0$ (so $ϵ_{i r r} = σ^{2} = 0$ ): every term vanishes and both curves plunge to $0$ and meet. The "good fit" ideal.
$m \to \infty$ . Every $\frac{1}{m}$ term vanishes; both curves flatten. What's left is $ϵ_{bia s} + ϵ_{i r r}$ — the too-simple floor plus the world's own noise that no data ever removes.

PICTURE. Three mini-panels: the forbidden left region $m < k$ , the ideal converge-to-zero, and the far-right flatline at $ϵ_{bia s} + ϵ_{i r r}$ .

The one-picture summary

Everything above, on one axis: rising red $J_{t r ain}$ , falling cyan $J_{v a l}$ , the amber gap $\frac{2 σ ^{2} k}{m}$ between them, the $ϵ_{bia s} + ϵ_{i r r}$ floor as a dashed line, and the three diagnoses labelled where the curves land. The two annotations on the figure are the reason the shapes exist: train is pulled down by fitting its own noise (the $- \frac{σ ^{2} k}{m}$ discount), val is pushed up by that same noise ( $+ \frac{σ ^{2} k}{m}$ ), and both discounts melt as $\frac{1}{m} \to 0$ .

Recall Feynman retelling — say it like a story

We had a machine that guesses, and we measured its wrongness with squared error: the gap between guess and truth, squared, then averaged over the points — that average is the error. We kept two piles: one it studied (train) and one it never saw (val). Then we slid a slider — the amount of study material.

With almost no material (and more knobs than points, so it could thread through them all), the machine just memorised the few examples, so it looked perfect on them (train error near zero) but was clueless on fresh ones (val error high). As we poured in more material — soon more points than knobs — it could no longer memorise everyone, so its train error crept up; but now it actually grasped the real shape, so val error slid down. Two curves, one rising, one falling.

Where they settle comes from a few ingredients. Bias ( $ϵ_{bia s}$ ) is the machine being too dumb to ever fit the shape. Irreducible noise ( $ϵ_{i r r}$ , which for squared error is just the per-point noise variance $σ^{2}$ ) is jiggle in the world nobody can predict. Both are fixed floors with no $m$ — so more data can't lift you off them. The moving part is the machine fitting the particular noise of this sample: each of its $k$ knobs catches some of the per-point jiggle $σ^{2}$ , and averaging over $m$ points cancels that leaked noise down to $\frac{σ ^{2} k}{m}$ — a fact we watched cancel: $m$ independent wobbles add to $m σ^{2}$ , the $\frac{1}{m}$ average squares to $\frac{1}{m ^{2}}$ , product $\frac{σ ^{2}}{m}$ . On its own points this leaked noise flatters the machine (a discount, subtracted); on fresh points the same leaked noise hurts (a penalty, added). That mirror image is the whole gap, $\frac{2 σ ^{2} k}{m}$ — big when $k$ is big (high variance, close it with more data or Regularization), tiny when $k$ is small (high bias, where only more capacity helps). Read where the two curves land, and the picture tells you exactly which cure to reach for.

Recall

A learning curve plots what against what? ::: Training error and validation error (y) versus training-set size $m$ (x). What loss are we using, and why squared? ::: Squared error $(h (x) - y)^{2}$ ; squaring kills the sign and lets independent noise variances add cleanly, which is what makes the $σ^{2} / m$ law hold. Why does training error rise with more data? ::: The "fits its own noise" discount $\frac{σ ^{2} k}{m}$ shrinks as $m$ grows, so train error stops looking artificially good and climbs toward the true floor. Which error terms have no $\frac{1}{m}$ , and what does that imply? ::: $ϵ_{bia s}$ and $ϵ_{i r r} = σ^{2}$ — no $\frac{1}{m}$ means more data can never remove them. How do $ϵ_{i r r}$ and $σ^{2}$ relate? ::: For squared-error loss they are the same number ( $ϵ_{i r r} = σ^{2}$ ); two names for two roles — noise going in vs unbeatable floor coming out. Where does the $\frac{1}{m}$ in the variance term come from? ::: $m$ independent wobbles add to variance $m σ^{2}$ ; the $\frac{1}{m}$ averaging squares to $\frac{1}{m ^{2}}$ ; product is $\frac{σ ^{2}}{m}$ — data averages the jiggle away. Where does the factor $k$ come from and why is it in both curves? ::: $k$ = number of free knobs; each catches sample noise, so leaked noise is $σ^{2} k / m$ , subtracted on train and added on val (same size, opposite sign). What must be true for "tiny $m$ ⇒ zero training error"? ::: The model must have at least as many knobs as points ( $k \geq m$ ) so it can interpolate them all. A large gap still shrinking at the right edge: diagnosis and fix? ::: High variance; collect more data or increase regularization.