Learning curves analysis: Foundations — AI-ML deep dive

Before you can read a learning curve, you must be able to read every mark on it. This page builds the whole alphabet from nothing. Nothing here assumes you have seen a formula before. We collect the pieces in order — each one leans on the one before it.

1. What is a "model"? (the guessing machine)

In the figure, follow the flow left to right: the input $x$ enters as the labelled arrow on the left (a house size). The black box is the rule $h$ . The red dial inside the box is $θ$ — turning that one red dial changes the guess that leaves on the right arrow. So the picture is literally the symbols: arrow-in $= x$ , box $= h$ , red dial $= θ$ , arrow-out $= h_{θ} (x)$ . Learning means finding the red-dial position that makes the output arrow least wrong.

Why does the topic need this? Because a learning curve is a story about one model being retrained on more and more data. If you don't picture $h_{θ}$ as an adjustable machine, the rest is meaningless.

See Model Complexity for what makes a model "simple" or "complicated" — that idea returns in Section 6.

2. The true answer, the guess, and the gap between them

The gap between $y$ and $\overset{y}{^}$ for a single example is what we are about to measure. A perfect guess has zero gap; a bad guess has a big gap.

3. Loss $L$ — measuring one mistake

In the figure, the black line is the model ( $\overset{y}{^} = h_{θ} (x)$ for every $x$ ). Each black dot is one true answer $y^{(i)}$ . The red vertical stick from a dot up to the line is that example's error $y^{(i)} - \overset{y}{^}^{(i)}$ — its length is exactly what $L$ squares.

Why this tool and not just "count the wrong ones"? For predicting a number (like price), there is no simple right/wrong — you are always a little off. Loss measures how much off, on a smooth scale, so the model can tell whether a knob-turn helped a little or a lot.

4. Averaging the mistakes: the cost $J$ and the sum $\sum$

We have a loss for each example. To score the model overall, we average all of them. That average is the cost $J$ .

Why divide by $m$ ? So the number stays comparable no matter how many examples you have. A total of 200 across 100 houses is not worse than 100 across 40 houses — the average (2 vs 2.5) is the fair comparison. This matters enormously here because the whole point of a learning curve is to change $m$ and watch $J$ .

5. Two datasets: training vs validation

In the figure, the left black box holds the $m$ training dots the model practices on; the right red box holds the $m_{v a l}$ validation dots it is quizzed on but never studies. The two boxes never share a dot — that separation is the whole point. See Train-Test Split and Cross-Validation for how this split is done reliably.

Why two costs and not one? Because the difference between "how well you do on studied material" and "how well you do on the quiz" is the entire diagnosis. A model can ace the practice sheet and flunk the exam — that gap is overfitting.

6. The horizontal axis: $m$ as a variable, not a fixed number

The single deepest idea for reading learning curves: we retrain the model at many different training-set sizes and plot the results.

Look back at Sample Size Determination — deciding how big $m$ needs to be is exactly the question a learning curve answers visually.

7. What happens as $m$ grows forever: the limit $lim$

In the figure, the horizontal axis is $m$ and the black curve is a cost $J (m)$ ; as you move right (more data) it flattens toward the dashed red line, which marks the limit — the height it never crosses. The whole diagnosis in the parent note ("both errors plateau high" vs "gap persists") is a statement about what these limits are and how far apart they sit.

Why do we need limits? Because a learning curve is only a few dots, but the question — "would even more data help?" — is about behaviour far to the right, past the data you have. The limit is the mathematical way to reason about "off the edge of the chart."

8. The named error pieces: bias, variance, noise

The parent note splits total error into named ingredients. Here is what each is, in plain words.

Definition The three error ingredients (and

$σ$ , $k$ , $c$ )

$ϵ_{bia s}$ (the Greek "epsilon" means "a small error amount") — the error from the model being too simple to capture the real pattern. Present even with infinite data. Big for a straight line trying to fit a curve. See Model Complexity.
$ϵ_{v a r ian ce}$ — the error from the model being too twitchy, reacting to the random quirks of which examples it happened to see. This is the extra error on fresh data. Shrinks as $m$ grows.
$σ$ (Greek "sigma") measures the spread of the random noise in the data. Formally, if $ϵ$ is the unpredictable wobble in each true answer, then $σ^{2} = Var [ϵ]$ — the variance of that noise, i.e. its average squared deviation. This $σ^{2}$ is the irreducible error: the floor no model can beat, no matter how much data or how clever the model.
$k$ is a number that grows with model complexity (how many knobs / how flexible the model is). A flexible high-degree model has large $k$ ; a simple line has small $k$ . See Model Complexity.
$c$ is a positive constant (a fixed number that does not depend on $m$ ). It just sets the size of the small-sample slack term below — you never compute it by hand; it is there so the shape " $\propto 1/ m$ " is written honestly rather than pretending the slack is exactly $1/ m$ .

Now the parent note's two key approximations, with the why spelled out and the noise handled correctly:

Both extra terms shrink via the same "divide by $m$ " move you met in Section 4 — bigger $m$ makes them small. What they cannot remove is $ϵ_{bia s}$ (fixed by model choice) or $σ^{2}$ (fixed by the data's own noise).

Tools that change these ingredients: Regularization and Feature Engineering lower variance or bias; more data mainly attacks the $\frac{k}{m}$ gap term.

Prerequisite map

Read it bottom-right: every arrow is a "you need this first." The alphabet of Sections 1–8 all pours into the single topic node.

Equipment checklist

Cover the right side and see if you can answer each before revealing.

What does

h_{θ} (x)

mean in plain words?

A guessing machine that turns input

x

into a prediction, using adjustable knobs

θ

What is

θ

The set of internal numbers (knobs) the model tunes to make better guesses.

Does the superscript in

x^{(i)}

mean a power?

No — it labels example number $i$ . A power would have no brackets.

What is the explicit squared loss

L (\overset{y}{^}, y)

L (\overset{y}{^}, y) = (\overset{y}{^} - y)^{2}

— the gap between guess and truth, squared.

What does

\sum_{i = 1}^{m}

tell you to do?

Add the following quantity once for each example, from

i = 1

up to

i = m

What is a cost

J

, and what does its subscript tell you?

The average loss over a set of examples; the subscript says which set (train or validation).

Why does

J

divide by

m

To get the average loss per example, so the score is comparable across different dataset sizes.

What values can

m

take, and why never

0

m \in N

m \geq 1

— you need at least one example to average, so

\frac{1}{m}

is always defined.

Difference between

m

and

m_{v a l}

m

= number of examples the model studies (training);

m_{v a l}

= number in the held-out quiz set (validation).

Why can we write

J_{v a l} (m)

when we defined

J_{v a l} (m_{v a l})

The quiz set stays a fixed size; on a learning curve we vary the training size

m

and re-measure, so we relabel the argument to

m

What does the bare umbrella symbol

J (m)

stand for?

Either

J_{t r ain} (m)

J_{v a l} (m)

— whichever curve a statement applies to; not a separate cost.

What does

lim_{m \to \infty} J (m)

ask?

The value the cost

J

settles to as training data grows without bound — the plateau height.

Define

Gap (m)

Gap (m) = J_{v a l} (m) - J_{t r ain} (m)

— the vertical distance between the two curves.

What do

σ^{2}

k

, and

c

stand for?

σ^{2} = Var [ϵ]

= irreducible noise floor;

k

= grows with model complexity;

c

= a fixed positive constant sizing the

1/ m

small-sample slack.

What is the true floor of the training curve as

m \to \infty

ϵ_{bia s} + σ^{2}

— model-simplicity error plus irreducible noise, both unremovable by more data.

Bias vs variance in one line each?

Bias = error from being too simple (wrong every time); variance = error from being too twitchy (wrong depending on which data it saw).