Learning curves analysis: Question bank — AI-ML deep dive

Quick vocabulary refresher (all built in the parent note):

Training error $J_{t r ain} (m)$ — the model's average mistake on the very examples it was trained on, using $m$ of them.
Validation error $J_{v a l} (m)$ — its average mistake on held-out examples it never trained on.
Learning curve — both of those plotted against the training set size $m$ .
High bias = underfitting (too simple). High variance = overfitting (too complex).
Gap $= J_{v a l} (m) - J_{t r ain} (m)$ .
Bias floor $ϵ_{bia s}$ — the error a model cannot beat because it's too simple to represent the pattern, no matter how much data you give it.
Irreducible noise $ϵ_{i r r e d u c ib l e}$ — random error baked into the data itself (measurement noise, unpredictable factors); no model can go below it.
Variance term $ϵ_{v a r ian ce}$ — the extra validation error that comes from the model fitting quirks of a particular training set; it shrinks as $m$ grows.
Complexity constant $k$ — a number that grows with how flexible/complex the model is; a bigger $k$ means the variance term $ϵ_{v a r ian ce} \cdot k / m$ fades more slowly with data.

Related builders: Train-Test Split, Cross-Validation, Regularization, Overfitting Detection, Model Complexity, Sample Size Determination.

True or false — justify

As training set size

m

grows, training error always decreases.

False. Training error typically rises with

m

— a handful of points is easy to fit perfectly, but more diverse points can't all be satisfied, so average training error climbs toward the model's true bias floor.

m

grows, validation error always decreases.

False — the word "always" breaks it. Validation error usually falls, but it flattens once the model has learned all it can, and it can even rise under distribution shift or a leaky split, so it does not always decrease.

If training and validation error both plateau at a high value with a tiny gap, the fix is more data.

False. A small gap at high error is the signature of high bias — the model can't represent the pattern, so more data just adds more points it also can't fit. Increase complexity instead.

A large, persistent gap between low training error and high validation error means high bias.

False. That is the opposite — a big gap with near-zero training error is high variance (overfitting); the model memorized the training set but failed to generalize.

If both curves converge to a low value, the model is well-fit.

True. Low converged error with a small gap means the model's complexity matches the data's complexity — it learned the real pattern without memorizing noise.

For a high-variance model, collecting more data can help.

True. The variance term shrinks roughly like

ϵ_{v a r ian ce} \cdot k / m

, so as

m

grows the variance penalty fades and validation error drops toward the training error.

Zero training error is proof of a good model.

False. Zero training error most often means the model memorized the data — that's the classic overfitting warning sign, not evidence of generalization. Check the validation curve.

Learning curves can be drawn from a single train/validation split.

True, but noisily. You can plot them from one split; using Cross-Validation to average over folds gives far smoother, more trustworthy curves.

If validation error is still visibly decreasing at your largest

m

, you have probably not collected enough data.

True. A validation curve that hasn't plateaued means the model is still learning from each new batch — more data is likely to keep helping.

Regularization moves you along the bias–variance spectrum.

True. More Regularization simplifies the effective model (raises bias, lowers variance); less regularization does the reverse. Learning curves let you see which direction you need.

Spot the error

"My training error is 2% and validation error is 35%, so I should make the model more complex."

Error: a huge gap with tiny training error is high variance. Adding complexity makes it worse. Reduce complexity or regularize, or gather more data.

"Both my errors sit at 48% and won't move; I'll collect 10× more data."

Error: both high and plateaued = high bias. More data keeps them at ~48%. You need more model capacity, more features, or less regularization — not more rows.

"Training error went up when I added data, so my model is getting worse."

Error: rising training error is expected and healthy — you can no longer memorize the larger set. Judge quality by validation error and the gap, not by training error rising.

"The gap between the curves is small, so there's no overfitting problem to worry about."

Error: a small gap only rules out variance. If both errors are high, you still have a problem — high bias — and the small gap is actually its signature.

"I'll just look at final training accuracy to pick between two models."

Error: training accuracy alone hides overfitting. Two models can both hit 99% training accuracy while differing wildly in validation accuracy. Compare validation curves and gaps.

"My validation error is below my training error, so something is broken."

Not necessarily an error — this can legitimately happen with small validation sets, easier validation splits, or regularization/dropout applied only during training. It's worth investigating, but it isn't automatically a bug.

Why questions

Why does training error increase with more data instead of decreasing?

With few points a model fits them exactly (even memorizes), giving near-zero error. Adding varied points introduces demands it can't all satisfy, so its average training mistake grows toward its true bias floor.

Why does more data not help a high-bias model?

The error floor is set by the model's limited capacity (the bias floor

ϵ_{bia s}

), not by data scarcity. Feeding it more examples of a pattern it structurally cannot represent leaves that floor untouched.

Why do high-bias curves have a small gap while high-variance curves have a large gap?

High bias makes the model equally wrong on training and validation data (systematic errors everywhere → small gap). High variance makes it right on memorized training data but wrong on new data (→ large gap).

Why do we plot error against training set size at all, rather than just reading one final score?

A single score can't distinguish "too simple" from "too complex" — both can look bad. The shape over

m

(plateau height + gap + whether it's still falling) reveals which problem you have and whether data will help. See Sample Size Determination.

Why can two models with the same final validation error still call for different next steps?

One might have both curves plateaued (data won't help → change the model), the other might still be falling (more data will help). The trajectory, not just the endpoint, dictates the action.

Why does the variance gap shrink like

k / m

rather than jumping straight to zero?

Each extra example only marginally reduces how much the model can chase noise; the averaging improves gradually with

m

, so the gap decays smoothly, and slowly when the complexity constant

k

is large.

Edge cases

m = 1

(a single training example), what does training error look like?

Essentially zero — any reasonable model can fit one point perfectly. This is the extreme left end where memorization is trivial, which is exactly why the training curve starts low and rises.

What if both training and validation error are already near zero at small

m

The task is easy relative to the model, or the model is well-matched and data-efficient — a genuinely good fit. There's little to gain from more data or more complexity here.

What happens to the curves as

m \to \infty

for a high-bias model?

Both converge to roughly the same high value near the bias floor:

J_{t r ain} \to ϵ_{bia s}

and

J_{v a l} \to ϵ_{bia s}

, since the variance term

ϵ_{v a r ian ce} \cdot k / m \to 0

. They meet high because capacity, not data, is the bottleneck.

What happens to the curves as

m \to \infty

for a high-variance model?

Training error stays near the irreducible noise floor

ϵ_{i r r e d u c ib l e}

, while validation error descends toward it as the variance term

ϵ_{v a r ian ce} \cdot k / m

vanishes — so the gap closes and an overfitting model can turn into a good fit.

If your validation set is tiny, how does that distort the learning curve?

J_{v a l}

becomes noisy and unreliable — apparent gaps may be measurement noise, not real variance. Use a larger held-out set or Cross-Validation before trusting the diagnosis.

What does it mean if adding data makes validation error rise?

Usually a red flag about your data pipeline: distribution shift between old and new data, a leaky or mislabeled batch, or a train/validation split that isn't independent. Genuine learning curves shouldn't trend upward on validation.

What if the training and validation curves cross each other?

With the usual orientation (training rising, validation falling) a crossing is unusual and hints at a very easy validation split, heavy regularization, or too little data to trust — inspect the split before drawing bias/variance conclusions.

Recall Self-check before you move on

Can you state, in one sentence each, the curve shape for high bias vs high variance vs good fit — and the correct next action for each? High bias ::: Both errors high, small gap, plateaued → add complexity / features, reduce regularization; more data won't help. High variance ::: Low training error, large gap, validation still falling → more data, less complexity, or more regularization. Good fit ::: Both errors low, small gap, both plateaued → you're done; stop tuning.