Underfitting vs overfitting diagnosis: Exercises

This is a self-testing page. Each problem states a scenario cleanly, you attempt it, then open the collapsible solution. Problems climb from L1 Recognition (spot the failure mode) to L5 Mastery (design a full diagnostic loop).

Before you start, recall the three numbers we live and die by (all built in the parent note):

Definition The three error numbers

$ϵ^{*}$ ::: the best possible error (human-level or Bayes error) — the floor no model can beat.
$J_{t r ain}$ ::: the error your model makes on the data it was trained on.
$J_{c v}$ ::: the error on held-out cross-validation data it never saw during training.
$m$ ::: the number of training examples you feed the model (the x-axis of a learning curve).

From these we compute two "budgets":

$Avoidable Bias = max (J_{t r ain} - ϵ^{*}, 0) Variance = J_{c v} - J_{t r ain}$

The $max (\cdot, 0)$ clips avoidable bias at zero: a model can appear to beat human-level on its own training set (giving a negative raw value), but you can never owe less than zero bias, so we floor it. When the raw value is negative we simply read the bias budget as $0$ .

Bigger of the two budgets = the thing to fix first. See also 2.6.01-Bias-variance-tradeoff.

A quick picture of what the two failure modes look like on a learning curve — study it now, it drives every L1 answer below:

Figure — Underfitting vs overfitting diagnosis

Level 1 — Recognition

Exercise 1.1 — Read the two numbers

A model reports $J_{t r ain} = 24%$ and $J_{c v} = 26%$ . Human-level error is $ϵ^{*} = 3%$ . Is this underfitting, overfitting, or good fit?

Recall Solution 1.1

Compute both budgets.

Avoidable bias $= J_{t r ain} - ϵ^{*} = 24% - 3% = 21%$ — huge.
Variance $= J_{c v} - J_{t r ain} = 26% - 24% = 2%$ — tiny.

Both errors are high and the gap is small: the model can't even fit what it has seen. Answer: Underfitting (high bias). Fix: increase capacity — see 3.2.01-Neural-network-capacity.

Exercise 1.2 — The mirror case

Now $J_{t r ain} = 2%$ , $J_{c v} = 19%$ , $ϵ^{*} = 1%$ . Diagnose.

Recall Solution 1.2

Avoidable bias $= 2% - 1% = 1%$ — tiny.
Variance $= 19% - 2% = 17%$ — huge.

Training error is near-perfect but validation is far worse: a classic generalization gap. Answer: Overfitting (high variance). Fix: regularize (2.6.05-Regularization-methods), get more data, or reduce capacity.

Exercise 1.3 — Which curve is which?

On the figure above (learning curves), one panel shows two lines that plateau together at a high value; the other shows a low flat line and a high line that slowly sink toward each other. Label each panel.

Recall Solution 1.3

Both lines high, flat, close together (left panel) = underfitting. Adding data doesn't help — the model's shape is wrong, so more examples of the pattern it can't represent change nothing.
Low $J_{t r ain}$ , high $J_{c v}$ slowly converging (right panel) = overfitting. The big red-arrow gap is the variance; it shrinks as data constrains the model. This connects to 2.6.03-Learning-curves-interpretation.

Level 2 — Application

Exercise 2.1 — Fill the error budget table

Given $ϵ^{*} = 5%$ , $J_{t r ain} = 8%$ , $J_{c v} = 20%$ , compute avoidable bias, variance, and total avoidable error. Name the bottleneck.

Recall Solution 2.1

$Avoidable Bias = 8% - 5% = 3%$ $Variance = 20% - 8% = 12%$ $Total Avoidable Error = J_{c v} - ϵ^{*} = 20% - 5% = 15%$ Variance ( $12%$ ) $≫$ bias ( $3%$ ). Bottleneck: variance (overfitting). Note the two components sum to the total: $3% + 12% = 15%$ . ✔

Exercise 2.2 — Predict the action

A cat/dog CNN: training accuracy $99%$ , validation accuracy $76%$ , $ϵ^{*} = 4%$ error. What single action should you take first, and why?

Recall Solution 2.2

Convert accuracy → error: training error $= 1%$ , validation error $= 24%$ .

Avoidable bias (raw) $= 1% - 4% = - 3%$ , which we clip to $0$ (you're below human-level on train, so bias is not the problem).
Variance $= 24% - 1% = 23%$ — dominant.

Action: attack variance. First choice = get more data / augment (4.1.02-Data-augmentation-strategies) or add regularization (2.6.05-Regularization-methods), because reducing capacity would risk re-introducing bias. Do not add layers — capacity is already excessive.

Exercise 2.3 — The linear-on-quadratic experiment

You fit a straight line $h_{θ} (x) = θ_{0} + θ_{1} x$ to data generated by $y = 3 x^{2} + 2 x + 1 + noise$ . Here $m$ is the number of training examples used (the learning-curve x-axis). The best possible error on this data is the irreducible noise level $ϵ^{*} \approx 1%$ (about $1$ in the same error units). You measure:

m=10    J_train=8.2   J_cv=8.5
m=1000  J_train=8.0   J_cv=8.1

Will collecting 10× more data (larger $m$ ) help? Explain using the numbers.

Recall Solution 2.3

Make the bias budget explicit against the floor $ϵ^{*} \approx 1$ :

Avoidable bias at $m = 1000$ $= J_{t r ain} - ϵ^{*} = 8.0 - 1 = 7.0$ — huge.
Variance at $m = 1000$ $= J_{c v} - J_{t r ain} = 8.1 - 8.0 = 0.1$ — tiny.

So almost all the wasted error is bias, not variance, and the plateau ( $\approx 8$ ) barely moved from $m = 10$ to $m = 1000$ . That is the signature of high bias: the line simply cannot bend into a parabola. Answer: No. More data cannot fix a model whose shape is wrong — data shrinks variance, and here variance is already $\approx 0$ . You must raise capacity (add an $x^{2}$ term).

Level 3 — Analysis

Exercise 3.1 — Two iterations, shifting bottleneck

Speech system, $ϵ^{*} = 2%$ .

Iter 1:  J_train=15%  J_cv=18%
Iter 2:  J_train=4%   J_cv=12%

Between iterations the team increased model capacity. Compute both budgets each iteration and explain why the bottleneck moved.

Recall Solution 3.1

Iter 1: bias $= 15% - 2% = 13%$ ; variance $= 18% - 15% = 3%$ . → Bias-limited. Iter 2: bias $= 4% - 2% = 2%$ ; variance $= 12% - 4% = 8%$ . → Variance-limited.

Increasing capacity let the model finally fit the training data, collapsing bias $13% \to 2%$ . But the same extra flexibility made it memorize training quirks, so variance grew $3% \to 8%$ . This is the 2.6.01-Bias-variance-tradeoff in action: capacity trades bias for variance. Next move: regularize / augment.

Exercise 3.2 — Spot the inconsistent report

An intern reports: $ϵ^{*} = 5%$ , $J_{t r ain} = 3%$ , $J_{c v} = 9%$ . Your senior says "the training error looks impossible." Is it? What does $J_{t r ain} < ϵ^{*}$ mean?

Recall Solution 3.2

Raw avoidable bias $= 3% - 5% = - 2%$ , which our $max (\cdot, 0)$ rule clips to $0$ . A negative raw value means the model beats human-level on the training set. This is not impossible — it usually means one of:

The model has memorized training examples (very high capacity), or
$ϵ^{*}$ (human-level) was estimated on a harder/different distribution than the training set.

The honest reading: bias budget $= 0$ and variance $= 9% - 3% = 6%$ is the real story. The report isn't impossible — it's a flag that overfitting is likely and/or $ϵ^{*}$ is mis-estimated.

Exercise 3.3 — Learning-curve crossover

On a learning curve, $J_{t r ain}$ rises with more data while $J_{c v}$ falls. A student claims "the model is getting worse." Rebut using what each curve measures.

Recall Solution 3.3

$J_{t r ain}$ rising is healthy: with few points a flexible model fits them almost perfectly (low $J_{t r ain}$ , but that's memorization). As data grows, it can no longer thread every point, so $J_{t r ain}$ climbs toward the true error. Meanwhile $J_{c v}$ falls because the model is now forced to learn the general pattern. The gap shrinking = variance shrinking = model improving. The model is getting better, not worse. The convergence value they approach is roughly $ϵ^{*}$ .

Level 4 — Synthesis

Exercise 4.1 — Design the whole diagnostic loop

You inherit a model: $ϵ^{*} = 1%$ , $J_{t r ain} = 12%$ , $J_{c v} = 30%$ . Write the ordered action plan (which failure to fix first, second) with the numeric justification at each stage, and predict roughly what the budgets should look like after a successful bias fix.

Recall Solution 4.1

Now: bias $= 12% - 1% = 11%$ ; variance $= 30% - 12% = 18%$ . Both are large. Order of attack: fix bias first. Reasoning: a high-bias model is underpowered; regularizing it now (to fight variance) would make it even weaker. You must first give it capacity to fit, then control the overfitting that capacity unlocks. Step 1 — reduce bias (bigger model / better features / train longer): expect $J_{t r ain}$ to drop, say to $\sim 3%$ . Variance will likely grow as capacity rises. Predicted after bias fix: e.g. $J_{t r ain} \approx 3%$ , $J_{c v} \approx 20%$ → bias $= 2%$ , variance $= 17%$ . Now variance is the bottleneck. Step 2 — reduce variance: regularization (2.6.05-Regularization-methods), more data / augmentation (4.1.02-Data-augmentation-strategies), or cross-validated capacity tuning (2.6.04-Cross-validation-techniques). Goal state: both budgets small and comparable, e.g. bias $\approx 1%$ , variance $\approx 2%$ .

Exercise 4.2 — Reconcile two diagnostics

Learning curves show $J_{t r ain}$ and $J_{c v}$ plateauing together at a high value — that screams high bias. Yet the raw budgets are $ϵ^{*} = 2%$ , $J_{t r ain} = 9%$ , $J_{c v} = 11%$ . Do the two tools agree? Give the combined verdict.

Recall Solution 4.2

Budgets: bias $= 9% - 2% = 7%$ (large), variance $= 11% - 9% = 2%$ (small). Learning curves: both high, small gap, plateaued. Both tools agree: high bias / underfitting. The gap ( $2%$ ) is small in both views; the wasted error lives in the $7%$ avoidable bias. Consistent diagnosis → confident action: increase capacity, don't bother regularizing.

Level 5 — Mastery

Exercise 5.1 — Full four-iteration case study

A team logs four iterations ( $ϵ^{*} = 2%$ throughout):

Iter	Change made	$J_{t r ain}$	$J_{c v}$
1	baseline shallow net	20%	22%
2	deeper net	3%	17%
3	+ dropout + L2	5%	8%
4	+ 4× data augmentation	4%	5%

For each iteration compute bias and variance, name the bottleneck, and say whether the change that led into that row was the correct move. Then state when to stop.

Recall Solution 5.1

Iter	bias $= J_{t r ain} - ϵ^{*}$	variance $= J_{c v} - J_{t r ain}$	bottleneck
1	$18%$	$2%$	bias
2	$1%$	$14%$	variance
3	$3%$	$3%$	balanced
4	$2%$	$1%$	balanced, near floor

Iter 1→2 (deeper net): correct — bias was $18%$ , so adding capacity was right. It crushed bias ( $18 \to 1$ ) at the cost of variance ( $2 \to 14$ ).
Iter 2→3 (dropout+L2): correct — variance was the bottleneck at $14%$ ; regularization cut it to $3%$ . A tiny bias rise ( $1 \to 3$ ) is an acceptable trade.
Iter 3→4 (augmentation): correct — chipped variance $3 \to 1$ with bias steady. Good marginal gain.
Stop rule: at Iter 4, $J_{c v} = 5%$ vs floor $2%$ , and both budgets are $\leq 2%$ . Total avoidable error is only $3%$ and evenly split. Stop (or accept diminishing returns) — further gains need a better $ϵ^{*}$ estimate, new features, or truly novel data, not more of the same knobs.

Exercise 5.2 — Derive the "no-free-lunch" bound

Show algebraically that total avoidable error is exactly the sum of avoidable bias and variance (when the raw bias is non-negative), and explain why this means you cannot lower $J_{c v}$ toward $ϵ^{*}$ without shrinking at least one of the two budgets.

Recall Solution 5.2

By definition (with raw bias $J_{t r ain} - ϵ^{*} \geq 0$ ): $Avoidable Bias + Variance = (J_{t r ain} - ϵ^{*}) + (J_{c v} - J_{t r ain})$ The $J_{t r ain}$ terms cancel: $= J_{c v} - ϵ^{*} = Total Avoidable Error.$ Since the total is a fixed sum of the two budgets, the only way to reduce $J_{c v} - ϵ^{*}$ is to reduce bias, variance, or both. You cannot "hide" error — it is fully accounted for by these two channels plus the irreducible floor $ϵ^{*}$ . This identity is why the 2.6.01-Bias-variance-tradeoff is a genuine tradeoff: pushing one down often pushes the other up, and only their sum measures real progress.

Recall Self-check summary

First number to read on any report ::: $J_{t r ain}$ — high $J_{t r ain}$ means underfitting before anything else. Formula for avoidable bias ::: $max (J_{t r ain} - ϵ^{*}, 0)$ (clipped at zero) Formula for variance ::: $J_{c v} - J_{t r ain}$ These two budgets always sum to (when bias is non-negative) ::: total avoidable error $= J_{c v} - ϵ^{*}$ Small gap + high errors means ::: high bias (underfitting) Large gap means ::: high variance (overfitting) Fix a high-bias-AND-high-variance model in which order ::: bias first (add capacity), then variance (regularize/augment) On a learning curve the x-axis is ::: $m$ , the number of training examples