Exercises — Learning curves analysis

Step 1 (WHAT). Compute the gap: $\text{Gap}=51-48=3$. Step 2 (WHY). Our rule says: small gap and both values high (48–51 is far above a zero-ish floor) and plateaued. That is the signature of the left panel in the figure above. Diagnosis: high bias (underfitting). Fix: add model capacity — higher polynomial degree, more features, or less regularization. Do not collect more data.

Gap $=19-1=18$ — large. Training error near the floor. Validation still falling. This is the right panel: high variance (overfitting). Fix: more training data, more regularization, or lower model complexity. Because the orange curve is still dropping, more data will help here (unlike L1.1).

Gap $=1$ (small), and both values are low and plateaued. Neither "high & together" (bias) nor "low with big gap" (variance): this is the good fit case. Ship it, or push complexity only if $8$ is unacceptable for the task.

WHY this formula. The term ${\sigma }^2/m$ is a finite-sample bonus: with very few examples the model can partly fit the noise, so its measured training error sits below the true floor ${ϵ}_{bias}$ by an amount ${\sigma }^2/m$. Wait — it sits above: the formula adds ${\sigma }^2/m$, so at small $m$ the measured training error is inflated and it decreases toward the floor ${ϵ}_{bias}$ as $m$ grows.

• $m=10:40+600/10=40+60=100$.
• $m=50:40+600/50=40+12=52$.
• $m\to \infty :40+0=40$. Direction check. As written, $J_{train}(m)={ϵ}_{bias}+{\sigma }^2/m$ is monotonically decreasing in $m$: the numbers fall $100\to 52\to 40$, and $J_{train}$ approaches ${ϵ}_{bias}$ from above. (The parent note describes the empirical high-bias curve rising because it uses a much smaller ${\sigma }^2$ and plots the region where the rise is visible; the algebra of this exact formula with these numbers decreases — trust the algebra.) The limit is what matters here: $\boxed{\underset{m\to \infty }{\lim }{J}_{train}={ϵ}_{bias}=40}$.

WHY. For a complex model, training error already hits the irreducible floor ${ϵ}_{irr}$, so the entire gap is the variance penalty ${ϵ}_{var}k/m$. $\text{Gap}(m)=J_{val}-J_{train}=\left({ϵ}_{irr}+{ϵ}_{var}\frac{k}{m}\right)-{ϵ}_{irr}={ϵ}_{var}\frac{k}{m}.$

• $m=100:1\cdot 1000/100=10$.
• $m=500:1\cdot 1000/500=2$. The gap shrinks $10\to 2$ as data grows fivefold — concrete proof that more data helps a high-variance model.

Set $\frac{1000}{m}=1\Rightarrow m=1000$. Interpretation for sample-size planning: halving the gap requires doubling the data — diminishing returns baked into the $1/m$ law.

Step 1 — training curve. Rises slowly $0\to 8$: stays near the floor. Complex model memorises easily. Step 2 — validation curve. Falls fast $80\to 28$ and is still dropping at $m=100$ (last step $35\to 28$ is a big $7$-unit drop). Step 3 — gap at the end. $28-8=20$: large. Diagnosis: high variance. Because the orange curve is still descending, collecting more data is a valid fix (unlike a bias problem). Also viable: raise regularization or lower degree.

Final gap $=18-17=1$ (small). Both curves flattened around $17$–$19$ and both sit low (well under a zero-ish floor's danger zone — a gap of $1$ is negligible). Diagnosis: good fit. Training rose to meet validation, both plateaued, tiny gap. This is exactly the convergence we want.

WHY convert. Our whole framework is written in errors (lower = worse). Error $=100\%-\text{accuracy}$.

• At 500: $J_{train}=1\%$, $J_{val}=35\%$, gap $=34\%$. Training near floor + huge gap $\Rightarrow$ high variance.
• At 5000: $J_{train}=5\%$, $J_{val}=12\%$, gap $=7\%$. Reading the trend. More data cut the gap $34\%\to 7\%$ and raised validation accuracy $65\%\to 88\%$. That downward-moving validation error is the fingerprint of a variance problem that data is fixing. Remaining $7\%$ gap = mild overfitting; add light regularization if you need more.

Model it. High bias means $J_{train},J_{val}\to {ϵ}_{bias}\approx 50$–$52\%$ regardless of $m$ — the ${\sigma }^2/m$ correction is already negligible once curves plateaued. Prediction after 100× data: $J_{train}\approx 50\%$, $J_{val}\approx 52\%$. Essentially unchanged. Verdict: the month is wasted. The bottleneck is model capacity, not data. Spend the effort on higher complexity / better features instead. This is the parent note's headline warning: "more data always helps" is false for high bias.

Model A: gap $=28\%$, train near floor $\Rightarrow$ high variance. Best fix without new data: stronger regularization / lower complexity. Model B: gap $=2\%$, both moderately high $\Rightarrow$ high bias. Best fix: more capacity / features. Deploy today: compare the honest number, $J_{val}$: A gives $30\%$ error, B gives $20\%$. Deploy B — it generalises better right now even though its training score looks worse. Never pick a model on training error; pick on validation error.

$k=\text{Gap}\cdot m=28\times 1000=28000$. Solve $\frac{28000}{m}=10\Rightarrow m=2800$. So $\approx 2800$ examples (2.8× current) gets the gap to $10\%$ — often realistic. But to reach gap $=2\%$ you'd need $m=14000$ (14× data): the $1/m$ law makes the last bit expensive. Combine data with regularization to reach it cheaper.

Minimum experiment. Plot the learning curve once on your current data: train the model on increasing fractions ($10\%,25\%,50\%,75\%,100\%$ of what you have) and record $J_{train}$ and $J_{val}$ at each. This single plot reveals both height and gap — no extra data needed yet. Decision logic:

flowchart TD
  A["Plot learning curve on current data"] --> B{"Both errors high and gap small"}
  B -->|yes| C["High bias: double capacity"]
  B -->|no| D{"Train low and gap large and val still dropping"}
  D -->|yes| E["High variance: collect 10x data"]
  D -->|no| F{"Gap large but val flat"}
  F -->|yes| G["High variance saturated: add regularization"]
  F -->|no| H["Good fit: ship it"]

Why this order. We test bias first (cheapest to rule out — no data needed), then split the variance case by whether the validation curve is still falling: if it is, data pays off; if it has flattened, extra data won't and regularization is the lever.

Condition. Estimate the local slope $\text{slope}=\frac{J_{val}(m_1)-J_{val}(m_2)}{m_2-m_1}(\text{errors per example, positive = improving}).$ "Worth it" if the projected gain over your next data batch $\Delta m$ meets your threshold $T$: $\text{slope}\times \Delta m\ge T.$ Apply. slope $=\frac{35-28}{100-70}=\frac{7}{30}\approx 0.233$ units/example. Over $\Delta m=30$ more examples the projected gain is $\frac{7}{30}\times 30=7\ge T=2$. Yes — collect more data. The validation curve is far from flat, so the investment clears the bar comfortably.

(a) Depth 1: gap $=0$, both high $\Rightarrow$ high bias (tree too shallow to split the classes). Depth 20: gap $=33\%$, train near floor $\Rightarrow$ high variance (tree memorises). Depth directly controls capacity, so it slides us along the bias–variance axis. (b) As depth increases $1\to 20$, bias falls (validation improves from $40\%$) but variance rises (gap opens). Validation error $=\text{bias}+\text{variance}$ therefore first decreases then increases — a U-shape — so its minimum lies at some intermediate depth, strictly below both endpoints' $J_{val}$ (both endpoints are $\ge 35\%$, and the U-minimum is below the higher shoulder). To find it: sweep depth $\in \{1,2,4,8,16,20\}$, and for each plot a learning curve; pick the depth whose validation error is lowest and whose gap is acceptably small.

Height of both curves tells you which failure mode ::: bias (high = high bias) The gap between the curves tells you which failure mode ::: variance (large = high variance) High bias, so more data will... ::: not help — add capacity/features instead High variance with validation still dropping, so more data will... ::: help — the gap keeps shrinking The correct error to rank and choose models on ::: validation error, never training error Why the $1/m$ gap law means diminishing returns ::: halving the gap requires doubling the data since gap $\times m$ is constant The curve you sweep complexity against to see the U-shape ::: the validation curve (training error only ever falls)