Visual walkthrough — ML project lifecycle

Parent: 5.3.1 ML project lifecycle · read the whole thing first if a term here feels new.

---

Step 1 — What is "error"? (start from a single tally)

Imagine we show the model $100$ examples and it gets some wrong. Define:

In the figure, $17$ of $100$ boxes are amber, so $\text{Error}=17/100=0.17=17\%$. That is the entire vocabulary we need to begin. One number.

---

Step 2 — The same model gives three different errors

Why bother with three numbers instead of one? Because the gaps between them tell us what to fix — that is the whole trick, and it is Step 3.

Recall

Why is $\text{DevErr}$ usually higher than $\text{TrainErr}$? ::: The model studied the training set, so it looks better there; the dev set is fresh, so it exposes what the model didn't truly learn.

---

Step 3 — Cutting the error into two named gaps

Put the three errors on a number line from $0\%$ up:

$\underset{\text{best achievable}}{\underbrace{\text{HLP}}}\le \text{TrainErr}\le \text{DevErr}$

Now define the two gaps, right where they sit on the ruler:

WHY this exact split and not some other? Because these are the only two places error can hide: either the model can't even fit what it saw (bias), or it fits what it saw but not the world (variance). No third bucket exists once HLP is your floor. This is the engine behind Error analysis and the Overfitting and generalization gap idea.

---

Step 4 — Worked numbers: which gap wins?

Take the parent's Example 2: $\text{HLP}=2\%$, $\text{TrainErr}=8\%$, $\text{DevErr}=10\%$.

$\text{avoidable bias}=8-2=6\%\text{variance}=10-8=2\%$

Bias ($6\%$) is three times variance ($2\%$). So we do not chase more data — we make the model stronger. This is the data-vs-model decision made visually: measure the brackets, attack the taller one.

Recall

If instead $\text{TrainErr}=3\%$ and $\text{DevErr}=11\%$, which lever? ::: Variance $=8\%$ dwarfs bias $=1\%$ — the model overfits, so gather more/cleaner data or regularize.

---

Step 5 — The degenerate case: when the ruler flips

$\text{HLP}\le \text{TrainErr}$ is not a law — it is only the usual case. A model can memorize noise and post training error below what a human achieves:

$\text{avoidable bias}=\underset{1\%}{\underbrace{\text{TrainErr}}}-\underset{2\%}{\underbrace{\text{HLP}}}=-1\%(\text{meaningless!})$

A negative avoidable bias is not "better than human" — it is the model cheating on the exam it studied for. When you see the bracket flip, stop reading it as bias and treat it as an overfitting alarm: all the real signal now lives in the variance gap. HLP is a floor on the best generalization error, never a strict floor on training error. See Human-level performance and Bayes error.

---

Step 6 — Add the axis nobody drew yet: time

Until now we measured error once. But a deployed model faces future data. Let $t$ be time (days since launch). Watch the live error $\text{Error}(t)$:

At launch the live curve sits near DevErr. Then it climbs. WHY does a fixed model get worse without any bug? Because the data feeding it is drifting away from what it trained on — Step 7.

---

Step 7 — Two ways the world can drift

First we need one symbol. $P(x)$ means "how likely each input $x$ is." $P(y\mid x)$ means "given input $x$, how likely the correct answer $y$ is." Read $\mid$ as "given."

This is the parent's Example 3: the word "covid" never appeared in training, so $P(x)$ shifted — data drift. The fix is not new code; it is new data. And "go get new data" means jumping back to an earlier stage — which finally forces the loop.

---

Step 8 — The rising curve forces the feedback arrow

Draw a horizontal alarm line at the worst error the business tolerates. The live curve rises (drift, Step 7) until it touches the line. Now what?

• Do nothing → the curve keeps rising → the product fails. Not allowed.
• Change code → useless, the code isn't wrong. It's the data that moved.
• Retrain on fresh data → the curve drops back down. This is the only move that works.

"Retrain on fresh data" = go back to Stage 2 (Data) or Stage 3 (Modeling), then redeploy. That backward jump, repeated every time the curve hits the line, is exactly the feedback arrow the parent drew. The sawtooth shape is the lifecycle. It is not a line because a line would only rise forever. Monitoring (see Model monitoring and observability) is what watches for the touch; safe redeploys use Deployment strategies (shadow, canary, blue-green); the automated retrain path is CI-CD for ML pipelines.

---

The one-picture summary

One figure compresses the whole walk: a single error number (Step 1) splits into three errors whose two gaps name the fix (Steps 2–4), the ruler can flip into overfitting (Step 5), then time turns the whole thing into a rising sawtooth (Steps 6–8) whose every tooth is one turn of the lifecycle loop.

Recall

Feynman retelling — say the whole page in plain words. ::: You start with one number: how often the model is wrong. You realise that number changes depending on which pile of data you measure — the data it studied, fresh data, or a human's score. The gaps between those three tell you what to fix: a big low gap means the model is too weak (bias), a big high gap means it memorized (variance). Weirdly, the "studied" score can dip below the human — that's not winning, that's cheating on the answer key, so you switch to watching the fresh-data score. Then you turn on the clock. Because the world keeps changing (new inputs, or new right answers), the model's live error slowly climbs even though nobody touched the code. It climbs until it hits the line you promised never to cross. The only cure is to go back, grab fresh data, retrain, and ship again — which knocks the error back down. Do that over and over and the error graph looks like saw teeth. Those saw teeth are the lifecycle loop. It's a circle, not a line, because the world never stops moving.