Worked examples — ML project lifecycle

Everything below uses only the tools built in the parent. If a symbol shows up, it was defined there (and we re-define it the first time it appears here, so you never have to scroll back).

---

The scenario matrix

Think of a "case class" as a distinct shape of situation. The three lifecycle tools each have their own edge cases. Here is the complete grid we will cover — every row gets at least one worked example, tagged by its cell ID.

Cell	Tool	Case class	The twist
EV+	Expected value	Payoff beats cost	Ordinary green-light
EV−	Expected value	Cost beats payoff	Red-light / kill it
EV0	Expected value	Break-even / degenerate	$p_{\text{success}}=0$ or exactly zero EV
BV-bias	Bias/variance	Bias $\gg$ variance	Attack the model
BV-var	Bias/variance	Variance $\gg$ bias	Attack the data
BV-neg	Bias/variance	TrainErr $<$ HLP	Overfitting alarm (bias negative)
BV-both	Bias/variance	Both gaps large	Which one first?
DRIFT-x	Monitoring	$P(x)$ moved	Data drift, fix in Stage 2
DRIFT-yx	Monitoring	$P(y\mid x)$ moved	Concept drift, relabel
WORD	Combined	Real-world word problem	Translate story → numbers
EXAM	Combined	Exam-style trap	Constraint / satisficing metric

The three quantities we lean on, restated in plain words. First the expected-value symbols, since two different families of variable appear on this page:

Two of the drift examples below route their fix to a specific lifecycle stage, so let us restate the stage numbers from the parent here so nothing is ambiguous:

Figure s01 — Bias/variance staircase. Three horizontal chalk rails stack the errors of one model: the HLP floor (white, 3%), TrainErr (blue, 8%), and DevErr (pink, 10%). The yellow double-arrows measure the two gaps — avoidable bias (floor → train) and variance (train → dev). The lesson the figure carries: your next move is decided by which yellow arrow is taller, not by any single bar.

Look at the chalk figure: HLP is the bottom rail. The avoidable bias is the gap from the rail up to the training bar; the variance is the extra gap from training up to dev. The whole game is which gap is taller.

---

Cell EV+ — the ordinary green-light

Steps.

1. Write the expected-value formula from the parent: $\mathbb{E}[\text{Value}]=p_{\text{success}}\cdot V_{\text{success}}-C_{\text{build}}-C_{\text{maintain}}.$ Why this step? It is the only number that prices feasibility ($p_{\text{success}}$) and the ongoing cost tail together; accuracy alone can't tell you to build.
2. Plug in: $0.7\times 200,000=140,000$. Why this step? $p_{\text{success}}\cdot V_{\text{success}}$ is the payoff weighted by the chance we actually collect it — this is the definition of expected value applied to the success outcome.
3. Subtract fixed costs: 140{,}000 - 50{,}000 - 40{,}000 = \50{,}000$. Why this step? Build and maintain are paid whether or not we succeed, so they are unweighted.

Answer: \mathbb{E}[\text{Value}] = \50{,}000 > 0$ → green-light in scoping.

Verify: Sanity check the magnitude — the payoff term alone ($140k) comfortably exceeds total cost ($90k), so a positive result is expected. Units: dollars throughout. ✓

---

Cell EV− — the red-light

Steps.

1. Formula, same as before. Why this step? Consistency: the decision rule doesn't change with the sign; only the inputs do.
2. Weighted payoff: $0.25\times 120,000=30,000$. Why this step? A quarter chance of a $120k win is worth only $30k in expectation — the low $p_{\text{success}}$ does the damage.
3. Subtract costs: 30{,}000 - 30{,}000 - 25{,}000 = -\25{,}000$.

Answer: \mathbb{E}[\text{Value}] = -\25{,}000 < 0$\to \ast \ast donotbuild\ast \ast (orfirstraise$p_{\text{success}}$ with a cheap prototype).

Verify: The weighted payoff ($30k) exactly cancels build cost, leaving maintenance as pure loss (−$25k). Matches. ✓

---

Cell EV0 — the degenerate break-even

Steps.

1. Case (a): $p_{\text{success}}=0$ kills the whole first term: 0\cdot V_{\text{success}} - 10{,}000 - 5{,}000 = -\15{,}000$. Why this step? When feasibility is zero, payoff cannot enter — you pay costs for a guaranteed failure. This is the mathematical form of "infeasible projects are pure loss."
2. Case (b): $0.5\times 100,000=50,000$; then 50{,}000 - 30{,}000 - 20{,}000 = \0$.\ast Whythisstep?\ast EVexactly$0$ is the indifference boundary — the formula says "no expected gain or loss." The tie-breaker becomes risk appetite and strategic value, not the number.

Answer: (a) -\15{,}000$(neverbuild);(b)$$0$ (business call, not a math call).

Verify: (a) $0\cdot V_{\text{success}}=0$ for every $V_{\text{success}}$, so answer is cost-only. (b) payoff term equals total cost → zero. ✓

---

Cell BV-bias — bias dominates, attack the model

Steps.

1. Avoidable bias $=\text{TrainErr}-\text{HLP}=11-3=8\%$. Why this step? This gap measures how far you are from what a human floor allows — a big value means the model is underfitting the training set itself. See Bias-variance decomposition.
2. Variance $=\text{DevErr}-\text{TrainErr}=13-11=2\%$. Why this step? This is the gap between seen and unseen data — small here.
3. Compare: $8\%\gg 2\%$. Why this step? The 80/20 rule: pull the taller lever first. Bias tall → bigger model / better features / train longer.

Answer: attack bias (model-centric this round). See Error analysis for how to tag the failure categories.

Verify: $8>2$ and TrainErr $>$ HLP (so bias is legitimately positive). ✓

---

Cell BV-var — variance dominates, attack the data

Steps.

1. Avoidable bias $=5-4=1\%$. Why this step? Tiny — training almost matches the human floor, so the model class is fine.
2. Variance $=15-5=10\%$. Why this step? This is the gap between seen and unseen data — huge here.
3. Compare: $10\%\gg 1\%$. Why this step? Variance is the tall lever. Fixes: more/cleaner data, regularization, augmentation — a data-centric move.

Answer: attack variance: more data + regularization.

Verify: $10>1$, both gaps non-negative (TrainErr $>$ HLP). ✓

---

Cell BV-neg — the overfitting alarm (bias goes negative)

Steps.

1. Avoidable bias $=\text{TrainErr}-\text{HLP}=2-5=-3\%$. Why this step? HLP is a floor on generalization, not on training error. A model can drive TrainErr below HLP by memorizing noise — so a negative bias is not a bonus, it's meaningless.
2. Read the sign as a flag, not a quantity. Why this step? The parent's caveat: $\text{TrainErr}<\text{HLP}$ is a red flag for overfitting, so the real signal lives entirely in the variance term.
3. Variance $=\text{DevErr}-\text{TrainErr}=12-2=10\%$. Why this step? This 10% gap confirms it — massive overfitting. Fix like BV-var (data + regularization), and ignore the negative bias.

Answer: overfitting alarm. Bias is $-3\%$ (discard it); act on the $10\%$ variance.

Verify: Bias sign is negative ($2<5$) and variance is large ($10\%$) — the two conditions that define the alarm. ✓

---

Cell BV-both — both gaps large, pick the first move

Steps.

1. Avoidable bias $=9-1=8\%$; variance $=18-9=9\%$. Why this step? Both are big and close — no clear winner by size alone.
2. Fix bias first anyway. Why this step? Variance is measured relative to training error. If you shrink data-variance while training error is still bad, you just converge both bars toward a still-high floor. Lower training error first, then re-measure the gap. Order matters because the two gaps are not independent.
3. Re-plan: after a bigger model drops TrainErr, recompute variance and decide the next lever. Why this step? The lifecycle is a loop — one diagnostic, one move, re-measure.

Answer: bias and variance are $8\%$ and $9\%$; reduce bias first, then reassess.

Verify: $8+1=9=$ TrainErr and $9+9=18=$ DevErr — the two gaps reconstruct the errors. ✓

---

Cell DRIFT-x — the input distribution moved

Steps.

1. Note: code unchanged, labels of old examples still correct → the rule $P(y\mid x)$ is intact. Why this step? Eliminates concept drift. See Data drift and concept drift.
2. New, never-seen inputs are arriving → the input distribution $P(x)$ has shifted. Why this step? By the parent's definition, changed $P(x)$ = data drift.
3. Route the fix to Stage 2 (Data) — go back to the Data stage and collect + label the new-topic examples, then retrain. Why this step? Retraining on old data can't cover inputs it never contained — the feedback arrow goes to the Data stage (Stage 2), not straight to Modeling. This is exactly what Model monitoring and observability alarms are for.

Answer: data drift; fix in Stage 2 (Data) by gathering fresh data.

Verify: Consistency check — $P(x)$ changed and $P(y\mid x)$ unchanged is precisely the data-drift definition. ✓

---

Cell DRIFT-yx — the rule itself moved

Steps.

1. Inputs unchanged → $P(x)$ is stable. Rule out data drift. Why this step? Data drift requires a moved input distribution, which we don't have.
2. Same $x$ now maps to a different correct $y$ → $P(y\mid x)$ changed. Why this step? That is the parent's definition of concept drift.
3. Fix: relabel recent data in the Data stage (Stage 2), then retrain in the Modeling stage (Stage 3) — i.e. Stage 2 → Stage 3. Why this step? Old labels encode an obsolete rule, so we first repair the data (Stage 2, relabel), and only then can Modeling (Stage 3) learn the new $P(y\mid x)$. Unlike data drift, the inputs are fine — the labels are what's stale, so the jump into Stage 3 (retrain) follows relabeling.

Answer: concept drift; relabel in Stage 2, then retrain in Stage 3.

Verify: $P(x)$ fixed, $P(y\mid x)$ changed → concept drift by definition. ✓

---

Cell WORD — a full real-world word problem

Steps.

1. Without canary: 0.6\times150{,}000 - 40{,}000 - 30{,}000 = 90{,}000 - 70{,}000 = \20{,}000$. Why this step? Baseline EV to compare against.
2. With canary: $p_{\text{success}}=0.75$, and build cost rises by $10k to $50k: 0.75\times150{,}000 - 50{,}000 - 30{,}000 = 112{,}500 - 80{,}000 = \32{,}500$.\ast Whythisstep?\ast Thecanarychanges\ast two\ast inputs:itraises$p_{\text{success}}$ (safer rollout) and adds cost. EV weighs both at once.
3. Compare the two EVs: \32{,}500 - $20{,}000 = $12{,}500$ uplift. Why this step? The decision is the difference in EV, not either value alone — the canary is worth it exactly when this difference is positive.

Answer: the canary raises expected value from $20{,}000 to $32{,}500, an uplift of $12{,}500 $>0$ → yes, add the canary.

Verify: With-canary ($32.5k) minus without ($20k) = $12.5k. Cross-check: the payoff uplift is 0.15\times150{,}000 = \22{,}500, minus the extra \10k cost = $12.5k — the two routes agree. ✓

---

Cell EXAM — the satisficing-metric trap

Steps.

1. Apply the satisficing metric first: keep only models with latency $<100$ ms. Why this step? A satisficing metric is a gate, not a score — you must pass it before accuracy is even compared.
2. Model B has $220$ ms $\ge 100$ ms → disqualified, regardless of its accuracy. Why this step? The constraint is hard; "$99\%$ but too slow" is a fail, not a win.
3. Among survivors (only A), maximize accuracy → ship A. Why this step? The north-star (accuracy) is only evaluated within the feasible set.

Answer: ship Model A — the $96\%$/40 ms model. Higher test accuracy $\ne$ better project.

Verify: Latencies: $40<100$ (A passes), $220\nless 100$ (B fails). Feasible set = {A}, so A wins by default. ✓

---

Recall

The three lifecycle formulas each have a full case-space — recall the edge cases. When is $\mathbb{E}[\text{Value}]$ pure loss regardless of payoff? ::: When $p_{\text{success}}=0$ (or costs exceed the weighted payoff). If TrainErr $<$ HLP, what does avoidable bias mean? ::: Nothing usable — it's a negative-signed overfitting alarm; act on variance. Same inputs, wrong answers now — which drift? ::: Concept drift ($P(y\mid x)$ changed); fix by relabeling in Stage 2, then retrain in Stage 3. New unseen inputs, rule intact — which drift? ::: Data drift ($P(x)$ changed); fix by collecting new data in Stage 2. A satisficing metric is used as a ...? ::: Hard gate you must pass before comparing the north-star metric.

---

Related: ML project lifecycle · CI-CD for ML pipelines · Model monitoring and observability · Error analysis · Bias-variance decomposition