ML project lifecycle: Exercises — AI-ML deep dive

Before we start, here is the one map to keep in your head. Every exercise lives on one of these five stages, and the arrows are why we call it a loop, not a line.

The three formulas you will reuse

We will only ever need three little tools. Let us re-state each in plain words before any symbol is used, so a reader who has never seen them can still follow.

Level 1 — Recognition

Recall Solution

Order: Scoping → Data → Modeling → Deployment → Monitoring. The stage that first quantifies worth is Scoping (it uses Tool 1, the expected-value formula). WHAT this looks like: the leftmost box in the figure above, before you ever touch data.

Recall Solution

(a) ::: Data drift (b) ::: Canary deployment (c) ::: Shadow deployment See Deployment strategies (shadow, canary, blue-green) and Data drift and concept drift.

Level 2 — Application

Recall Solution

Plug into Tool 1: $E [Value] = 0.7 \times 200, 000 - 50, 000 - 40, 000 = 140, 000 - 90, 000 = $50, 000.$ WHY this step: we price feasibility ( $0.7$ ) and ongoing cost ( $40 k$ ) in one number. $\mathbb{E}[\text{Value}] = +\$ 50{,}000 > 0$ → green-light in scoping.

Recall Solution

Set $E [Value] = 0$ and solve for $p_{success}$ : $p_{success} \cdot 200, 000 - 90, 000 = 0 \Rightarrow p_{success} = \frac{90 , 000}{200 , 000} = 0.45.$ WHY this step: at break-even the payoff exactly covers all fixed costs. Any feasibility $p_{success} > 0.45$ makes it worthwhile. Answer: == $0.45$ ==.

Level 3 — Analysis

Recall Solution

Tool 2: $Avoidable bias = 9 - 1 = 8%, Variance = 12 - 9 = 3%.$ Bias ( $8%$ ) $≫$ variance ( $3%$ ), so the biggest lever is the model / features (bigger model, better architecture, richer features), not more data. See Bias-variance decomposition.

Recall Solution

$Avoidable bias = 5 - 4 = 1%, Variance = 15 - 5 = 10%.$ Now variance ( $10%$ ) dominates. The lever is more / cleaner data and regularization — attack the generalization gap, not the model size.

Recall Solution

$Avoidable bias = 3 - 5 = - 2%.$ A negative avoidable bias is meaningless as a "gap to close." WHAT it really means: $TrainErr < HLP$ , so the model beats humans on the training set — the classic signature of overfitting (it memorized noise). See Human-level performance and Bayes error. The real signal now lives entirely in the variance term ( $11 - 3 = 8%$ ): fix generalization.

Level 4 — Synthesis

Recall Solution

(a) ::: Data drift — the inputs $P (x)$ changed (new vocabulary); the rule $P (y ∣ x)$ is fine. (b) ::: Jump back to Stage 2 (Data) — collect + label the new inputs, then retrain. Retraining on old data cannot help because old data lacks the new inputs. (c) ::: Shadow deployment — the new model runs on live traffic with output ignored, so you compare silently and spot the mismatch before it reaches users. See Model monitoring and observability.

Recall Solution

(a) ::: Concept drift — same $P (x)$ , but $P (y ∣ x)$ changed (same input, new correct answer). (b) ::: Relabeling new data under the new policy helps; the old labels are now wrong, so you must re-label to the new concept and retrain. This is why concept drift is nastier than data drift — even your history is stale.

Level 5 — Mastery

Worked example

L5.1 — Full loop under a constraint. You must pick between two candidate models for a live API that requires latency $< 100$ ms (a satisficing metric):

Model	Accuracy	Latency
A	$99%$	$130$ ms
B	$96%$	$40$ ms

(a) Which model ships, and why? (b) After shipping, you want to try Model A again later without risking users — name the deployment pattern and the automation level that lets you compare A's predictions against B's live decisions with zero blast radius.

Recall Solution

(a) ::: Model B. The north-star is the business metric subject to the satisficing constraint. Model A violates latency ( $130 > 100$ ms), so its higher accuracy is disqualified — a fast $96%$ beats a too-slow $99%$ . This is a scoping decision. (b) ::: Shadow deployment at the "Shadow" automation level: A runs on live traffic, its output is logged but never used, so you compare A vs B silently with zero user risk. See Deployment strategies (shadow, canary, blue-green).

Recall Solution

(a) Tool 1: $E [Value] = 0.55 \times 300, 000 - 60, 000 - 50, 000 = 165, 000 - 110, 000 = $55, 000 > 0.$ Build it — positive expected value even after pricing the ongoing monitoring team. (b) ::: Data-centric. The dominant error category (60%) is noisy labels, so relabeling that one category (fixing the data for a fixed model) is the 80/20 move — clean data generalizes better. See Data-centric vs model-centric AI.

Recall

One-line self-check: which single formula would you use for each of — "is it worth building?", "where do I spend the week?", "which arrow after a production drop?" Answer ::: Tool 1 (expected value); Tool 2 (bias/variance gaps); Tool 3 (data vs concept drift).