An ML project is not "train a model once and ship it." It is a loop that keeps turning
because the world (your data) keeps changing. Think of it like farming, not manufacturing: you
don't build a car once and it's done — you plant, grow, harvest, then the season changes and you
replant. The whole point of the lifecycle is to make this loop cheap, safe, and repeatable .
WHY: A model is a function learned from past data, but it is deployed to act on future data.
The moment reality drifts from the training distribution, the model rots. So we need a structured
process to (a) know what we are optimizing, (b) get data, (c) build, (d) ship, and
(e) watch it and come back . Skipping any stage doesn't remove the work — it just moves the
pain to a worse time (usually production, at 2 a.m.).
The canonical stages (Google/Andrew Ng framing):
Scoping — define the problem, metrics, feasibility, value.Data — collect, label, clean, establish a baseline.Modeling — train, error-analyze, iterate.Deployment — serve predictions to real users/systems.Monitoring & Maintenance — detect drift, retrain, loop back.
ML project lifecycle := the iterative sequence of stages
Scoping → Data → Modeling → Deployment → Monitoring \text{Scoping} \to \text{Data} \to \text{Modeling} \to \text{Deployment} \to \text{Monitoring} Scoping → Data → Modeling → Deployment → Monitoring feedback arrows from every later stage back to earlier ones.
WHAT: Decide whether and what to build before touching data.
WHY: A model that hits 99% accuracy on the wrong metric is worthless. Most project failures are
scoping failures , not modeling failures.
HOW (checklist):
Identify the business/user problem and the decision it drives.
Choose a north-star metric and a satisficing metric (e.g. maximize accuracy subject to
latency < 100 < 100 < 100
Estimate feasibility (is human-level performance possible? is signal in the data?) and
value (does a 2% lift matter?).
WHAT: Get data whose distribution matches deployment , label it consistently , and set a
baseline . HOW you measure "good data" is label consistency , not just quantity.
WHY it dominates: in most real projects, improving data beats improving the model.
Baseline := a simple reference performance (human level, a heuristic, or a trivial model) that
tells you whether your fancy model is actually adding value and how much room there is to improve.
WHAT: Train, run error analysis , iterate. WHY the loop: your first model is a diagnostic tool ,
not the product. HOW: tag mispredictions into categories, fix the biggest category first (80/20).
Model-centric ("improve the model on fixed data") vs data-centric ("improve the data for a fixed
model") — modern practice leans data-centric because clean data generalizes better.
WHAT: Put the model where real inputs arrive. WHY it's hard: code that works in a notebook meets
messy live traffic, latency limits, and versioning. HOW to de-risk: ship gradually.
Deployment patterns (each reduces blast radius):
Shadow deployment — model runs on live traffic but its output is not used ; you compare
silently.Canary deployment — route a small % (e.g. 5%) of traffic, watch, then ramp up.Blue-green deployment — two full environments, switch traffic instantly, roll back instantly.
Degrees of automation: Human only → Shadow → AI assist → Partial autonomy → Full autonomy.
WHAT: Watch the live model for decay and loop back. WHY: the training distribution and the live
distribution silently diverge — this is drift .
Data drift := input distribution P ( x ) P(x) P ( x ) Concept drift := the relationship
P ( y ∣ x ) P(y\mid x) P ( y ∣ x ) frozen model even if
its code is bug-free.
Feedback loop: monitor a metric → alarm fires → error-analyze → jump back to Data or Modeling →
redeploy. This is why the lifecycle is a cycle , not a line.
Example 1 — Should we build a spam classifier?
Given: p success = 0.6 p_{\text{success}}=0.6 p success = 0.6 V_{\text{success}}=\ 100{,}000, , , , ,
, . .
. .
**Why this step?** We plug into the expected-value formula because it forces us to price *both*
feasibility and maintenance — a positive result means "green-light in scoping." (If maintenance were
\ 60k, EV would be -\ 10k$: don't build.)
Example 2 — Where do I spend my week? (bias/variance)
HLP = 2 % = 2\% = 2% = 8 % = 8\% = 8% = 10 % = 10\% = 10% = 8 − 2 = 6 % = 8-2 = 6\% = 8 − 2 = 6% = 10 − 8 = 2 % = 10-8 = 2\% = 10 − 8 = 2% Why this step? Bias (6%) ≫ \gg ≫ biggest lever is the model/features, not
more data. This is the 80/20 stage decision — attack the dominant gap. (Note: this reasoning is
valid because TrainErr > > > below HLP, we'd suspect overfitting
instead.)
Example 3 — Diagnosing a production drop.
Live accuracy falls from 92% to 85% but retraining on old data doesn't help; new inputs contain a
word ("covid") never seen in training. Why this step? Same code, new P ( x ) P(x) P ( x ) data
drift ; the fix lives in Stage 2 (collect new data + relabel), proving the feedback arrow.
"Training error can never go below human-level / Bayes error."
Why it feels right: HLP is "the best possible," so surely nothing beats it.
The fix: HLP is a floor on the best generalization (dev/test) error, not on training error. An
overfit model memorizes noise and can post TrainErr below HLP. When that happens, "avoidable
bias" goes negative and is meaningless — treat it as an overfitting alarm and focus on the variance term.
"Higher test accuracy = better project."
Why it feels right: accuracy is the number we stared at all through modeling.
The fix: the north-star is the business/decision metric under constraints (latency, fairness,
cost). A slower 99% model can be worse than a fast 96% one. Scoping defines the real metric.
"Deploy once, done."
Why it feels right: in traditional software, shipped code keeps working.
The fix: ML depends on a live data distribution that drifts, so a deployed model decays .
Monitoring + retraining is mandatory, not optional.
"More data always beats better data."
Why it feels right: deep learning famously loves scale.
The fix: if variance is already tiny and labels are inconsistent, adding noisy data hurts.
Use the bias/variance split to decide; often consistent labels win.
"Go straight to full autonomy."
Why it feels right: automation is the goal.
The fix: shadow → canary de-risk failures cheaply. Skipping them turns a small bug into a full
outage.
Recall Quick self-test (hide answers, forecast first)
Name the 5 stages in order.
What is the difference between data drift and concept drift?
Given HLP 1%, Train 5%, Dev 6% — where do you focus?
Why is shadow deployment safer than canary for a first rollout?
Write the expected-value-of-project formula and name each term.
What does it mean if TrainErr comes out below HLP?
Recall Feynman: explain to a 12-year-oldImagine you train a puppy (the model) using treats from last summer . You define what "good
behavior" means (scoping), collect treats and show examples (data), teach the puppy (modeling),
then let it loose at a party (deployment). But people at the party act differently than in
practice, so you keep watching and go back to re-teach when it messes up (monitoring). You
never stop watching — that's why it's a loop, not a one-time trick.
"Silly Dogs Model Dangerously Missing" → S coping, D ata, M odeling,
D eployment, M onitoring. (And "Missing" reminds you: without Monitoring you miss the drift.)
What are the 5 stages of the ML project lifecycle in order? Scoping → Data → Modeling → Deployment → Monitoring & Maintenance.
Why is the ML lifecycle a loop rather than a line? Because deployment reveals drift and errors that feed back into earlier stages (data/modeling) for retraining.
What is the difference between data drift and concept drift? Data drift = P(x) changes; concept drift = P(y|x) changes (same input, different correct label).
Define avoidable bias and how it's computed. Avoidable bias = TrainErr − Human/Bayes level; large positive value → improve model/features.
Define variance in the error-decomposition sense. Variance = DevErr − TrainErr; large value → get more/cleaner data or regularize.
Is HLP/Bayes error a strict lower bound on TRAINING error? No. An overfit model can achieve training error below HLP/Bayes; HLP only floors the best achievable generalization (dev/test) error.
What does it signal if TrainErr < HLP? Overfitting — the model memorizes noise; avoidable bias becomes negative/meaningless, so focus on the variance (generalization) gap.
What does shadow deployment do? Runs the model on live traffic but ignores its outputs, letting you compare it to the current system safely.
What does canary deployment do? Routes a small fraction of traffic to the new model, monitors, then gradually ramps up.
Write the expected-value-of-project formula. E[Value] = p_success · V_success − C_build − C_maintain.
Why can a lower-accuracy model be the better deployment choice? Because the north-star is a business metric under constraints (latency, cost, fairness), not raw accuracy.
What is a baseline and why establish one early? A simple reference (human level / heuristic / trivial model) that shows whether the real model adds value and how much headroom exists.
Data-centric vs model-centric AI? Data-centric improves the data for a fixed model; model-centric improves the model on fixed data.
Expected Value E of Project
North-star & Satisficing Metrics
Intuition Hinglish mein samjhoDekho, ML project ek "ek baar train karo aur bhool jao" wali cheez nahi hai — ye ek loop hai.
Sochoo farming ki tarah: plant karo, ugao, kaato, phir season badalta hai to dubara plant karo.
Paanch stages hote hain: Scoping (problem aur metric decide karo), Data (data collect,
label aur clean karo, ek baseline set karo), Modeling (train karo aur error analysis karo),
Deployment (real users tak model pahuchao), aur Monitoring (drift pakdo, retrain karo, wapas
loop me jao).
Sabse important baat: scoping sabse high-leverage stage hai. Agar tum galat metric optimize kar
rahe ho to 99% accuracy bhi bekaar hai. Expected value formula
E [ V a l u e ] = p s u c c e s s ⋅ V − C b u i l d − C m a i n t a i n E[Value] = p_{success}\cdot V - C_{build} - C_{maintain} E [ V a l u e ] = p s u ccess ⋅ V − C b u i l d − C main t ain nahi hai — agar model overfit ho
jaye to TrainErr HLP se bhi neeche ja sakta hai (noise memorize karke). Us case me "avoidable bias"
negative ho jata hai aur matlab yahi ki overfitting ho rahi hai — tab variance/generalization gap par
focus karo.
Deployment me seedha full rollout mat karo — pehle shadow (output use nahi hota, sirf compare),
phir canary (thoda sa traffic), phir ramp up. Aur deploy karne ke baad kaam khatam nahi hota:
duniya ka data change hota rehta hai, isko drift kehte hain (data drift = input badla, concept
drift = same input par sahi answer badla). Isiliye monitoring lagakar rakhna zaroori hai, warna 2
baje raat ko model chup-chaap fail ho jayega. Yaad rakho: ye cycle hai, line nahi.
