Model retraining pipelines — Hinglish notes

5.3.15 · AI-ML › MLOps & Deployment

WHAT hai ek retraining pipeline?

Key word hai automated & repeatable. Manually notebook re-run karna ek pipeline nahi hai; woh versioned, testable, ya triggerable nahi hota.

WHY hum retrain karte hain? (Drift — first principles)

Ek model $f_{θ}$ ek training distribution $P_{train} (X, Y)$ par error minimise karne ke liye fit hota hai. Production mein woh $P_{serve} (X, Y)$ dekhta hai. Performance tab degrade hoti hai jab yeh dono diverge karte hain. Iske do flavours hain:

WHY yeh distinction matters hai: agar sirf $P (X)$ shifted hai, toh kabhi-kabhi reweighting/zyada data collect karna fix kar deta hai. Agar $P (Y ∣ X)$ shifted hai, toh aapko ZAROOR retrain karna hoga — purane labels ab reality describe nahi karte.

"Kya mujhe retrain karna chahiye?" signal derive karna

Hum ek scalar chahte hain jo kahe "distributions differ kar rahe hain." Ek classic hai Population Stability Index (PSI), jo KL divergence jaisi idea se derive hota hai.

Ek feature ko $k$ bins mein split karo. Maano $a_{i}$ = expected (training) samples ka fraction bin $i$ mein, $b_{i}$ = actual (recent production) samples ka fraction bin $i$ mein.

Yeh form kyun? Hum ek aisa term chahte hain jo $a_{i} = b_{i}$ par $0$ ho aur jaise yeh differ karte hain, symmetrically badhe. KL se shuru karo: $\sum b_{i} ln (b_{i} / a_{i})$ . Woh asymmetric hai. PSI ise symmetrise karta hai:

$PSI = \sum_{i = 1}^{k} (b_{i} - a_{i}) ln \frac{b _{i}}{a _{i}}$

HOW structured hoti hai ek pipeline?

Trigger types (WHY har ek exist karta hai):

Trigger	Kab use karein	Kyun
Scheduled (cron: nightly/weekly)	Steady, predictable drift	Simple, reason karna cheap
Performance-based	Aapke paas fast ground-truth labels hain	Retrain sirf tab jab zaroorat ho → compute bachata hai
Drift-based (PSI/KS test)	Labels delayed hain	Input shift accuracy drop se pehle detect karo
Data-volume-based	Kaafi naya data accumulate ho gaya	Naya data = naya information seekhne layak

Critical gate: kabhi bhi worse model promote mat karo

Fix hai champion–challenger evaluation:

$promote ⟺ (metric (challenger) - metric (champion)) > δ$

jahan $δ$ noise absorb karta hai (taaki aap random $\pm 0.1%$ wiggles par flip-flop na karo).

Worked examples

Common mistakes

Active recall

Recall Khud test karo (answers chhupa lo)

Drift ke do types kya hain, aur kaun sa zaroor retraining force karta hai?
Har PSI term guaranteed $\geq 0$ kyun hoti hai?
Retrain bahut baar karna net loss kyun ho sakta hai?
Champion–challenger gate kis cheez se protect karta hai?
Eval set temporally held out kyun hona chahiye?

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tumne pichle saal apni class ke saare popular songs seekh liye, toh tum achhe se guess kar sakte ho ki tumhare dosto ko kya pasand hai. Lekin is saal naye bachche aaye aur sabko naye songs pasand hain — tumhare purane guesses miss hone lage! Ek retraining pipeline ek helper robot jaisi hai jo class par nazar rakhti hai, notice karti hai jab songs bahut badal gaye hain, aur chupke se naye favorites re-learn kar leti hai. Lekin woh careful hai: apne naye guesses par trust karne se pehle, woh apne purane self ke against ek chhota game khelti hai, aur sirf tab switch karti hai jab naya version sach mein better guess kare. Iss tarah woh kabhi accident se worse nahi hoti.

Flashcards

Model retraining pipeline kya hoti hai?

Ek automated, repeatable workflow jo fresh data ingest karta hai, validate karta hai, retrain karta hai, candidate ko production ke against evaluate karta hai, aur promote karta hai sirf tab jab quality gates pass ho jaayein (warna rollback ho jaata hai).

Covariate (data) drift vs concept drift?

Covariate:

P (X)

changes,

P (Y ∣ X)

same. Concept:

P (Y ∣ X)

khud changes — yahi wala retraining force karta hai.

PSI formula likho.

PSI = \sum_{i} (b_{i} - a_{i}) ln (b_{i} / a_{i})

, binned expected fractions

a_{i}

aur actual fractions

b_{i}

par.

Har PSI term non-negative kyun hoti hai?

(b_{i} - a_{i})

aur

ln (b_{i} / a_{i})

hamesha same sign share karte hain, toh unka product

\geq 0

hota hai; sum

0

hoga iff distributions match karein.

PSI interpretation thresholds?

<0.1 stable, 0.1–0.25 moderate (watch karo), >0.25 significant drift (retrain trigger karo).

Champion–challenger kya hai?

Champion = current prod model; challenger = freshly retrained candidate. Challenger ko promote karo sirf tab jab woh champion ko margin

δ

se recent held-out data par beat kare.

Har ghante retrain kyun nahi karte?

Retraining ka fixed cost

C_{r}

hota hai; agar drift (staleness cost) chhota hai, toh benefit < cost → net loss. Frequency ko drift rate se match karo.

Chaar common retraining triggers?

Scheduled (cron), performance-based, drift-based (PSI/KS), aur data-volume-based.

Evaluation set temporally held out kyun hona chahiye?

Train–serve leakage avoid karne ke liye aur real production mimic karne ke liye; dekhe hue data reuse karne se metrics inflate hote hain aur rot chhup jaata hai.

High PSI matlab model broken hai?

Nahi — PSI input drift measure karta hai, accuracy nahi. Decide karne se pehle actual performance se confirm karo.

Data + code + params saath version kyun karo?

Ek model ko reproduce aur debug karne ke liye; artifact akela nahi bata sakta ki usse kisne produce kiya (lineage).

Connections

Data drift and concept drift
Model monitoring and observability
Champion-Challenger and A-B testing
Feature stores
KL divergence (PSI ki parent idea)
CI-CD for ML (CI-CD-CT)
Data versioning and lineage
Shadow deployment and canary release