Experiment tracking and reproducibility — Hinglish notes

5.3.2 · AI-ML › MLOps & Deployment

WHAT track ho raha hai?

Ek ML training run ek aisi function hai jo dono trained model aur uske measured metrics return karti hai:

$(artifacts, metrics) = f (code, data, hyperparameters, environment, random seed)$

Ek result reproduce karne ke liye aapko right-hand side ka har argument pin karna hoga. Agar ek bhi un-pinned raha, toh $f$ non-deterministic ban jaata hai aur result reproducible nahi hota.

Wo paanch cheezein jo aapko ZAROOR version karni hain:

Axis	KYA pin karna hai	Tool example
Code	git commit SHA	git
Data	dataset hash / version	DVC, LakeFS
Config	saare hyperparameters	YAML / config logged to tracker
Environment	library versions, CUDA	Docker image digest, `requirements.txt` lock
Randomness	seed	`seed=42` logged

Figure — Experiment tracking and reproducibility

WHY matters karta hai? (wo failure jo ye rokta hai)

Economic 80/20: ~20% tracking discipline (log config + git SHA + data hash + seed) ~80% "cannot reproduce" incidents rokti hai. Full bit-for-bit determinism (remaining 20%) expensive hai aur aksar zaroori nahi.

HOW: metadata schema ko first principles se derive karna

Hum ye yaad nahi karte ki ek tracker kya store karta hai — hum ise derive karte hain. Poocho: "Run ko recreate karne aur compare karne ke liye minimum kya chahiye?"

Recreate karne ke liye → $f$ ka har argument chahiye: code_sha, data_hash, params, env, seed.
Compare karne ke liye → outputs chahiye: metrics (accuracy, loss curves).
Reuse karne ke liye → produced object chahiye: artifacts (model weights, plots).
Baad mein dhundhne ke liye → identity + context chahiye: run_id, timestamp, tags, user.

Ye derivation hi MLflow / Weights & Biases ka schema hai:

Reproducibility quantify karna

Common mistakes (Steel-manned)

Active recall

Recall Answers cover karo. Pehle predict karo (Forecast-then-Verify), phir reveal karo.

Training function $f$ ke 5 arguments kya hain? → code, data, hyperparameters, environment, random seed.
$f$ ke 2 outputs kya hain? → artifacts (model, plots) aur metrics.
Stage determinism probabilities multiply kyun hoti hain? → whole-pipeline reproducibility ko saare stages simultaneously reproducible chahiye (independent events ka AND multiply hota hai).
Git kya pin NAHI karta? → data, env/library versions, seed, GPU nondeterminism.
Reproducibility ka 80/20 kya hai? → log config+SHA+data hash+seed → zyaatar incidents cheaply rokta hai.

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tumne ek amazing cake banaya. Exactly wohi cake phir se banane ke liye, "cake" likhna kaafi nahi — tumhe recipe chahiye (steps), ingredients aur unka brand (data), har ek kitna (settings), tumhare oven ka temperature (environment), aur woh kaun sa chamcha tha jisse tune hilaaya agar wo taste change karta hai (random seed). Experiment tracking matlab hai ye sab har baar likh lena. Tab jab tumhara cake great ho, tum phir bana sakte ho — aur jab bura ho, tum apne notes dekh sakte ho aur exactly dekh sakte ho kya alag kiya tha.

Connections

Data Versioning with DVC — $f$ ka data axis pin karta hai.
Docker and Containerization — environment axis pin karta hai.
Random Seeds and Determinism in PyTorch — seed axis pin karta hai, har $p_{i}$ badhata hai.
Hyperparameter Tuning — wo bahut saare runs generate karta hai jo tracking ko compare karne hote hain.
Model Registry and Deployment — tracked artifacts consume karta hai.
CI-CD for Machine Learning — pinned pipeline ko automatically re-run karta hai.

Experiment tracking kya ki systematic recording hai, taaki runs compare aur reproduce ho sakein?

Har input (code, data, hyperparameters, environment, seed), output (metrics), aur product (artifacts) ki ek training run ki.

Training function f ke wo paanch arguments jo reproducibility ke liye pin karne zaroori hain?

Code (git SHA), Data (hash/version), Hyperparameters/config, Environment (library+CUDA versions), Random seed.

Training function f ke do outputs kya hain?

Artifacts (model weights, plots) aur metrics (accuracy, loss curves) — dono run tuple mein capture hone chahiye.

Git alone reproducibility guarantee kyun NAHI karta?

Git sirf code pin karta hai; ye data, library/env versions, runtime seed, ya GPU nondeterminism track nahi karta.

n independent stages wali pipeline ki reproducibility probability ka formula?

P_repro = i ke upar p_i ka product (har stage simultaneously reproducible hona chahiye → AND events multiply hote hain).

Stage determinism probabilities multiply kyun hoti hain, add kyun nahi?

Whole-pipeline reproducibility ke liye SAARE stages ek saath reproducible hone chahiye; independent AND events multiply hote hain.

Ek unseeded stage jiska p=0.5 ho, ek careful pipeline ke saath kya karta hai?

Overall P_repro ko ≤0.5 tak cap kar deta hai regardless of other stages, kyunki ye product mein ek factor hai.

'Orphan metric' kya hota hai?

Ek recorded result number jiske saath koi inputs (config/data/code) attached nahi, toh use reproduce ya explain nahi kiya ja sakta.

Reproducibility ka 80/20 kya hai?

Config + git SHA + data hash + seed log karna cheaply ~80% "cannot reproduce" incidents rokta hai.

Kya np.random.seed(42) PyTorch GPU training ko deterministic banata hai?

Nahi — Torch ka apna RNG hai aur GPU reductions nondeterministic hain; saare RNGs seed karne padte hain aur deterministic algorithms enable karne padte hain.

CHESS mnemonic expand hota hai?

Code(SHA), Hyperparameters, Environment, Seed, Set-of-data(hash).

Debugging mein data hash kyun valuable hai?

Changed hash turant reveal karta hai ki data file silently modify ho gayi, data problem ko code problem se alag karta hai.

Do runs compare karte waqt kisi hyperparameter ko credit dene ke liye kya constant rakhna chahiye?

Random seed (aur baaki saare axes) — warna metric difference seed noise ho sakta hai, hyperparameter ka effect nahi.