Experiment tracking and reproducibility

WHAT is being tracked?

An ML training run is a function that returns both the trained model and its measured metrics:

$(\text{artifacts},\ \text{metrics}) = f(\text{code},\ \text{data},\ \text{hyperparameters},\ \text{environment},\ \text{random seed})$

To reproduce a result you must pin every argument on the right-hand side. If even one is un-pinned, $f$ becomes non-deterministic and the result is not reproducible.

The five things you MUST version:

Axis	WHAT to pin	Tool example
Code	git commit SHA	git
Data	dataset hash / version	DVC, LakeFS
Config	all hyperparameters	YAML / config logged to tracker
Environment	library versions, CUDA	Docker image digest, `requirements.txt` lock
Randomness	the seed	`seed=42` logged

Figure — Experiment tracking and reproducibility

WHY does this matter? (the failure it prevents)

The economic 80/20: ~20% of tracking discipline (log config + git SHA + data hash + seed) prevents ~80% of "cannot reproduce" incidents. Full bit-for-bit determinism (the remaining 20%) is expensive and often unnecessary.

HOW: deriving the metadata schema from first principles

We don't memorize what a tracker stores — we derive it. Ask: "What is the minimum I need to recreate this run and to compare it?"

To recreate → I need every argument of $f$ : code_sha, data_hash, params, env, seed.
To compare → I need the outputs: metrics (accuracy, loss curves).
To reuse → I need the produced object: artifacts (model weights, plots).
To find it later → I need identity + context: run_id, timestamp, tags, user.

That derivation is the schema of MLflow / Weights & Biases:

Formula The tracked-run tuple

$\text{Run} = \big(\underbrace{\text{run\_id}}_{\text{identity}},\ \underbrace{\text{params}, \text{code\_sha}, \text{data\_hash}, \text{env}, \text{seed}}_{\text{inputs (all 5 axes of } f)},\ \underbrace{\text{metrics}}_{\text{outputs}},\ \underbrace{\text{artifacts}}_{\text{products}}\big)$

Why each field? Drop data_hash → cannot reproduce after data changes. Drop env → cannot reproduce after a library upgrade. Drop metrics → cannot compare. Drop run_id → cannot reference. Nothing here is decorative — the five input fields are exactly the five arguments of $f$ , and metrics/artifacts are exactly its two outputs.

Quantifying reproducibility

Common mistakes (Steel-manned)

Active recall

Recall Cover the answers. Predict first (Forecast-then-Verify), then reveal.

What are the 5 arguments of the training function $f$ ? → code, data, hyperparameters, environment, random seed.
What are the 2 outputs of $f$ ? → artifacts (model, plots) and metrics.
Why do stage determinism probabilities multiply? → whole-pipeline reproducibility needs all stages reproducible simultaneously (AND of independent events).
What does git NOT pin? → data, env/library versions, seed, GPU nondeterminism.
What's the 80/20 of reproducibility? → log config+SHA+data hash+seed → prevents most incidents cheaply.

Recall Feynman: explain to a 12-year-old

Imagine you baked an amazing cake. To bake the exact same cake again, writing down "cake" isn't enough — you need the recipe (steps), the ingredients and their brand (data), how much of each (settings), your oven's temperature (environment), and even which spoon you stirred with if it changes the taste (the random seed). Experiment tracking is writing down all of that every single time. Then when your cake is great, you can make it again — and when it's bad, you can look at your notes and see exactly what you did differently.

Connections

Data Versioning with DVC — pins the data axis of $f$ .
Docker and Containerization — pins the environment axis.
Random Seeds and Determinism in PyTorch — pins the seed axis, raises each $p_i$ .
Hyperparameter Tuning — generates the many runs tracking must compare.
Model Registry and Deployment — consumes the tracked artifacts.
CI-CD for Machine Learning — automates re-running the pinned pipeline.

Experiment tracking is the systematic recording of what, so runs can be compared and reproduced?

Every input (code, data, hyperparameters, environment, seed), output (metrics), and product (artifacts) of a training run.

The five arguments of the training function f that must be pinned for reproducibility?

Code (git SHA), Data (hash/version), Hyperparameters/config, Environment (library+CUDA versions), Random seed.

The two outputs of the training function f?

Artifacts (model weights, plots) and metrics (accuracy, loss curves) — both must be captured in the run tuple.

Why does git alone NOT guarantee reproducibility?

Git pins code only; it doesn't track data, library/env versions, the runtime seed, or GPU nondeterminism.

Formula for a pipeline's reproducibility probability with n independent stages?

P_repro = product over i of p_i (each stage must be reproducible simultaneously → AND events multiply).

Why do the stage determinism probabilities multiply rather than add?

Whole-pipeline reproducibility requires ALL stages reproducible at once; independent AND events multiply.

A single unseeded stage with p=0.5 does what to a careful pipeline?

Caps overall P_repro at ≤0.5 regardless of other stages, because it's a factor in the product.

What is an 'orphan metric'?

A recorded result number with no inputs (config/data/code) attached, so it can't be reproduced or explained.

The 80/20 of reproducibility?

Logging config + git SHA + data hash + seed cheaply prevents ~80% of 'cannot reproduce' incidents.

Does np.random.seed(42) make PyTorch GPU training deterministic?

No — Torch has its own RNG and GPU reductions are nondeterministic; you must seed all RNGs and enable deterministic algorithms.

CHESS mnemonic expands to?

Code(SHA), Hyperparameters, Environment, Seed, Set-of-data(hash).

Why is a data hash valuable during debugging?

A changed hash instantly reveals the data file was silently modified, distinguishing a data problem from a code problem.

When comparing two runs to attribute an improvement to a hyperparameter, what must be held constant?

The random seed (and all other axes) — otherwise the metric difference could be seed noise, not the hyperparameter's effect.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, machine learning basically ek science experiment hai jo bahut fast chalta hai. Aap sau-sau training runs karte ho, har baar thoda sa change — kabhi learning rate, kabhi data, kabhi seed. Experiment tracking ka matlab hai ek proper lab notebook rakhna, jisme har run ka input (code ka git SHA, data ka hash, hyperparameters, environment, seed), output (metrics jaise accuracy, loss) aur artifacts (model file) sab automatically likha jaaye. Warna 6 mahine baad jab boss bole "wahi 94% wala model chahiye", tab aap fir se train karoge aur 89% aayega — aur pata hi nahi chalega kya badla.

Reproducibility ka core funda ye hai: aapka training ek function hai, (artifacts, metrics) = f(code, data, hyperparameters, environment, seed). Agar in paanch inputs me se ek bhi cheez pin nahi hui, toh result non-deterministic ho jaata hai. Bahut log sochte hain "git hai na, sab track ho gaya" — galat! Git sirf code pin karta hai, data-env-seed nahi. Isliye CHESS yaad rakho: Code, Hyperparams, Environment, Seed, Set-of-data.

Ek important math intuition: agar pipeline ke n stages hain aur har stage $p_i$ probability se deterministic hai, toh poora pipeline $P = p_1 \times p_2 \times \dots \times p_n$ se reproduce hoga — multiply, add nahi. Kyunki sabhi stages ko ek saath match karna zaroori hai. Iska matlab ek hi unseeded stage (jaise $p=0.5$ ) pura kaam kharaab kar deta hai. Isliye saare RNGs seed karo (python, numpy, torch) aur GPU ke liye deterministic algorithms on karo.

Aur ek practical tip: jab do experiments compare karo taaki pata chale kaunsa change accha tha, toh ek time pe ek hi cheez badlo aur seed fix rakho — warna 1% ka fayda sirf luck (seed noise) ho sakta hai. Yahi tracking ki asli power hai: mystery ko simple diff bana deta hai.

Experiment tracking and reproducibility

WHAT is being tracked?

WHY does this matter? (the failure it prevents)

HOW: deriving the metadata schema from first principles

Quantifying reproducibility

Common mistakes (Steel-manned)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections