Experiment tracking and reproducibility: Exercises

The single tool we reuse everywhere below is the pipeline determinism product. Let me restate it so no symbol is unearned.

Formula The tool: determinism probability of a pipeline

A training run is a chain of stages (load → shuffle → train → eval). Number them $i = 1, 2, \dots, n$ .

$n$ = how many stages there are (just a counting number).
$p_{i}$ = the probability that stage $i$ is deterministic — meaning: give it the same input twice, and twice you get exactly the same output. $p_{i} = 1$ means "always identical"; $p_{i} = 0.5$ means "identical only half the time".
The symbol $\prod$ (capital Greek "pi") means multiply everything in the list, exactly like $\sum$ means add. So $\prod_{i = 1}^{n} p_{i} = p_{1} \times p_{2} \times \dots \times p_{n}$ .

$P_{repro} = \prod_{i = 1}^{n} p_{i} .$

Why multiply and not add? The whole run reproduces only when stage 1 AND stage 2 AND … AND stage $n$ all reproduce at once. For events joined by "AND", probabilities multiply. Adding would let a good stage "cancel" a bad one, which is nonsense — a broken stage cannot be repaired by a good stage upstream.

The figure below draws the pipeline of L3.1 (load 1.0 → shuffle 0.7 → train 0.9 → eval 1.0). Read it left to right: the height of each bar is the fraction of reruns still bit-identical after that stage. The pale bar is the fraction arriving into a stage; the solid bar is what survives it (input $\times p_{i}$ ). Watch the solid bar shrink from $1.00$ to $0.70$ at the shuffle (the biggest single drop, because $0.7$ is the smallest $p_{i}$ ), stay at $0.70 \times 0.9 = 0.63$ through training, and the final magenta label reads the product $0.63$ . The picture makes the weakest-link lesson visible: the output stream can never be taller than the shortest solid bar upstream.

Figure — Experiment tracking and reproducibility

Level 1 — Recognition

L1.1 — Name the axes

The training run is written as $(artifacts, metrics) = f (code, data, hyperparameters, environment, seed) .$ List the five inputs you must pin to reproduce a run, and the two outputs you compare/reuse.

Recall Solution

Five inputs (the five arguments of $f$ ): code (git commit SHA), data (dataset hash/version), hyperparameters (config), environment (library + CUDA versions), random seed. Two outputs: artifacts (model weights, plots) and metrics (accuracy, loss curves). Mnemonic: C-D-H-E-S in, A-M out.

L1.2 — Match tool to axis

For each axis, name a standard tool: (a) data, (b) environment, (c) code.

Recall Solution

(a) Data → DVC (or LakeFS). (b) Environment → Docker image digest / locked requirements.txt. (c) Code → git commit SHA.

L1.3 — Repeatability vs reproducibility vs replicability

Fill the blank with the correct level: "A different team reads only your written description and gets the same result" = ____ .

Recall Solution

Reproducibility (using the levels defined in the parent note). Caveat — the words are not universal. Different bodies swap these labels: the ACM uses repeatability = same team same setup, reproducibility = different team same setup, and replicability = different setup same conclusion; some science journals flip reproducibility and replicability. So the safe habit is to state which axis you hold fixed rather than trust the word alone. On the parent note's ladder (repeatability → reproducibility → replicability), "different team, same written description, same result" is reproducibility.

Level 2 — Application

L2.1 — Compute pipeline reproducibility

A pipeline has four stages with determinism probabilities $p_{1} = 1.0$ (load), $p_{2} = 0.8$ (augment), $p_{3} = 0.75$ (train), $p_{4} = 1.0$ (eval). Compute $P_{repro}$ .

Recall Solution

Step — multiply (the AND rule). Why multiply? All four must reproduce simultaneously (and we take the stages as independent — separately seeded). $P_{repro} = 1.0 \times 0.8 \times 0.75 \times 1.0 = 0.60.$ Only 60% of reruns match bit-for-bit.

L2.2 — Effect of adding a leaky stage

Take the pipeline from L2.1 ( $P = 0.60$ ) and insert a new unseeded shuffle stage with $p = 0.5$ . What is the new $P_{repro}$ , and by what factor did it drop?

Recall Solution

Step — multiply the extra factor in. Why? A new independent AND-stage multiplies. $P_{new} = 0.60 \times 0.5 = 0.30.$ It halved (dropped by factor $0.5$ ). One unseeded stage can silently destroy half your reproducibility.

L2.3 — Solve for a required stage quality

You need $P_{repro} \geq 0.95$ . Three stages already sit at $0.99, 0.99, 0.99$ . What minimum $p$ must a fourth stage have?

Recall Solution

Step — isolate the unknown factor. Why? $P = 0.9 9^{3} \cdot p$ , and we want it $\geq 0.95$ . $0.9 9^{3} = 0.970299, p \geq \frac{0.95}{0.970299} = 0.97908 \dots$ So $p ≳ 0.9791$ — roughly $0.979$ . Even three near-perfect stages leave almost no slack; the fourth must be $\geq 97.9%$ deterministic.

Level 3 — Analysis

L3.1 — Locate the reproducibility leak

A run reproduces only $63%$ of the time. Stages and probabilities: load $= 1.0$ , shuffle $= 0.7$ , train $= 0.9$ , eval $= 1.0$ . (a) Verify the $63%$ . (b) You can fix one stage to $p = 1.0$ . Which single fix raises $P$ the most, and to what value?

Recall Solution

(a) $1.0 \times 0.7 \times 0.9 \times 1.0 = 0.63$ . ✓ (b) Step — try each fix, keep the largest gain. Why? Fixing stage $i$ divides out $p_{i}$ (replaces it with $1$ ), so the new $P$ is $0.63/ p_{i}$ . Dividing by the smallest $p_{i}$ gives the biggest jump.

Fix shuffle ( $p = 0.7$ ): $0.63/0.7 = 0.90$ .
Fix train ( $p = 0.9$ ): $0.63/0.9 = 0.70$ .

Fixing the shuffle (the smallest factor) wins → $P = 0.90$ . Attack the smallest $p_{i}$ first.

L3.2 — Confounded comparison

Run A: lr=0.01, seed=1 → $91.0%$ . Run B: lr=0.001, seed=7 → $92.0%$ . Someone claims "the lower learning rate gave $+ 1%$ ." Is this conclusion valid? What do you do?

Recall Solution

Step 1 — count what changed. Why? A causal claim about lr requires that only lr differs. Here two things changed: lr and seed. The $+ 1%$ could be pure seed noise. Step 2 — control the confounder. Re-run both learning rates with the same fixed seed (and ideally several seeds, comparing the mean). Only then is the lr's effect isolated. Conclusion: the claim is not yet valid — it is a confounded comparison. Tracking the seed is precisely what lets you detect and fix this.

L3.3 — Hash diagnosis

Last week hash(train.csv) = a1b2c3. Today the identical training script logs hash(train.csv) = ff9900 and accuracy dropped from $92%$ to $85%$ . Code SHA is unchanged. Where is the fault, and how did tracking prove it in seconds?

Recall Solution

Step — compare fingerprints. Why? A content hash is a fingerprint: same bytes → same hash, different hash → the bytes changed. Code SHA identical rules out a code change; the data hash differing pinpoints the culprit. Conclusion: the data was silently modified (someone re-exported train.csv). The metric drop is a data problem, not a code bug. Without the logged hash this is a week of blind debugging; with it, a two-second diff.

Level 4 — Synthesis

L4.1 — Design a minimal reproducible-run schema

Design the smallest record that lets a teammate (a) reproduce the run and (b) compare/reuse it. Justify every field directly from the arguments and outputs of $f$ — no decorative fields.

Recall Solution

Start from the run function $(artifacts, metrics) = f (code, data, hyperparameters, environment, seed)$ . The minimal record is exactly its inputs and outputs — nothing else: $Run = (the 5 inputs of f code_sha, data_hash, hyperparameters, env_digest, seed, the 2 outputs of f metrics, artifacts) .$ Note on naming: hyperparameters is the same axis the parent note logs as the config / params field — I keep the full word here to stay tied to $f$ 's argument list. Justification, field by field (each is an argument or output of $f$ , so none is decorative):

drop code_sha → cannot pin the code argument;
drop data_hash → cannot reproduce after data drift (the data argument);
drop hyperparameters → cannot recreate the config argument;
drop env_digest → breaks after a library upgrade (see Docker and Containerization);
drop seed → the training stage has $p_{i} < 1$ ;
drop metrics → cannot compare (output 1);
drop artifacts → cannot reuse the model (output 2).

What about run_id / timestamp / tags? These are not arguments or outputs of $f$ , so by the "no decorative fields" rule they are excluded from this minimal schema. Real trackers (MLflow, W&B) add them purely for lookup and indexing, not for reproduction — they are convenience metadata, and the question asked only for what is strictly required by $f$ .

L4.2 — Hit a determinism budget

Your pipeline load→shuffle→train→eval has $p = (1.0, 0.7, 0.9, 1.0)$ , so $P = 0.63$ . You may apply two fixes, each pushing a stage to $\approx 1.0$ : (i) seed the shuffle, (ii) torch.use_deterministic_algorithms(True) for training. What is $P$ afterward, and what theoretical maximum can this pipeline reach?

Recall Solution

Step — remove the two sub-unity factors. Why these two? They are the only stages with $p < 1$ ; fixing them replaces the offending factors with $\approx 1$ . $P = 1.0 \times 1.0 \times 1.0 \times 1.0 \approx 1.0.$ The theoretical max is $1.0$ (all four stages deterministic). See Random Seeds and Determinism in PyTorch for the exact PyTorch calls and CUBLAS_WORKSPACE_CONFIG.

L4.3 — Cost-aware plan (the 80/20)

You have limited engineering hours. Rank these four actions by reproducibility-incident prevented per hour, cheapest wins first: (A) full bit-for-bit determinism across GPUs, (B) log config + git SHA, (C) log data hash, (D) log the seed. Give the ordering and the reason.

Recall Solution

Order (do first → last): B, C, D, then A. Why? B, C, D are each a one-line log call and together pin four of five axes — the parent's 80/20: cheap tracking prevents ~ $80%$ of "cannot reproduce" incidents. A (bit-for-bit across hardware) is expensive, often infeasible, and buys the last, rarely-needed slice. Rule of thumb: log everything cheap first; chase full determinism only when the metric variance actually breaks a decision.

Level 5 — Mastery

L5.1 — Two pipelines, same average, different reproducibility

Pipeline X: $p = (0.9, 0.9, 0.9, 0.9)$ . Pipeline Y: $p = (1.0, 1.0, 1.0, 0.6)$ . (a) Show both have the same average stage quality. (b) Compute each $P_{repro}$ . (c) Which is more reproducible, and what deep lesson does the gap teach?

Recall Solution

(a) Average X $= (0.9 \times 4) /4 = 0.9$ . Average Y $= (1.0 + 1.0 + 1.0 + 0.6) /4 = 0.9$ . Equal. ✓ (b) $P_{X} = 0. 9^{4} = 0.6561$ . $P_{Y} = 1. 0^{3} \times 0.6 = 0.6$ . (c) X wins ( $0.6561 > 0.6$ ). Lesson: the product is dominated by its worst factor. Many mildly-imperfect stages (X) can beat one badly-broken stage (Y) even at equal average — a single $p_{i} = 0.6$ caps everything. Reproducibility is a weakest-link quantity, not an average one. Kill the smallest $p_{i}$ before polishing the good ones.

L5.2 — Decide under a tolerance

Runs of the same config give accuracies $90.8%, 91.1%, 90.9%, 91.2%$ . Your reproducibility tolerance is $ϵ = \pm 0.5%$ . A colleague says "these don't match, the pipeline is broken." Adjudicate.

Recall Solution

Step — measure the spread against $ϵ$ . Why? Reproducibility means "metric within $\pm ϵ$ ", not bit-identical.

Mean $= (90.8 + 91.1 + 90.9 + 91.2) /4 = 91.0%$ .
Max deviation from mean $= ∣91.2 - 91.0∣ = 0.2% < 0.5%$ . Full range $= 91.2 - 90.8 = 0.4% < 1.0%$ band.

Verdict: every run is inside $\pm 0.5%$ → the pipeline is reproducible by the agreed definition. The colleague is applying a bit-for-bit standard the project never chose.

L5.3 — Combine everything: full diagnosis

A prod retrain scores $89%$ vs the logged $94%$ . From the tracker you see: code_sha identical, data_hash identical, seed identical, but env_digest differs (pandas $1.5 \to 2.0$ ) and the pipeline's train stage has $p = 0.9$ (GPU nondeterminism, tolerance $ϵ = \pm 0.5%$ ). Rank the two suspects and state your action plan.

Recall Solution

Step 1 — separate "in-tolerance noise" from "real regression". GPU nondeterminism at $p = 0.9$ produces run-to-run wobble, but only within $\pm 0.5%$ — it cannot explain a $5%$ drop. So it is not the primary cause. Step 2 — follow the one axis that changed. Diff the tracked axes: code_sha, data_hash, seed are all identical, so those three arguments of $f$ are ruled out. Only env_digest differs (pandas $1.5 \to 2.0$ ). An unpinned environment is the classic silent breaker (the parent's "it worked last Tuesday" catastrophe). Ranking: (1) environment change — primary, and the only changed axis, large enough to explain the full $5%$ ; (2) GPU nondeterminism — secondary, bounded at $\pm 0.5%$ noise, far too small to be the cause. Action: rebuild in the exact logged environment via the Docker digest / locked requirements, retrain, and confirm accuracy returns to $94% \pm 0.5%$ ; then enforce env pinning in CI so it cannot recur.

Recall Final self-check (predict, then reveal)

Weakest-link rule ::: reproducibility is a product, so it is dominated by the smallest $p_{i}$ — fix that stage first. Confounded comparison fix ::: change one variable at a time; hold the seed fixed to isolate a hyperparameter's effect. Reproducibility definition to use in practice ::: metrics within $\pm ϵ$ , not bit-identical weights. First suspect for a big prod-vs-logged gap ::: the tracked axis that changed (often environment), not the scariest-sounding one. Why does the product formula assume independence ::: whole-pipeline reproduction is an AND of stage events; multiplying is exact only when those events do not share a hidden dependency (e.g. one global RNG).