Experiment tracking and reproducibility: Worked examples

Before any example, we earn the four objects the whole page rests on: the function $f$ , the per-stage dial $p_{i}$ , the whole-chain number $P_{repro}$ , and the tolerance $ϵ$ .

Definition The three symbols we reuse everywhere

$p_{i}$ = the probability that stage number $i$ of the pipeline produces the exact same output when fed the exact same input. It is a number between $0$ and $1$ . $p_{i} = 1$ means "perfectly deterministic"; $p_{i} = 0.5$ means "a coin-flip". Think of it as the trustworthiness dial of one step. The little subscript $i$ is just an index — " $p_{3}$ " means "the dial of the 3rd stage". When we speak of a generic stage without a specific number, we write plain $p$ (same meaning, unnamed slot); the moment we count stages we put the $i$ back.
$n$ = the number of stages in the pipeline (how many boxes the data passes through, e.g. load → shuffle → train → eval gives $n = 4$ ).
$P_{repro} = p_{1} \times p_{2} \times \dots \times p_{n}$ = the trustworthiness of the whole chain of $n$ stages, because every stage must behave for the run to repeat. Multiplying is the "AND of independent events" — like needing every link of a chain to hold.

Figure — Experiment tracking and reproducibility

Figure s01 — the pipeline as a chain of links. Read it left to right: four labelled boxes (load, shuffle, train, eval) each carry their own dial $p_{i}$ . Cyan boxes are perfect links ( $p = 1.0$ ); the amber box is the one weak link ( $p = 0.7$ ). The formula printed above them multiplies the four dials into a single number, $P_{repro} = 0.63$ here. Concrete takeaway: the chain is exactly as strong as the product of its links, so one amber link drags the whole product down while three perfect links add nothing. Almost every example below is just "spot the amber link".

The scenario matrix

We enumerate every class of case this topic throws. Each later example is tagged with the cell it covers.

#	Case class	Meaning	Example that covers it
C1	Perfect chain	every $p_{i} = 1$	Ex 1
C2	One weak link	exactly one $p_{i} < 1$	Ex 2
C3	Many weak links	several $p_{i} < 1$ compounding	Ex 3
C4	Degenerate input $p_{i} = 0$	a truly non-deterministic stage	Ex 4
C5	Limiting behaviour $n \to \infty$	long pipeline, each link near-perfect	Ex 5
C6a	Zero variables changed	identical inputs, imperfect pipeline	Ex 6
C6b	Confounded comparison	two variables changed at once	Ex 7
C6c	Controlled experiment	exactly one variable changed	Ex 8
C7	Silent data drift	inputs to $f$ changed unnoticed	Ex 9
C8	Tolerance vs. bit-identity	metric within $\pm ϵ$ , not identical weights	Ex 10
C9	Real-world word problem	rebuild a 6-month-old model	Ex 11
C10	Exam-style twist	"improve $P$ cheapest" optimisation	Ex 12

The "signs / quadrants" of this subject are the values a $p_{i}$ can take ( $= 1$ , between $0$ and $1$ , exactly $0$ ) and how many variables changed in a comparison — the three cases $0$ (C6a), $1$ (the valid controlled experiment, C6c), and more than $1$ (C6b). We visit each.

The worked examples

Ex 1 — The perfect chain (cell C1)

Ex 2 — Exactly one weak link (cell C2)

Worked example The unseeded shuffle

Stages: load $p = 1.0$ , shuffle $p = 0.7$ (someone forgot the seed), train $p = 1.0$ , eval $p = 1.0$ . Find $P_{repro}$ , and say what fixing the shuffle would do.

Forecast: three of the four stages are perfect. Will $P$ be close to $1$ , or will the one weak link pull it down hard?

Step 1 — multiply. Why? Same rule; the perfect stages contribute a factor of $1$ and simply "pass through". $P_{repro} = 1.0 \times 0.7 \times 1.0 \times 1.0 = 0.7.$

Step 2 — read the lesson. Why? Notice $P$ equals the single weak factor. When only one link is imperfect, the whole chain inherits exactly that link's value. Figure s01 shows this: one amber $0.7$ link, three unbreakable cyan links.

Step 3 — apply the fix. Why? Seeding the shuffle pushes its $p_{i} \to 1$ , so the product becomes $1.0 \times 1.0 \times 1.0 \times 1.0 = 1.0$ .

Verify: the fix raised $P$ by $1.0 - 0.7 = 0.3$ , and that gain came entirely from the one link we touched — matching the " $P$ inherits the single weak factor" claim. See Random Seeds and Determinism in PyTorch for how to seed it.

Ex 3 — Many weak links compound (cell C3)

Worked example Two sources of drift at once

Pipeline: shuffle $p = 0.7$ (unseeded) and train $p = 0.9$ (GPU nondeterminism), the other two stages $p = 1.0$ . This is the parent note's headline example — re-derive it, then decompose the loss.

Forecast: you might expect $P \approx 0.9$ or $\approx 0.7$ . It is actually lower than both. Why?

Step 1 — multiply all four. Why? Every stage must behave simultaneously, so both sub-unity factors bite. $P_{repro} = 1.0 \times 0.7 \times 0.9 \times 1.0 = 0.63.$

Step 2 — see why it is below either factor. Why? Multiplying two numbers below 1 gives something below both: $0.63 < 0.7$ and $0.63 < 0.9$ . Weak links compound, they do not average.

Step 3 — fix both. Why? Seed the shuffle ( $0.7 \to 1.0$ ) and enable deterministic kernels ( $0.9 \to\approx 1.0$ ). Then $P \approx 1.0 \times 1.0 \times 1.0 \times 1.0 = 1.0$ .

Verify: $1.0 \cdot 0.7 \cdot 0.9 \cdot 1.0 = 0.63$ , so only $63%$ of reruns match; the two fixes remove exactly the two sub-unity factors that dominated the product.

Figure s02 — compounding, as a bar chart. Each of the four stages is a bar of height $p_{i}$ (two cyan perfect stages, two amber sub-unity ones), and the tall white bar on the right is the whole-pipeline value $P_{repro} = 0.63$ . The dashed amber line marks that $0.63$ : notice the white bar sits below both amber stage bars ( $0.7$ and $0.9$ ). This is the picture of "weak links compound, they do not average" — the product dives beneath the smallest factor, not toward the mean.

Ex 4 — A degenerate stage, $p_{i} = 0$ (cell C4)

Worked example The catastrophic single stage

Suppose one stage is fully non-deterministic — a race condition that never repeats, so $p = 0.0$ . The other three are perfect. What is $P_{repro}$ , and what does it tell you about where to spend effort?

Forecast: three perfect stages… surely $P$ is still high?

Step 1 — multiply. Why? The multiplication rule does not care that the rest are perfect. $P_{repro} = 1.0 \times 0.0 \times 1.0 \times 1.0 = 0.0.$

Step 2 — interpret the zero. Why? A single $0$ factor annihilates the whole product — like a broken link that guarantees the chain snaps. No amount of care elsewhere can rescue it. This is the "cap" the parent note warns about, taken to its extreme.

Step 3 — prioritise. Why? This proves that effort should go to the lowest $p_{i}$ first, never the already-high ones. Fixing a $1.0 \to 1.0$ stage buys nothing.

Verify: $P = 0$ means no rerun ever matches — which is exactly what a stage that never repeats ( $p = 0$ ) should force, no matter how good the other three are.

Ex 5 — Limiting behaviour: many near-perfect stages (cell C5)

Worked example The long pipeline

A big production pipeline has $n = 50$ stages, each almost perfect at $p_{i} = 0.99$ . Guess $P_{repro}$ — is "almost perfect $\times$ 50" still almost perfect?

Forecast: $0.99$ feels basically like $1$ . Will $P$ stay near $1$ , or leak?

Step 1 — use the power form. Why? All $n = 50$ factors are equal, so the product collapses to a single power: $P_{repro} = 0.9 9^{50} .$

Step 2 — evaluate. Why? To see the actual erosion, not just trust the feeling. $0.9 9^{50} \approx 0.605.$

Step 3 — read the limiting lesson. Why? As $n$ grows, even tiny per-stage imperfection compounds toward $0$ : $lim_{n \to \infty} 0.9 9^{n} = 0$ . Long pipelines demand higher per-stage $p_{i}$ , not the same. To keep $P_{repro} \geq 0.9$ over 50 stages you need each $p_{i} \geq 0. 9^{1/50} \approx 0.9979$ .

Verify: raising the required $0.9979$ back to the 50th power returns $\approx 0.9$ , confirming the threshold; and $0.605$ sits far below the naïve $0.99$ , confirming the leak is real.

Figure s03 — erosion curves $p^{n}$ . The horizontal axis is the number of stages $n$ ; each curve plots $P_{repro} = p^{n}$ for a fixed per-stage $p$ (cyan $0.99$ , amber $0.95$ , white $0.999$ ). Every curve slides downhill as $n$ grows — the whole point is that no value of $p$ below $1$ stays flat. The amber dot marks $0.9 9^{50} \approx 0.605$ at the dotted $n = 50$ line: a "99% reliable" stage, chained 50 times, is barely a coin-flip better than even. This is why CI-CD for Machine Learning pipelines pin each stage hard.

Ex 6 — Zero variables changed: identical inputs, imperfect pipeline (cell C6a)

Worked example Same config, different result

You run the exact same config twice — same code, data, hyperparameters, environment, seed — yet accuracy comes back $92.0%$ then $91.4%$ . Nothing in $f$ 's five inputs changed. How is that possible, and is it a bug?

Forecast: if every input is identical, the outputs must be identical… right?

Step 1 — separate "inputs pinned" from "pipeline deterministic". Why? Pinning $f$ 's five inputs guarantees the right-hand side is fixed, but the function itself can still contain a non-deterministic stage (e.g. a GPU reduction with $p_{i} < 1$ ). Fixed inputs $\neq =$ fixed output when some $p_{i} < 1$ .

Step 2 — locate the culprit via $P_{repro}$ . Why? A gap between two identical-input runs is direct evidence that $P_{repro} < 1$ , i.e. at least one stage has $p_{i} < 1$ . Here the train stage's GPU nondeterminism is the amber link.

Step 3 — decide bug-or-not against tolerance $ϵ$ . Why? Zero changed inputs means this is not a confounded comparison; the wobble is the pipeline's own residual randomness. Judge it against the tolerance $ϵ$ we defined: $∣92.0 - 91.4∣ = 0.6%$ .

Verify: $∣92.0 - 91.4∣ = 0.6%$ ; with a tight $ϵ = 0.5%$ this run fails tolerance and you must raise the train stage's $p_{i}$ (deterministic kernels). With zero inputs changed, the diagnosis is unambiguous: the leak is inside $f$ , not in the inputs.

Ex 7 — Confounded comparison: more than one variable changed (cell C6b)

Worked example Is the lower learning rate really better?

Run A: lr=0.01, seed=1 → $91.0%$ . Run B: lr=0.001, seed=7 → $92.0%$ . Your teammate says "lower lr wins by $1%$ ." Are they right?

Forecast: the number went up — case closed? Look again at what changed.

Step 1 — measure the gap. Why? Before attributing a cause we must quantify the effect we are trying to explain, so we compute the difference in the two metrics. $Δ = 92.0% - 91.0% = 1.0%.$

Step 2 — count changed variables. Why? A controlled experiment changes exactly one input of $f$ ; only then is the output difference attributable to that input. Here both lr and seed changed — two variables — so the $1.0%$ could be from either.

Step 3 — re-run with the seed held fixed. Why? Fixing seed to a constant turns it from a confounder into a control, so any remaining difference is caused only by lr. Tracking the seed is precisely what makes this fix possible — see Hyperparameter Tuning.

Verify: the effect size is $Δ = 1.0%$ , but the number of changed variables is $2 \neq = 1$ , so the comparison is invalid as stated — your teammate is not yet justified.

Ex 8 — Controlled experiment: exactly one variable changed (cell C6c)

Worked example The valid comparison, done right

After Ex 7 you re-run cleanly. Run A: lr=0.01, seed=1 → $91.0%$ . Run C: lr=0.001, seed=1 → $92.5%$ . Only lr changed; the seed is held at $1$ . With a tolerance $ϵ = 0.5%$ , can you now credit the lower lr?

Forecast: exactly one variable moved this time. Predict two things: (a) how many changed variables must a valid comparison have, and (b) is $92.5% - 91.0%$ big enough to beat the $ϵ = 0.5%$ noise floor?

Step 1 — count changed variables. Why? Causal attribution is only legal when exactly one input of $f$ differs. Here code, data, env, and seed are all identical; only lr moved. Count $= 1$ ✓ — a controlled experiment.

Step 2 — measure the effect. Why? With the comparison now valid, the metric gap is the causal effect of the lr change. $Δ = 92.5% - 91.0% = 1.5%.$

Step 3 — test the effect against the noise floor $ϵ$ . Why? A difference only counts as real if it exceeds the pipeline's own wobble $ϵ$ ; otherwise it could be residual randomness (Ex 6). Compare $Δ = 1.5%$ with $ϵ = 0.5%$ : since $1.5% > 0.5%$ , the effect clears the floor.

Verify: changed variables $= 1$ (valid) and $Δ = 1.5% > ϵ = 0.5%$ , so the lower lr genuinely helped — this is exactly the diagnosis Ex 7 could not make because its seed was a confounder.

Ex 9 — Silent data drift (cell C7)

Worked example The hash that changed by itself

Last week: hash(train.csv) = a1b2. Today the same script logs hash(train.csv) = ff99, and accuracy dropped from $94%$ to $88%$ . Is this a code bug or a data problem?

Forecast: accuracy dropped, so your instinct is to open the training code and hunt a bug. Predict instead: which of $f$ 's five inputs does a changed hash point at, and can the code even be the culprit if its SHA is unchanged?

Step 1 — compare the fingerprints. Why? A hash is a fingerprint of the file's bytes; identical files share a hash, different files do not. $a 1 b 2 \neq = f f 99$ , so the "same" file is not the same bytes.

Step 2 — localise the fault by elimination. Why? Exactly one input of $f$ changed (the data), while code_sha is identical. If code is unchanged it cannot explain a new result; by elimination the $6%$ drop is a data problem, not a code problem — someone re-exported train.csv.

Step 3 — restore the pinned data. Why? Checking out the dataset version whose hash is a1b2 restores the original input, so $f$ receives exactly what it did before. This is what Data Versioning with DVC exists to do.

Verify: the accuracy delta is $94% - 88% = 6%$ and only the data hash changed, so the fault is isolated to data — a seconds-long diagnosis with a logged hash, a lost week without.

Ex 10 — Tolerance, not bit-identity (cell C8)

Worked example Two GPUs, "different" weights

You retrain the same config on a different GPU. Weights differ in the 6th decimal place; accuracy is $92.0%$ vs $92.03%$ . Your reproducibility target is "metric within $\pm ϵ$ " with $ϵ = 0.5%$ . Reproducible or not?

Forecast: the weights are literally different numbers — has reproducibility failed?

Step 1 — pick the right definition. Why? Bit-for-bit identical weights are often infeasible across hardware (floating-point reductions differ). The useful target is that the metric reproduces within a tolerance $ϵ$ .

Step 2 — test the metric gap against $ϵ$ . Why? That is the operational reproducibility criterion we defined. $∣92.03% - 92.0%∣ = 0.03% \leq ϵ = 0.5%.$

Step 3 — conclude. Why? The gap sits inside tolerance, so by the sensible definition the run is reproducible, even though the raw weights differ.

Verify: $∣92.03 - 92.0∣ = 0.03 \leq 0.5$ , so the metric-tolerance test passes even though bit-identity would have (wrongly) reported failure.

Ex 11 — Real-world: rebuild a 6-month-old model (cell C9)

Worked example The word problem

A regulator asks you to reproduce a model you trained 6 months ago. Your tracker logged: code_sha=3f9a, data_hash=a1b2, params={lr:0.01}, env=docker@sha256:d4e5, seed=42. Which of the five axes of $f$ do you still need, and what is your rebuild probability if each restored stage is deterministic at $p = 1.0$ except the GPU train stage at $p = 0.95$ ?

Forecast: all five inputs are logged — is the rebuild guaranteed?

Step 1 — map logged fields to $f$ 's five inputs. Why? $f (code, data, params, env, seed)$ ; reproducing needs every argument pinned. Check them off: code ✓ (3f9a), data ✓ (a1b2), config ✓ (lr:0.01), env ✓ (Docker digest, see Docker and Containerization), seed ✓ (42). All five present.

Step 2 — compute the rebuild probability. Why? Even with inputs pinned, residual GPU nondeterminism keeps one $p_{i} < 1$ . $P_{repro} = 1.0 \times 1.0 \times 1.0 \times 1.0 \times 0.95 = 0.95.$

Step 3 — decide if that suffices. Why? With a $\pm ϵ$ tolerance target (Ex 10), $0.95$ means $95%$ of rebuilds land within tolerance — acceptable for a regulator, and pushable to $\approx 1$ with deterministic kernels. Register the rebuilt model via Model Registry and Deployment.

Verify: all 5 axes are logged and $P_{repro} = 0.95$ , so nothing was un-pinned — the residual $5%$ is GPU wobble, not a lost recipe.

Ex 12 — Exam twist: improve $P$ the cheapest way (cell C10)

Worked example The optimisation twist

A pipeline has stages with $p = [1.0, 0.7, 0.9, 1.0]$ (current $P = 0.63$ ). You have budget to push exactly one stage to $1.0$ . Which stage, and what new $P$ results? Prove your choice beats the alternative.

Forecast: should you fix the $0.9$ stage or the $0.7$ stage? Intuition says "the worst one" — let's prove it.

Step 1 — try fixing the $0.7$ stage. Why? Set that factor to $1.0$ and multiply to get the counterfactual $P$ . $P_{0.7 \to 1} = 1.0 \times 1.0 \times 0.9 \times 1.0 = 0.9.$

Step 2 — try fixing the $0.9$ stage instead. Why? We need the competing option's $P$ to compare against Step 1. $P_{0.9 \to 1} = 1.0 \times 0.7 \times 1.0 \times 1.0 = 0.7.$

Step 3 — pick the larger, and prove the rule. Why? $0.9 > 0.7$ , so fixing the lowest $p_{i}$ (the $0.7$ shuffle) gives the bigger jump: gain $0.9 - 0.63 = 0.27$ versus $0.7 - 0.63 = 0.07$ . The reason is structural: raising a factor from $p$ to $1$ multiplies the product by $\frac{1}{p}$ , and $\frac{1}{0.7} \approx 1.43$ is a bigger boost than $\frac{1}{0.9} \approx 1.11$ . Rule: always attack the smallest $p_{i}$ first — it removes the largest drag from the product.

Verify: $P_{0.7 \to 1} = 0.9$ and $P_{0.9 \to 1} = 0.7$ , with gains $0.27 > 0.07$ , so fixing the lowest link wins — the optimisation proof is complete.

Active recall

Recall Predict, then reveal.

Which single value of $p_{i}$ makes $P_{repro} = 0$ no matter the other stages? ::: $p_{i} = 0$ (a fully non-deterministic stage annihilates the product). If all $n$ stages have $p = 0.99$ , why is $P$ not $\approx 0.99$ ? ::: They multiply: $0.9 9^{n}$ erodes toward $0$ as $n$ grows; for $n = 50$ it is $\approx 0.605$ . Two runs share all five inputs of $f$ yet differ. Bug or not? ::: Not necessarily — fixed inputs do not fix the output if some stage has $p_{i} < 1$ ; judge the gap against tolerance $ϵ$ . What does the tolerance $ϵ$ mean, and how do you choose it? ::: The largest metric difference you still call "reproduced"; set it to a gap that would not change any decision you make. A comparison changed both lr and seed. Why is it invalid? ::: Two variables changed, so the metric difference cannot be attributed to lr alone — the seed is a confounder. A comparison changed only lr (seed fixed) and $Δ = 1.5% > ϵ = 0.5%$ . Verdict? ::: Valid controlled experiment and effect beats the noise floor — the lr change genuinely helped. Given $p = [1.0, 0.7, 0.9, 1.0]$ and one fix, which stage do you fix? ::: The $0.7$ stage (the lowest $p_{i}$ ) — it lifts $P$ from $0.63$ to $0.9$ .