Visual walkthrough — Experiment tracking and reproducibility

This is the flagship visual walkthrough for the parent topic. Read it after the parent, or as your way in.

---

Step 1 — What is a "run"? A machine that eats inputs and coughs up outputs

WHAT. Before we can talk about reproducing a run, we must agree on what a run is. A training run is just a machine: you feed it some ingredients on the left, turn the crank, and out comes a trained model plus its measured scores on the right.

WHY. You cannot ask "will I get the same thing twice?" until you know exactly what goes in. If two runs differ, the difference must live in one of the inputs — there is nowhere else for it to hide. So the very first act of reproducibility is listing every input.

PICTURE. Look at the figure. On the left, five labelled arrows feed the box — these are the five things the parent called the arguments of $f$: code, data, hyperparameters, environment, seed. On the right, two arrows leave: artifacts (the model) and metrics (the scores).

$(\underset{\text{the model file}}{\underbrace{\text{artifacts}}},\underset{\text{accuracy, loss}}{\underbrace{\text{metrics}}})=f(\underset{\text{git SHA}}{\underbrace{\text{code}}},\underset{\text{hash}}{\underbrace{\text{data}}},\underset{\text{settings}}{\underbrace{\text{params}}},\underset{\text{lib versions}}{\underbrace{\text{env}}},\underset{\text{one number}}{\underbrace{\text{seed}}})$

Each symbol is a knob. If a knob moves without you noticing, the output moves too. Pinning all five knobs is what DVC (for data), Docker (for env), and git (for code) do for you.

---

Step 2 — A run is not one machine, it's a chain of machines

WHAT. Zoom inside the box from Step 1. A real training run is not one operation — it is a line of smaller machines, each passing its output to the next: load the data → shuffle it → train the model → evaluate it.

WHY. Reproducibility is not a property of the whole box as a mystery. It is a property that either survives or dies at each link in the chain. To reason about the whole, we must reason about one link at a time — that is the only way the algebra will come out clean.

PICTURE. Four boxes in a row, each with an arrow into the next. We call them stages $1,2,3,4$ — in general, stages $1$ through $n$. The output of stage $i$ becomes the input of stage $i+1$.

$\text{data}\stackrel{\text{stage }1}{\to }\bullet \stackrel{\text{stage }2}{\to }\bullet \stackrel{\text{stage }3}{\to }\bullet \stackrel{\text{stage }4}{\to }\text{result}$

Here $n$ is just "how many stages" — count the boxes. The $\bullet$ dots are the intermediate results handed forward.

---

Step 3 — What does "deterministic" mean for ONE stage? Meet $p_i$

WHAT. Take a single stage in isolation. Feed it the exact same input twice. Does it produce the exact same output twice? Sometimes yes, sometimes no. We turn that yes/no into a number between 0 and 1.

WHY. We need a measurable handle, not a vibe. So we define, for stage number $i$:

PICTURE. One stage, fed the same green input arrow twice. Top run and bottom run. If the two output arrows are identical we score a match (✓); the fraction of matches over many tries is $p_i$. A missing seed is exactly what drops $p_i$ below $1$.

---

Step 4 — Two stages in a row: why the chances multiply

WHAT. Now put two stages back to back. For the pair to repeat, stage 1 must repeat and stage 2 must repeat. We want the probability of that combined event.

WHY multiplication and not addition? This is the crux, so slow down. "AND" of two independent events multiplies; "OR" would add. Reproducing the pair is an AND: we need both to go right, simultaneously. If they were unrelated and you added, two 90%-stages would give 180% — nonsense. Multiplication is the tool that answers "both, at once?"

PICTURE — a probability tree. From the start, stage 1 either repeats (probability $p_1$, blue branch) or fails ($1-p_1$, pink). Only along the blue branch does stage 2 even get its identical input, so we then split again: repeats ($p_2$) or fails. The single branch where both repeat has probability $p_1\times p_2$ — trace the pale-yellow highlighted path.

$P(\text{stage 1 AND stage 2 both repeat})=\underset{\text{first link holds}}{\underbrace{p_1}}\times \underset{\text{second link holds}}{\underbrace{p_2}}$

---

Step 5 — All $n$ stages: the parent's formula, now earned

WHAT. Extend Step 4 down the whole chain. The pipeline reproduces only if link 1 holds AND link 2 AND … AND link $n$ — every single one, at the same time.

WHY. There is no partial credit. If any link breaks, the wrong intermediate result flows downstream and the final model differs. One AND across all links → one long product.

PICTURE. The chain from Step 2, each link now stamped with its own $p_i$, and a big $\times$ sitting between them collapsing into a single answer at the end.

---

Step 6 — The killer consequence: the weakest link caps everything

WHAT. Because every factor is $\le 1$, multiplying by a small factor can only shrink the total. So the smallest $p_i$ sets a ceiling that no amount of care elsewhere can lift.

WHY it matters. This is the whole moral of reproducibility. You can have four perfect stages at $p=1.0$ and one lazy stage at $p=0.5$ — the product is $0.5$. Your pipeline reproduces half the time, and polishing the already-perfect stages does nothing.

PICTURE. Five bars at heights $1.0,1.0,0.5,1.0,1.0$. A dashed pink line drops from the shortest bar all the way across to the product — the ceiling. Fixing that one bar (seed it → $p\to 1$) is the only move that raises the ceiling.

---

Step 7 — Degenerate & edge cases (never leave a scenario unshown)

WHAT. We check the corners of the formula so no reader hits a surprise.

WHY. A formula you trust must behave sanely at its extremes. Four corners:

PICTURE. Four mini-panels, one per corner, each showing the bar picture and its resulting ceiling: all-tall→1, one-zero→floor, single-bar→itself, no-bars→1.

---

The one-picture summary

Everything above compressed: five inputs feed a chain of stages; each stage has a repeat-chance $p_i$; the chain reproduces only when all hold, so the chances multiply; the smallest chance is the ceiling; seed everything to push each $p_i\to 1$.

Recall Feynman retelling — explain the whole walkthrough to a friend

A training run is a machine with five knobs feeding in and two things dropping out (Step 1). Inside, it's really a line of little machines, one after another (Step 2). For each little machine I ask: "fed the same thing twice, do you do the same thing twice?" — that chance is $p_i$ (Step 3). For the whole line to repeat, every machine must repeat at once, and "both at once" means I multiply their chances, not add them (Step 4). Multiply all $n$ of them and I get $P_{\text{repro}}=\prod p_i$ (Step 5). Since every chance is at most 1, the smallest one is a ceiling nobody can beat — fix the worst machine or nothing improves (Step 6). Corners: all-perfect gives 1, one-dead gives 0, one machine gives itself, no machines gives 1 (Step 7). The cure is to seed every random generator and pin every knob, dragging each $p_i$ up to 1.

Recall Predict-then-verify checks

• Why multiply instead of add? ::: The pipeline repeats only if every stage repeats — that's an AND of independent events, and AND multiplies.
• Stages $1.0,0.8,1.0,0.5$ — what is $P$? ::: $1.0\times 0.8\times 1.0\times 0.5=0.40$.
• If one stage has $p=0$, what is $P$? ::: $0$ — zero annihilates the whole product.
• Which stage should you fix first? ::: The one with the smallest $p_i$ (the ceiling), not the average.
• What does ${\prod }_{i=1}^n$ mean in words? ::: Multiply the terms together for $i$ from $1$ to $n$.

Where next: the values of $p_i$ come from choices in Random Seeds and Determinism in PyTorch (the seed knob), Docker and Containerization (the env knob), and Data Versioning with DVC (the data knob). Getting to $P_{\text{repro}}\approx 1$ is what makes Model Registry and Deployment and CI-CD for Machine Learning trustworthy. Compare runs with Hyperparameter Tuning only after the seed is pinned — otherwise seed noise masquerades as a real effect.