Experiment tracking and reproducibility: Question bank

The recurring model behind every question is the training run as a function: $(artifacts, metrics) = f (code, data, hyperparameters, environment, seed)$ Reproducing a result means pinning all five arguments; comparing runs means isolating a change in one of them.

True or false — justify

A git commit fully guarantees a reproducible model

False. Git pins only the code argument of

f

; data, library versions, and the seed are unpinned, so the same commit can still yield different models. See Data Versioning with DVC and Docker and Containerization for the other axes.

Setting np.random.seed(42) makes a PyTorch training run deterministic

False. NumPy has its own RNG; PyTorch and CUDA have theirs, and GPU parallel reductions can be nondeterministic even when seeded. You must seed all generators — see Random Seeds and Determinism in PyTorch.

Reproducibility requires bit-for-bit identical weights across machines

False. That's often infeasible across hardware/driver versions. The useful definition is that metrics and conclusions reproduce within a tolerance

\pm ϵ

, not that weight tensors are byte-identical.

Repeatability and reproducibility mean the same thing

False. Repeatability is same team, same setup → same result; reproducibility is different team, same description → same result. The second is stricter because it tests whether your written description is complete.

If two runs use different seeds, any metric difference between them is caused by the hyperparameters

False. Changing the seed alone already shifts metrics (seed noise). A difference is attributable to hyperparameters only when the seed is held fixed, so seed becomes a confounder otherwise.

Logging the final accuracy number is enough to reproduce a run later

False. A metric with no inputs attached is an orphan metric — you can't recreate or explain it. Metrics must be stored together with the full run tuple (id, params, code SHA, data hash, env, seed).

A deterministic pipeline of many stages is only as reproducible as its weakest stage

True. With independent stages,

P_{repro} = \prod_{i} p_{i}

, so one stage at

p_{i} = 0.5

halves the whole product no matter how perfect the others are.

Full bit-for-bit determinism is always worth the engineering cost

False. The 80/20 rule says logging config + git SHA + data hash + seed prevents ~80% of "cannot reproduce" incidents cheaply; forcing the last fraction of determinism is expensive and often unnecessary.

Storing a data hash is redundant if the data file has a fixed filename

False. A filename is a label, not a fingerprint; a file can be silently re-exported while keeping the same name. The hash detects that the contents changed even when the name did not.

Spot the error

"I logged lr, epochs, and accuracy to a spreadsheet, so my experiment is tracked." — what's missing?

The code SHA, data hash, environment (library/CUDA versions), and seed are absent — four of the five inputs of

f

. Without them the row is not reproducible; it's an orphan metric with a couple of decorations.

"Runs A and B differ only in learning rate, so the 1% gap proves the lower lr is better." — find the flaw

The claim silently assumes the seed and everything else were identical. If the seed also changed, the gap could be pure seed noise; you must re-run with the seed fixed to isolate the lr's causal effect.

"My pipeline has stages with

p = 1.0, 1.0, 0.9

, so it's basically deterministic at 90%." — what's wrong with calling it "basically deterministic"?

90% means one in ten reruns disagree — that's not "basically deterministic", it's a real leak. The

p = 0.9

stage (likely GPU nondeterminism) should be pushed toward 1 with deterministic kernels before you rely on the pipeline.

"I set torch.manual_seed(0) and still got different results, so seeding is pointless." — where's the reasoning error?

Seeding the CPU RNG isn't the whole story: the DataLoader workers, CUDA RNG, and nondeterministic GPU algorithms remain unseeded. The fix is to seed all generators and call torch.use_deterministic_algorithms(True), not to abandon seeding.

"We can't reproduce last month's model, so the code must have a bug." — what jump is unjustified?

A reproduction failure points to an unpinned axis, not necessarily a code bug. Data may have been overwritten, a library upgraded, or the seed changed; the diagnosis comes from diffing the run tuple, not from assuming code.

"I froze requirements.txt, so my environment is pinned." — what can still drift?

An unpinned (or loosely pinned) requirements.txt lets transitive dependency versions and system-level pieces like CUDA/cuDNN drift. A lock file plus a Docker image digest pins the environment far more tightly.

Why questions

Why do stage-level determinism probabilities multiply rather than add?

Because whole-pipeline reproducibility needs every stage reproducible simultaneously — a logical AND of independent events, and independent AND-probabilities multiply.

Why is tracking the seed the prerequisite for a fair A/B comparison of hyperparameters?

Only a logged, fixed seed lets you hold randomness constant so that the one variable you changed is the sole difference; without it the seed becomes an uncontrolled confounder.

Why does the derived run schema contain exactly the fields it does, and nothing decorative?

The five input fields are precisely the five arguments of

f

(needed to recreate), metrics and artifacts are its two outputs (needed to compare and reuse), and run_id/timestamp give identity (needed to find it). Every field maps to one of those needs.

Why is a silent data change so dangerous compared to a code change?

Code changes are visible in version control and reviews, but a re-exported data file leaves no diff unless you fingerprint it. The data hash turns an invisible change into a detectable one, saving days of misdirected debugging.

Why can two teams following the same written description still fail to reproduce a result?

Because the description omitted an axis of

f

— often the environment or seed — which is exactly what distinguishes reproducibility (harder) from mere repeatability on the original machine.

Why does the parent call ML "empirical science run at industrial speed", and how does that motivate tracking?

You run hundreds of tiny experimental variations fast, so without a disciplined log you literally cannot recall which change caused which result — tracking is the lab notebook that makes the science attributable.

Edge cases

If a pipeline has a single nondeterministic stage with

p = 0

, what is the whole-pipeline reproducibility?

Zero, because the product

\prod_{i} p_{i}

contains a factor of 0. A totally nondeterministic stage makes the entire pipeline never reproduce, regardless of the other stages.

If every stage is perfectly deterministic (

p_{i} = 1

for all

i

), what is

P_{repro}

, and does adding more such stages hurt?

It is exactly 1, and adding more stages with

p = 1

leaves it at 1 — multiplying by 1 never lowers the product, so deterministic stages are "free" for reproducibility.

What does reproducibility mean when the hardware itself changes (different GPU model)?

Bit-for-bit weight identity is generally unattainable, so you fall back to the tolerance definition: the run is reproducible if metrics land within

\pm ϵ

of the original despite different low-level kernels.

Is a run with a logged seed but an unlogged data hash reproducible?

No — pinning one axis while leaving another free still lets

f

's output drift when the data changes. Reproducibility needs all five arguments pinned simultaneously, not most of them.

What is the reproducibility status of an experiment whose environment was captured only as "latest" tags on dependencies?

Effectively unpinned — "latest" resolves to whatever version exists at install time, so the environment axis silently varies. It should be a locked version set or a Docker image digest, not a floating tag.

Recall One-line self-test before you leave

How many axes must be pinned to reproduce, and name the one git cannot pin? ::: Five; git cannot pin data, environment, or seed — pick any one.
Reproducibility is defined against metrics within == $\pm ϵ$ ==, not identical weights.
A single $p_{i} < 1$ stage caps the whole pipeline's reproducibility via the product rule.