Exercises — Model versioning and registries

• (a) Immutable — the weights ARE the artifact; changing them makes a different version.
• (b) Immutable — the hash is computed from the bytes, so it is fixed the moment the bytes are.
• (c) Mutable — Production is a label that slides from one version to another on each promotion.
• (d) Mutable — champion is an alias (a nickname pointer); flipping it is the whole trick behind fast rollback.
• (e) Immutable — the metric is a measurement of a fixed version on a fixed eval set; it is stored once and never edited. (You add a new metric row for a new eval set; you don't overwrite the old one.)

Rule of thumb: anything computed from the bytes is immutable; anything that points at a version is mutable.

The 4-tuple is $V=({\text{code}}_{\text{hash}},{\text{data}}_{\text{hash}},\text{params},\text{seed})$. Missing: data_hash and params (hyperparameters).

• Without data_hash: retraining on a silently updated dataset gives different weights even with the same code + seed.
• Without params: a different learning rate / depth / epochs gives a completely different model. Code being identical does NOT pin the config passed to that code.

• (a) MINOR — same interface, retrained, metrics moved.
• (b) PATCH — serving/wrapper fix, weights unchanged.
• (c) MAJOR — the input schema changed; every caller must update. This breaks the contract.

• v5: measured on d17 — the same eval set as v4. $0.930>0.910$, and the comparison is apples-to-apples. Valid improvement.
• v6: auc=0.955 looks higher, but it was measured on d21, a different eval set. 0.955 vs 0.910 is apples vs oranges — the numbers are not comparable. Not a valid comparison until you re-score v6 on d17.

Only v5 is a defensible challenger. The absolute size of v6's number is a distraction.

Ordered operations (all are pointer/label writes):

1. set_alias("champion", "v5")
2. v5.stage = "Production"
3. v4.stage = "Archived"

Artifacts moved: 0. Every operation edits a label or a pointer. The weight bytes of v4 and v5 sit untouched in storage — that immovability is exactly why the swap is atomic and instant, and why v4 is still there for rollback.

(a) Redeploy time $=\frac{800\text{ MB}}{50\text{ MB/s}}=16\text{ s}$. (b) Alias flip $=0.020\text{ s}$. (c) Speed-up $=\frac{16}{0.020}=800\times$.

This is the quantitative version of "rollback is a pointer flip, not a data move." The artifact never travels, so rollback cost is independent of model size.

(a) Rebuild:

1. git checkout b12 (pins code).
2. Load data snapshot s09 (pins the training data).
3. Set seed = 42 and params = {depth:6, lr:0.1}.
4. Run the (deterministic) training pipeline.

(b) Compute the content hash of the resulting artifact and check it equals 9c1a.... Because the full 4-tuple was pinned and the pipeline is deterministic, you get byte-identical weights, so the hashes match — that equality is the proof.

(c) Missing data_hash / data snapshot s09: you could rerun the exact code with the exact seed and params but on today's data, which has drifted. Different inputs → different weights → different hash. (Any one of the four missing breaks it, but data is the one people most often forget because it "lives elsewhere," e.g. in Data versioning (DVC).)

• GPU non-determinism — parallel float reductions add in unpredictable order. Fix: deterministic kernels / cudnn.deterministic=True. See Reproducibility and random seeds.
• Different library versions — a new BLAS/framework build changes numerics. Fix: pin the environment (requirements.txt / conda hash) as part of lineage. (This is the "5th ingredient," the environment, that the parent note flags in the .pkl mistake.)
• Un-seeded auxiliary randomness — data shuffling, dropout, or augmentation using a second RNG that wasn't seeded. Fix: seed all RNGs (framework, numpy, python random).

Lesson: "pinned 4-tuple" reproduces byte-identically only if the pipeline is genuinely deterministic — the tuple is necessary, not sufficient.

Two structural mistakes:

1. Mutable artifact path — overwriting model_prod.pkl destroys every previous version. No history ⇒ no rollback, no audit, and you can't even identify which model is live.
2. No version identity / lineage — nothing records what produced each nightly model, so even the current one can't be rebuilt.

Fix: make versions append-only and immutable in a registry; each nightly retrain becomes v15, v16, ... with lineage. Serving points at a mutable alias champion (or reads stage=Production), never a raw overwritten file. Then rollback = flip champion to yesterday's version. Connects to Model serving and deployment patterns and Model monitoring and drift detection (which would have alerted on day 14).

Ordered gate:

1. Lineage complete? — v_new must carry (code_hash, data_hash, params, seed) + environment hash. Pass = all present. (No lineage ⇒ can't audit/rebuild, reject.)
2. Same-ruler eval? — v_new must be scored on eval hash d17 (the same set as v_cur). Pass = eval hash matches.
3. Metric threshold — Pass = auc_new >= auc_cur + δ for a chosen margin δ (e.g. 0.005) so noise doesn't trigger churn.
4. No regression on slices — Pass = no protected subgroup drops below its floor (fairness / safety).
5. Canary — route small traffic %, compare live metrics; Pass = live metrics hold within tolerance.
6. Promote atomically — flip alias champion → v_new, set stages; artifacts don't move.

Every gate is either a lineage check (auditability) or a same-ruler comparison (validity). Note steps 2 and 3 together defeat the L2 trap.

(a) Hot $=30\times 800\text{ MB}=24000\text{ MB}=24\text{ GB}$. (b) Total versions $=365$; total size $=365\times 800=292000\text{ MB}=292\text{ GB}$. (c) Rebuttal: deleting old versions destroys your rollback target and audit evidence — exactly the failure in L3.3, and the regulator request in L3.1 becomes impossible. Move them to cold storage instead of deleting.

(a) For SHA-256 the output is 256 bits, so there are $2^{256}$ possible hashes — astronomically more than any number of models we'll ever train; a random collision is effectively impossible in practice. (b) We rely on collision resistance: it is computationally infeasible to find two different byte-strings with the same hash. Determinism ("same bytes → same hash") gives identity; collision resistance gives distinctness. (c) Not a bug — it's the point. If two retrains are byte-identical, they should share a hash: they are the same model. Content-addressing deduplicates automatically. A shared hash means "provably the same artifact," which is a feature (that's also how you prove a reproduction in L3.1).

(a) Directed: each edge points backward in time (a version points at the data/code that produced it). Acyclic: you can never reach yourself by following backward pointers, because an artifact cannot be an input to its own creation — that would require it to exist before it existed. (b) A cycle would say "v5 was trained on data that was produced by v5," a time-travel contradiction. Honest provenance is strictly past-pointing, so cycles can't occur. (If your tooling shows a cycle, it's a logging bug, e.g. a mislabelled edge.) (c) The walk terminates at root nodes: pinned, externally-given inputs with no further provenance — a raw dataset snapshot, a specific code commit, a fixed seed. These are the leaves of reproducibility; once every path ends in a pinned root, the model is fully rebuildable.

Immutable (append-only): each night's artifact bytes, its content hash, its full lineage 4-tuple + environment hash, and its metrics-with-eval-hash. Never overwritten; kept for 7 years (cold storage after 30 days, per L4.2). Mutable: stage labels (Staging/Production/Archived) and aliases (champion) — the only movable things, enabling instant rollback. Determinism enforcement: pin the environment; enable deterministic GPU kernels (cudnn.deterministic, fixed reduction order); seed all RNGs — verified by hash-matching a re-run (the L3.2 lesson). Link out: data snapshots to Data versioning (DVC) (registry stores only the hash pointer, not gigabytes of raw data); experiment metadata to Experiment tracking and metadata logging; the nightly promotion to CI-CD for machine learning; and drift alerts to Model monitoring and drift detection.

This is the whole topic in one design: immutable identity + mutable pointers + deterministic pipeline + external stores linked by hash.

• Which items in a registry are immutable vs mutable, and why does the split exist?
• Why is a same-eval-hash the precondition for any metric comparison?
• Why is rollback time independent of model size?
• What two named properties (collision resistance, acyclicity) hold the whole system together?
• Given only (code, seed), why might two runs still differ — and what completes the fix?