Model versioning and registries: Worked examples

The scenario matrix

Before working examples, let us enumerate the case classes this topic can generate. A model-registry "problem" always lives in one of these cells. Think of it like listing every quadrant before solving triangles — we want zero surprises.

Cell	Case class	The stress it applies	Example
A	Normal promotion (challenger beats champion, same eval set)	The happy path	Ex 1
B	"Better" number but different eval set	The metric-comparison trap	Ex 2
C	Emergency rollback	Immutability + atomic pointer flip	Ex 3
D	Rebuild from lineage (audit)	Deterministic pipeline replay	Ex 4
E	Degenerate input: identical bytes retrained	Content hash must collapse to same ID	Ex 5
F	Zero / empty: registering with no metrics or empty artifact	Boundary handling	Ex 6
G	SemVer decision (which of MAJOR/MINOR/PATCH?)	Interface-vs-data-vs-code reasoning	Ex 7
H	Limiting behaviour: many versions, cheap comparison	Why 256-bit IDs scale	Ex 8
I	Real-world word problem (loan regulator)	End-to-end lineage walk	Ex 9
J	Exam-style twist (silent break, no error thrown)	The `.pkl`-only failure mode	Ex 10

The figures below carry the geometry of the pointer-flip and the hash mapping — refer to them as we go.

Figure — Model versioning and registries

Look at the figure above: the left column is the immutable version list (never edited, only appended). The coloured arrows on the right are the mutable pointers (champion, Production). Every example on this page is really "which arrow moves, and which box must stay frozen?"

Worked examples

Worked example Example 1 — Cell A: normal promotion, same eval set

fraud-detector v4 is in Production with AUC = 0.91. You train v5 and measure AUC = 0.93 on the identical frozen eval set (data hash d17). Should v5 go to Production, and what physically moves?

Forecast: guess now — does the v4 artifact get deleted? Does the alias move before or after canary?

Register v5 with metrics = {auc: 0.93} and lineage = {data: d17, code: a3f9}. Why this step? The metric 0.93 is meaningless unless it is bound to the exact eval set. Binding it to hash d17 (the same hash v4 was scored on) makes the comparison legal.
Compute the improvement: $Δ = 0.93 - 0.91 = 0.02$ , a relative gain of $\frac{0.02}{0.91} \approx 2.20%$ . Why this step? You want a number to justify the promotion in the change log, not a vibe.
Set v5.stage = "Staging", run a canary on $5%$ of traffic. Why this step? Offline AUC can lie about live behaviour; test small before trusting.
If live metrics hold, flip: champion → v5, v5.stage = "Production", v4.stage = "Archived". Why this step? Flipping an alias is atomic and instant — the big artifact never moves (see the s01 arrows).

Verify: Both AUCs share data hash d17 ⇒ comparison is apples-to-apples ✓. $0.93 > 0.91$ ⇒ promote ✓. v4 is Archived, not deleted ⇒ rollback still possible ✓.

Worked example Example 2 — Cell B: the different-eval-set trap

v4: AUC = 0.91 on eval set d17. v5: AUC = 0.95 on eval set d20 (a newer, easier eval set). Is v5 better?

Forecast: $0.95 > 0.91$ , so surely yes? Guess before reading.

Read the eval-data hash attached to each metric: v4 → d17, v5 → d20. Why this step? A metric without its eval hash is a naked number. The hash is the "same triangle" you must draw before comparing angles.
Since $d 17 \neq = d 20$ , the numbers are not comparable. Refuse the direct comparison. Why this step? $0.95$ on an easier set can be worse than $0.91$ on a hard set — apples vs oranges.
Fix: re-score v5 on d17. Suppose it gives AUC = 0.90. Why this step? Now both live on d17.
Compare on the common set: $0.90 < 0.91$ ⇒ v5 is actually worse. Keep v4.

Verify: After re-scoring, both metrics carry hash d17 ✓. $0.90 < 0.91$ ⇒ do not promote ✓. The naive $0.95 > 0.91$ conclusion was a trap ✓.

Worked example Example 3 — Cell C: emergency rollback

v5 is live in Production. A feature pipeline breaks (drift — see Model monitoring and drift detection) and v5 starts returning garbage. Roll back.

Forecast: does rollback require retraining or redeploying gigabytes? Guess.

set_alias("champion", "v4"). Why this step? v4's artifact was never deleted (immutability). Rollback is a pointer flip, not a rebuild — the red→green arrow swap in figure s01.
Set v5.stage = "Archived" with a note "data-drift feature X null, 2024-xx". Why this step? Keep the evidence for the post-mortem; never delete a failing version.
Measure rollback latency. Redeploying a $200 MB$ artifact over the network at, say, $50 MB/s$ would take $\frac{200}{50} = 4$ seconds just to copy — but an alias flip touches only a few bytes of metadata, so it is effectively instantaneous. Why this step? This is the whole reason the immutable/mutable split exists.

Verify: v4 still stored ⇒ pointer flip valid ✓. Artifact copy avoided: $200/50 = 4$ s saved per flip, and the flip itself is $O (bytes)$ ✓.

Worked example Example 4 — Cell D: rebuild from lineage

You must reproduce byte-identical weights for v3, whose stored artifact hash is 9c1. Its lineage: code = commit b12, data = snapshot s09, seed = 42.

Forecast: will re-running give exactly the same weights, or just "close"? Guess.

Pin all four of the reproducibility tuple $V = (code, data, params, seed)$ : git checkout b12, load s09, set seed = 42, load the stored params. Why this step? The 5.3.3 contract says: all four pinned and a deterministic pipeline ⇒ byte-identical output. See Reproducibility and random seeds.
Re-run training, obtain artifact bytes $b^{'}$ . Why this step? Produce a fresh artifact to test against the original.
Compute $H (b^{'})$ and compare to the stored 9c1. Why this step? The content hash is the proof of reproduction (see figure s02: identical bytes → identical hash cell).

Verify: If the pipeline was deterministic, $H (b^{'}) = 9c1$ exactly — a hash match is a bit-for-bit guarantee, not "similar" ✓. If it differs, some part of the tuple was not truly pinned (GPU non-determinism, unlogged lib version) ⇒ audit the pipeline ✓.

Worked example Example 5 — Cell E (degenerate): identical bytes, "new" training run

Two teammates train the "same" model with the identical 4-tuple. Both produce artifacts with byte content $b$ . What version IDs do they get?

Forecast: two runs, so two version numbers? Or one ID? Guess.

Content ID is $ID = H (b)$ . Same bytes $b$ ⇒ same $H (b)$ . Why this step? A hash is deterministic by definition: identical input → identical output (figure s02).
The registry sees the incoming hash already exists ⇒ it deduplicates: no new immutable artifact is stored. Why this step? Storing the same bytes twice wastes space and pollutes lineage.
The human counter (v6, v7) may still differ — that is a mutable label, decoupled from identity. Why this step? Machine identity (hash) ≠ human label (v7), exactly the Git SHA-vs-branch split.

Verify: $H (b) = H (b)$ trivially ✓. So the machine identity collapses to one node even though two humans "trained twice" ✓. This is the degenerate case where "two versions" is really one artifact.

Worked example Example 6 — Cell F (zero / empty boundary): register with no metrics

A pipeline registers a model but the eval step crashed, so metrics = {} (empty) and, in a broken run, the artifact bytes are empty ( $b = \emptyset$ ). What should the registry do?

Forecast: does an empty artifact get a valid, unique ID? Is it promotable?

Empty metrics {} ⇒ the version is registerable but not promotable: with no metric on a known eval hash, no comparison is possible. Why this step? Promotion requires a metric bound to an eval hash (Cell A/B rule). Empty ⇒ fails the precondition.
Empty bytes: $H (\emptyset)$ is still a well-defined, fixed hash (SHA-256 of the empty string is a specific 256-bit value). So it gets an ID — but a guard should reject a zero-length artifact as invalid. Why this step? The math never crashes (hash of empty is defined), but semantically an empty model is useless — the policy layer, not the hash, must reject it.

Verify: $H (\emptyset)$ is defined and deterministic ✓ (the hash function has no "divide by zero"). Promotion blocked because $∣ metrics ∣ = 0$ ✓. This is the graceful-degenerate case: identity works, promotion doesn't.

Worked example Example 7 — Cell G: which SemVer bump?

Decide the version bump for three separate changes to recommender 2.4.1. Recall MAJOR.MINOR.PATCH.

Forecast: guess each of the three before reading.

Change 1: you retrain on fresh data; input schema and output meaning unchanged; metrics shift. → MINOR bump: $2.4.1 \to 2.5.0$ . Why? Same interface, new data ⇒ MINOR (and PATCH resets to 0).
Change 2: you fix a bug in the serving wrapper; weights unchanged. → PATCH bump: $2.4.1 \to 2.4.2$ . Why? Wrapper-only, weights identical ⇒ PATCH.
Change 3: the model now outputs a probability instead of a raw score — output meaning changed. → MAJOR bump: $2.4.1 \to 3.0.0$ . Why? The contract broke; downstream code interpreting the output must change. MINOR and PATCH reset to 0.

Verify: MINOR: $2.5.0$ (patch→0) ✓. PATCH: $2.4.2$ ✓. MAJOR: $3.0.0$ (minor & patch →0) ✓. A consumer pinning >=2.0.0,<3.0.0 accepts changes 1 and 2 but is protected from change 3 ✓ (see CI-CD for machine learning).

Worked example Example 8 — Cell H (limiting behaviour): comparison cost at scale

Your registry holds $N = 10, 000$ versions, each artifact $200 MB$ . You need to check whether a new artifact already exists. Compare "diff the bytes" vs "compare hashes".

Forecast: how much data must you touch in each strategy?

Byte-diff strategy: to check against all versions you would read up to $N \times 200 MB = 10, 000 \times 200 = 2, 000, 000 MB = 2 TB$ . Why this step? Shows the naive cost grows linearly in artifact size, which is huge.
Hash strategy: each ID is $256$ bits $= 32$ bytes. Comparing against all $N$ hashes touches $10, 000 \times 32 = 320, 000$ bytes $= 320 KB$ , and with a hash-index lookup it is $O (1)$ . Why this step? The hash compresses gigabytes to 32 bytes while preserving identity — this is exactly why content-addressing scales.
Ratio of data touched: $\frac{2 TB}{320 KB} = \frac{2 \times 1 0 ^{12}}{3.2 \times 1 0 ^{5}} = 6, 250, 000$ . Why this step? Quantifies the ~6-million-fold saving in the limit of many large models.

Verify: $10000 \times 200 = 2, 000, 000$ MB $= 2$ TB ✓. $10000 \times 32 = 320000$ bytes ✓. $2 \times 1 0^{12} /3.2 \times 1 0^{5} = 6.25 \times 1 0^{6}$ ✓.

Worked example Example 9 — Cell I (real-world word problem): the regulator

On 2024-03-01 a loan was denied. A regulator demands: reproduce the exact model that made that decision, and prove it. Registry log says: Production model that day was v3, hash 9c1, lineage {code: b12, data: s09, seed: 42}.

Forecast: what single object proves you reproduced the right model?

Query the registry's stage history for 2024-03-01 ⇒ Production alias pointed at v3. Why this step? The mutable pointer's history tells you which frozen version was live on that date.
Walk the lineage DAG backward: v3 → data s09 → code b12 → seed 42. Each node is immutable. Why this step? Reproduction needs every input pinned; the DAG walk gathers them.
Rebuild (as in Ex 4), get bytes $b^{'}$ , compute $H (b^{'})$ , and check $H (b^{'}) = ? 9c1$ . Why this step? The hash match is the legal proof — bit-for-bit identity to the model that actually denied the loan.

Verify: Deterministic pipeline + all-pinned tuple ⇒ $H (b^{'}) = 9c1$ ✓ (same guarantee as Ex 4). The single proving object is the matching content hash ✓. Data lineage relies on Data versioning (DVC) and metadata from Experiment tracking and metadata logging.

Worked example Example 10 — Cell J (exam twist): the silent break

Exam question: "You saved only model_prod.pkl. Six months later you pickle.load it under a newer sklearn. It loads with no error and predicts. Is everything fine? What is the hidden failure?"

Forecast: no exception was raised — so it's safe, right? Guess.

Loading succeeded, but the pickle recorded none of the 4-tuple: no data hash, no code commit, no seed, no library versions. Why this step? "It loaded" tests reload, not reproduce — two different guarantees.
A newer sklearn can change internal algorithms/defaults ⇒ predictions can silently differ while the API stays identical. No exception fires. Why this step? This is the dangerous case: failure with no error signal.
Detect it: recompute predictions on a frozen eval set and compare to the recorded metric. If today's AUC $\neq =$ the logged AUC on the same eval hash, the environment drifted. Why this step? The eval-hash-bound metric is your tripwire for silent breaks.
Fix: log the full 4-tuple + environment (requirements.txt hash) in the registry, per 5.3.3.

Verify: "Loads without error" ⇏ "reproduces the same predictions" ✓ — the two are logically independent. The tripwire is a metric mismatch on a shared eval hash ✓. Conclusion: not fine ✓.

Active recall

Recall Which cell, which pointer? (reveal after answering)

In a promotion, what physically moves and what stays frozen? ::: The alias/stage pointer moves; the immutable artifact never moves.
When are two metric numbers legally comparable? ::: Only when bound to the same eval-data hash.
Two identical-byte training runs — how many artifact IDs? ::: One; the content hash deduplicates them.
What object proves you reproduced an audited model? ::: A matching content hash of the rebuilt bytes.
"Pickle loaded without error" — does it prove reproducibility? ::: No; it proves reload, not reproduce.