5.3.3 · D5MLOps & Deployment

Question bank — Model versioning and registries

1,882 words9 min readBack to topic

A quick vocabulary refresher so every term below is grounded first:

Recall The words we use here (reveal if any feel fuzzy)
  • Artifact = the serialized bundle (weights + format + manifest) that IS the model on disk.
  • Registry = a name → ordered list of immutable versions, each with a mutable stage label.
  • 4-tuple = (code hash, data hash, hyperparameters, seed) — pin all four + a deterministic pipeline → byte-identical weights.
  • Content hash = a cryptographic fingerprint of the artifact bytes ; same bytes → same ID, any change → new ID.
  • Stage / alias = a mutable pointer (Staging, Production, champion) that says which version plays a role right now.
  • Lineage = backward pointers forming a DAG: version → data version → code commit → run.

True or false — justify

TRUE or FALSE: Two teammates running the exact same training script will always get the same weights.
FALSE — training is a stochastic process; without a pinned seed (and deterministic GPU ops) the same code produces different weights. This is precisely why the seed is part of the 4-tuple.
TRUE or FALSE: If you pin only the code commit and the data hash, the model is reproducible.
FALSE — you still need the hyperparameters and the seed. All four elements plus a deterministic pipeline are required for byte-identical weights; drop one and re-runs can diverge.
TRUE or FALSE: A model registry replaces Git.
FALSE — Git still versions the text/code; the registry versions the binary artifact + lineage. They are linked (the version stores its code commit hash), not substitutes.
TRUE or FALSE: Promoting v5 to Production physically copies the artifact to the serving path.
FALSE — promotion flips a stage/alias pointer. The artifact never moves; that's why the change is atomic and instant, and why rollback is equally fast.
TRUE or FALSE: A higher AUC always means the newer version is better.
FALSE — the metrics are only comparable if measured on the same frozen eval set (same eval-data hash). Different eval data makes 0.93 vs 0.91 an apples-vs-oranges comparison.
TRUE or FALSE: Once created, a version's weights and metadata can be edited to fix a typo.
FALSE — versions are immutable and append-only. To "fix" something you register a new version; only the mutable stage label may change on the existing one.
TRUE or FALSE: The content hash and the human label v3 carry the same information.
FALSE — the hash is an exact machine identity (immutable, byte-derived); v3 is a convenient human counter. The registry deliberately keeps both, mirroring Git's SHA vs branch-name split.
TRUE or FALSE: A .pkl file that loads and predicts correctly is a complete, reproducible record of the model.
FALSE — it can be reloaded but not rebuilt: it omits the data snapshot, code commit, seed, and library versions. Unpickling under a different library version can silently change predictions.
TRUE or FALSE: Deleting an archived version to save storage is fine once a newer one is in Production.
FALSE — deleting it destroys your rollback target and audit evidence. Archived ≠ deletable; immutability is what makes emergency rollback a pointer flip instead of a rebuild.

Spot the error

Spot the flaw: "We version by overwriting model_prod.pkl on every deploy — serving always reads one fixed path, nice and simple."
Overwriting destroys the previous version, so there is no rollback and no audit trail. Fix: keep versions immutable/append-only and let only the alias/path pointer change.
Spot the flaw: "SHA-256 of the artifact is our version number, so we increment it v1, v2 by adding 1 to the hash."
A hash is not an ordered counter — you can't "add one" and it has no notion of newer/older. The monotonic v1, v2 label is layered on top of the hash, not derived by arithmetic from it.
Spot the flaw: "We changed the input feature schema but only bumped MINOR because the code diff was tiny."
A new input schema is a breaking change → it must bump MAJOR. MINOR is for retraining on new data with the same interface; consumers pinning <2.0.0 would break silently.
Spot the flaw: "We stored the metric auc: 0.93 but not which dataset it was measured on — it's just a number."
A metric without its eval-data hash is uninterpretable and can't be compared across versions. Always store the frozen eval-set identity next to the metric.
Spot the flaw: "To reproduce the old model we just git checkout the commit and retrain — good enough."
Checking out the code is one of four pins. Without the data snapshot, hyperparameters, and seed, retraining can yield different weights whose hash won't match the audited one.
Spot the flaw: "We bumped MAJOR because we fixed a typo in the serving wrapper; weights unchanged."
A wrapper/serving bug fix with unchanged weights and interface is a PATCH, not MAJOR. Over-bumping MAJOR wrongly signals a breaking contract change to consumers.

Why questions

WHY is a cryptographic hash a better identity than a filename like final_model_v2_REAL.pkl?
The hash is deterministic and collision-resistant — it changes iff the bytes change, so it can never lie about identity. Filenames are human-mutable and routinely reused/overwritten, giving false identity.
WHY does the registry keep the immutable version separate from the mutable stage?
So the exact model (weights+lineage) is preserved forever for audit/reproduction, while the role it plays (Staging/Production/champion) can be reassigned cheaply and atomically — separating "what it is" from "what job it has now."
WHY is registry rollback measured in seconds, not hours?
Because the old version's artifact is still stored (immutable), so rollback just flips the champion/stage pointer back — no rebuild, retrain, or file copy is needed.
WHY do we canary/shadow-test a Staging model before flipping it to Production?
Offline eval metrics can hide live problems (traffic skew, latency, upstream feature drift). Sending a small traffic slice first surfaces real-world failure before it hits all users.
WHY do consumers pin a range like >=2.0.0, <3.0.0 instead of a single version?
It lets them automatically receive safe retrains/patches (MINOR/PATCH) while the <3.0.0 bound guards against a breaking interface change (MAJOR). It trusts the SemVer contract.
WHY store lineage as a directed acyclic graph rather than a flat list?
Reproduction requires walking backward through provenance edges (model → data → code → run) until every input is a pinned immutable node; the DAG structure captures those multi-parent dependencies with no cycles.
WHY is "same code" not enough to guarantee "same model"?
Training reads data and randomness too; different data snapshots or an unpinned seed (or non-deterministic GPU ops) make identical code produce different weights.

Edge cases

EDGE CASE: The pipeline is not deterministic (e.g. non-deterministic GPU kernels). Does pinning the 4-tuple still guarantee byte-identical weights?
No — the reproducibility guarantee assumes a deterministic pipeline. Non-determinism breaks the byte-identical claim; you'd fall back to reproducing statistically similar (not identical) weights unless you also force deterministic ops.
EDGE CASE: You retrain and get literally the same weights as the previous version. What ID does it get?
Identical artifact bytes → the same content hash. Content-addressing means it's the same immutable content; the registry may still expose a new human label, but the hash de-duplicates it.
EDGE CASE: Two genuinely different models happen to collide to the same hash. Is versioning broken?
In practice no — SHA-256 is collision-resistant, so distinct models producing the same hash is astronomically unlikely and treated as impossible. The design leans on this exact property.
EDGE CASE: The very first model — there is no previous version to compare against or roll back to.
It registers as v1/Production with its full lineage; there's simply no rollback target yet. The immutability rule still applies so that the next deploy has something to fall back to.
EDGE CASE: You only changed the serving wrapper, weights untouched. Does the artifact hash change and is it a new version?
If the wrapper is part of the hashed artifact bundle, the bytes change → new hash → new version, but SemVer-wise it's a PATCH (interface and weights unchanged). If the wrapper is versioned separately, only the wrapper's version bumps.
EDGE CASE: A regulator asks you to reproduce a model but the referenced data snapshot was deleted.
Reproduction fails — the data hash points to a node that no longer exists, breaking the lineage DAG. This is exactly why data snapshots must be immutably retained (see Data versioning (DVC)), not just referenced.
EDGE CASE: Live metrics silently degrade while the version and its stored eval AUC stay unchanged. Is the registry wrong?
No — the registry records the model as it was; the degradation is drift in incoming data, not a change to the model. Detecting it is the job of Model monitoring and drift detection, which then triggers a new version.

Active recall

Recall Rapid self-check (reveal after answering all)
  • Name the one thing that moves during a rollback. (The alias/stage pointer.)
  • Give one reason "same code" ≠ "same model". (Unpinned seed / different data / non-deterministic ops.)
  • When is a metric comparison between versions valid? (Only on the same frozen eval-data hash.)
  • Which SemVer field bumps for a new input schema? (MAJOR.)
  • What guarantee do you LOSE if the pipeline is non-deterministic? (Byte-identical reproducibility.)

Flashcards

Why is overwriting model_prod.pkl on each deploy a mistake?
It destroys the previous version, so you lose rollback and audit; versions must be immutable/append-only with only the pointer mutable.
Why can't you "increment" a content hash to get the next version?
A hash is an unordered fingerprint, not a counter; the monotonic v1/v2 label is layered on top of the hash, not computed from it.
What guarantee breaks if the training pipeline is non-deterministic?
Byte-identical reproducibility — pinning the 4-tuple only reproduces exact weights when the pipeline is deterministic.
Two different models with the same SHA-256 hash — why is this not a real concern?
SHA-256 is collision-resistant, so distinct artifacts colliding is astronomically improbable and treated as impossible by design.
A new input schema requires which SemVer bump and why?
MAJOR — it's a breaking interface change, so consumers pinning below the next major version won't silently break.