Visual walkthrough — Data versioning (DVC)

We invent DVC step by step. Every step answers three questions: WHAT we just did, WHY we did it, and WHAT IT LOOKS LIKE (the figure).

Step 1 — The thing we are versioning: a file is just bytes

WHAT. A file — say data.csv — is nothing mystical. It is a long ribbon of numbers, each between 0 and 255, called bytes. A 2 GB file is roughly 2 billion of these numbers in a row.

WHY start here. Everything DVC does keys off content — the actual byte ribbon — not the file's name. If we don't picture the file as bytes, the rest won't make sense.

PICTURE. Below, the whole file is drawn as one horizontal strip cut into byte-cells. The strip is the object we will fingerprint.

The name data.csv is not in this ribbon. The name lives in the folder listing; the content lives in the ribbon. Keep those two apart — the whole design depends on it.

Step 2 — Fingerprint the content with a hash

WHAT. We feed the entire byte ribbon into a hash function and get back one short fixed-length string, e.g. a1b2c3d4... (32 hex characters for MD5). Call it $h$.

$h=\text{MD5}(b_1,b_2,\dots ,b_N)$

• $\text{MD5}(\cdots )$ ::: the "blender" — reads all $N$ bytes and stirs them into a fixed 128-bit output.
• $h$ ::: the fingerprint, always the same short length no matter how big $N$ is.

WHY a hash and not, say, the file size or a timestamp? We need a label with two magic properties:

1. Same content → same label. Two identical ribbons always blend to the same $h$. (Size alone can't do this — two different 2 GB files share a size.)
2. One byte differs → the label changes completely. Flip a single number in the ribbon and $h$ becomes unrecognisably different. (A timestamp can't do this — it ignores content entirely.)

A hash is the only everyday tool that gives us both at once. That is why Git, Docker, and DVC all reach for it — see Content-addressable storage.

PICTURE. Two ribbons go into the blender. Identical ribbons drop out with the same red fingerprint; a one-byte-changed ribbon drops out with a totally different one.

Step 3 — Store the file by its fingerprint (content addressing)

WHAT. Instead of keeping the file under its human name, we move it into a cache folder and rename it after its hash $h$:

.dvc/cache/a1/b2c3d4...    ← the real 2 GB bytes

(The first two hex characters become a sub-folder just to avoid one giant directory.)

WHY address by content? Because now the storage location is computed from the data itself. This gives us deduplication for free: if two experiments use the exact same dataset, both compute the same $h$, both point at the same one cached copy. Identical data is stored once, never twice.

PICTURE. The named file data.csv in your working folder is lifted out and filed in the cache under its fingerprint. The working folder now has a hole where the file was.

Step 4 — Leave a tiny pointer where the file used to be

WHAT. The working folder can't just have a hole — you still want to find your data. So DVC drops a tiny text file next to it, data.csv.dvc, holding the fingerprint and a few facts:

outs:
- md5: a1b2c3d4...
  size: 2147483648
  path: data.csv

• md5 ::: the fingerprint $h$ — the address of the real bytes in the cache.
• size ::: $N$, the byte count (2147483648 bytes = 2 GB) — a sanity check.
• path ::: the human name to restore the file back to.

This pointer is about 100 bytes. That is the whole trick: we swapped a 2 GB object for a 100-byte note that knows where the 2 GB object lives.

WHY put only the pointer under Git version control? Git stores a full copy of every version of every file it tracks. Track the 2 GB file directly and ten edits = ~20 GB inside .git/. Track the 100-byte pointer instead and ten edits = ~1 KB. Git stays feather-light while still recording which data each commit used.

PICTURE. Left: the heavy file sits in the cache. Right: Git holds only the small pointer card. A red arrow shows the pointer resolving to the cached bytes via the hash.

Step 5 — Bind code + data in one Git commit

WHAT. Git already versions your train.py and hyperparameters. Now it also versions data.csv.dvc. So a single Git commit freezes both: the exact code AND the exact data-fingerprint.

$\underset{\text{one snapshot}}{\underbrace{\text{Git commit}}}\supset \{\underset{\text{code}}{\underbrace{\text{train.py}}},\underset{\text{data fingerprint}}{\underbrace{\text{data.csv.dvc}(h)}}\}$

• The commit contains the pointer, which contains $h$, which resolves to the exact bytes.

WHY this is the whole point. Reproducibility in ML means: restore everything that produced a result, together. Because the data's identity is now just a hash living inside a normal Git commit, "check out the commit" restores code and the link to the exact data in one move.

PICTURE. The versioning invariant as a chain: commit → pointer($h$) → data at hash $h$. The red link is the resolution step DVC performs.

Step 6 — Restore: read the pointer, replay the file

WHAT. To go back to an old dataset you do two moves:

1. git checkout <commit> — Git puts the old pointer (old $h$) back in your folder.
2. dvc checkout — DVC reads that $h$, finds the matching bytes in the cache, and copies them back to path, re-creating data.csv.

WHY two commands? Separation of concerns. Git only ever moves the light note; DVC only ever moves the heavy cargo. The mantra: Git tracks the note, DVC tracks the boat.

PICTURE. Old commit selected → old pointer lands in the folder → DVC follows the red hash-arrow into the cache and rebuilds the file.

Step 7 — The remote: when the cache is empty (the degenerate case)

WHAT. So far the cache lived on your machine. But a teammate cloning fresh, or you on a new laptop, has an empty cache. dvc checkout then finds the pointer but no bytes — it can restore nothing.

WHY it fails, and the fix. A fresh git pull brings the 100-byte pointer, not the 2 GB. The bytes were only ever in your cache until you pushed them. So:

• After adding data: dvc push → uploads cached bytes to a remote (Object storage (S3/GCS), SSH, etc.).
• On a fresh clone: git pull (pointers) then dvc pull (bytes) before dvc checkout.

PICTURE. Three boxes — your cache, the S3 remote, the teammate's empty cache. Red arrows show dvc push (up to remote) and dvc pull (down to teammate). Without the pull, the teammate's box stays empty and checkout has nothing to copy.

Step 8 — Edge case: change one byte, how much new storage?

WHAT. Flip a single byte in the 2 GB file and run dvc add. Recall Step 2's avalanche: one byte → an entirely new $h$. A new hash means a new cache slot, filled with a full fresh 2 GB copy. The old copy stays too (old commits still point at it).

WHY know this. DVC's default is whole-file content hashing. It does not store a small "diff" of your one-byte edit. So version snapshots, not tiny scratch edits — otherwise the cache balloons.

$\text{new bytes stored}=N\approx 2\text{GB},\text{not }1\text{byte}$

PICTURE. Two full-length ribbons in the cache under two different red fingerprints — even though only one cell differs.

The one-picture summary

Everything above, compressed: the byte ribbon is fingerprinted by a hash, stored by that hash in the cache (and mirrored to a remote), while a 100-byte pointer rides inside a Git commit alongside the code — so any commit re-creates the exact code + data pair.

Connections

• Git version control — versions the tiny pointer + code; DVC borrows its content-hashing idea.
• Content-addressable storage — Step 3's principle: address = content hash.
• Reproducibility in ML — Step 5's invariant is one leg of it (code + data + environment).
• Object storage (S3/GCS) — the remote in Step 7 that holds the heavy bytes.
• MLOps pipelines — these pointers feed dvc.yaml DAGs and CI/CD retraining.
• Experiment tracking (MLflow) — logs metrics while DVC pins the data/artifacts.