Data versioning (DVC): Exercises — AI-ML deep dive

Before we start, one picture fixes the whole mental model — keep it in view.

Which of the following does a Git commit by itself physically contain after you run dvc add data.csv; git add data.csv.dvc; git commit? (a) the 2 GB data.csv, (b) the ~100-byte data.csv.dvc pointer, (c) both.

Recall Solution 1.1

Answer: (b). The commit stores only the pointer. The heavy bytes are still sitting in your local cache; they do not travel with the commit and are not in .git/. This is the entire reason Git stays light. To ship the bytes you must additionally dvc push.

Exercise 1.2 (L1)

Match each command to its job: dvc add, dvc push, git checkout, dvc checkout. Jobs: (i) upload heavy bytes to remote, (ii) restore the pointer to an old version, (iii) create the pointer + move file to cache, (iv) rebuild the real file from the pointer.

Recall Solution 1.2

dvc add → (iii) create pointer, move file into cache.
dvc push → (i) upload cache bytes to remote.
git checkout → (ii) restore the pointer text to an old version.
dvc checkout → (iv) read the pointer, rebuild the real file from cache/remote. Mantra: "Git tracks the note, DVC tracks the boat."

Exercise 1.3 (L1)

True or false: two files with identical content but different filenames produce the same hash in the cache.

Recall Solution 1.3

True. DVC hashes content, not name. Identical bytes → identical MD5 → stored once (deduplication). The filename lives only inside the pointer's path: field, not in the hash.

L2 — Application

Exercise 2.1 (L2)

Write the exact command sequence to: initialise DVC, register an S3 remote s3://mybucket/store as default, version data/train.csv, and make both Git and the remote hold the new version.

Recall Solution 2.1

dvc init
dvc remote add -d store s3://mybucket/store
dvc add data/train.csv
git add data/train.csv.dvc data/.gitignore
git commit -m "add v1 training data"
dvc push

Why git add the .gitignore? dvc add writes data/.gitignore so Git ignores the real file. If you forget to commit that ignore file, a teammate's Git could try to track the heavy file. Why dvc push last? Without it the bytes never leave your laptop's cache.

Exercise 2.2 (L2)

A teammate just cloned the repo. They run dvc checkout and nothing appears. In two commands, fix it and explain each.

Recall Solution 2.2

dvc pull       # fetch the bytes from the remote into the local cache
dvc checkout   # rebuild the working file from the now-populated cache

A fresh clone has an empty cache — only pointers came with Git. dvc checkout can only copy from cache; with nothing there it has nothing to copy. dvc pull fills the cache from the remote first. (In fact dvc pull alone usually also checks out.)

Exercise 2.3 (L2)

You edited data/train.csv in place (added rows). git status shows nothing new for the data. Why, and what single DVC command records the new version?

Recall Solution 2.3

Git shows nothing because the real file is in .gitignore; Git only watches the pointer, which you have not changed. DVC does not auto-detect the edit — it re-hashes only when asked. Run:

dvc add data/train.csv     # or: dvc commit

This recomputes the hash, stores the new bytes in cache, and rewrites the pointer. Then git add data/train.csv.dvc && git commit && dvc push.

L3 — Analysis

Exercise 3.1 (L3)

You have a 2 GB file. You change 1 byte and run dvc add. How much new data lands in the cache, and how large is the total cache now (assuming default whole-file hashing)? Give exact byte figures using $2 GB = 2, 147, 483, 648$ bytes.

Recall Solution 3.1

Changing one byte flips the whole-file MD5 → a brand-new hash → DVC stores a full new copy.

New data added: $2, 147, 483, 648$ bytes ( $= 2$ GB).
Total cache (old + new versions kept): $2 \times 2, 147, 483, 648 = 4, 294, 967, 296$ bytes ( $= 4$ GB). Moral: version snapshots, not micro-edits. This is the deliberate cost of whole-file content addressing.

Exercise 3.2 (L3)

Two files, a.csv and b_copy.csv, are byte-for-byte identical (each 500 MB). Both are dvc add-ed. How many bytes does the cache use for the actual data (ignore the tiny pointers)? Take $500 MB = 524, 288, 000$ bytes.

Recall Solution 3.2

Identical content → identical hash → the bytes are stored once. Cache data usage $= 524, 288, 000$ bytes ( $= 500$ MB), not 1 GB. Two pointers (a.csv.dvc, b_copy.csv.dvc) both name the same hash. This is content-addressable deduplication in action — see Content-addressable storage.

Exercise 3.3 (L3)

A dataset went through 5 versions of a 2 GB file: v1 fresh, then each later version changed at least one byte from the previous. All 5 are committed and pushed. What is the worst-case remote storage, and what is the best case if versions v2 and v4 happened to be byte-identical to v1?

Recall Solution 3.3

Worst case (all 5 distinct): $5 \times 2, 147, 483, 648 = 10, 737, 418, 240$ bytes ( $= 10$ GB).
Best case (v2, v4 identical to v1): distinct contents are just ${v 1, v 3, v 5}$ → 3 copies: $3 \times 2, 147, 483, 648 = 6, 442, 450, 944$ bytes ( $= 6$ GB). Deduplication only rescues you when content actually repeats; a genuinely changed byte always costs a full copy under whole-file hashing.

Look at the storage-growth picture:

L4 — Synthesis

Exercise 4.1 (L4)

Write a dvc.yaml with two stages: preprocess turns data/raw.csv into data/clean.csv via python prep.py, and train turns data/clean.csv + train.py into model.pkl. Then explain what dvc repro does if only train.py changed.

Recall Solution 4.1

stages:
  preprocess:
    cmd: python prep.py
    deps: [prep.py, data/raw.csv]
    outs: [data/clean.csv]
  train:
    cmd: python train.py
    deps: [train.py, data/clean.csv]
    outs: [model.pkl]

dvc repro hashes each stage's deps. preprocess deps (prep.py, data/raw.csv) are unchanged → skipped, data/clean.csv reused. train sees train.py changed → re-runs, producing a new model.pkl. This is make-like caching over a DAG; see MLOps pipelines.

Exercise 4.2 (L4)

Design the minimal recipe so a colleague can reproduce exactly the model you trained last Tuesday: which artifacts must be bound in a single Git commit, and which command chain restores everything?

Recall Solution 4.2

Bind in one commit: the training code (train.py, Git), hyperparameters/dvc.yaml/dvc.lock (Git), and the data pointer data/clean.csv.dvc (Git). Because the pointer holds the data's hash, the commit binds code hash + data hash together. Restore chain:

git checkout <tuesday-commit>   # exact code + pointers
dvc pull                        # fetch matching data/model bytes
dvc checkout                    # rebuild working files at those hashes
dvc repro                       # (optional) re-run to regenerate the model

This is precisely the invariant from Reproducibility in ML: reproducibility $=$ code hash $+$ data hash bound in one commit.

L5 — Mastery

Exercise 5.1 (L5)

A teammate does git checkout old-commit (restoring an old pointer) but forgets dvc checkout. Their working data.csv on disk is still the new version, yet the pointer says the old hash. Describe the inconsistent state precisely and how DVC detects/repairs it.

Recall Solution 5.1

State: the pointer (Git-tracked) names hash $h_{old}$ , but the file on disk hashes to $h_{new} \neq = h_{old}$ . Git is "clean" (pointer matches commit), yet the data is wrong for that commit. Detection: dvc status re-hashes the working file, finds $h_{new} \neq = h_{old}$ , and reports the file as changed/modified relative to the pointer. Repair: dvc checkout overwrites the working file with the bytes at $h_{old}$ from cache (or dvc pull first if $h_{old}$ isn't cached). Now disk hash $= h_{old} =$ pointer → consistent.

Exercise 5.2 (L5)

You push v1 and v2 (both 2 GB, distinct) to the remote, then run dvc gc (garbage-collect) after checking out only v2, with the flag that keeps only what the current workspace needs. Assuming v1 is referenced by no checked-out commit or tag, how many bytes remain retrievable, and what have you lost?

Recall Solution 5.2

dvc gc deletes cache/remote objects not reachable from the retained references. With only v2's workspace retained and v1 unreferenced:

Remaining: $2, 147, 483, 648$ bytes ( $= 2$ GB, the v2 copy).
Lost: the v1 bytes — even though its pointer may still sit in old Git history, git checkout old-commit; dvc checkout will now fail to restore v1 because those bytes are gone. Lesson: dvc gc is a sharp knife; keep the reference flags (--all-commits/tags) that cover every version you might ever need to reproduce.

Exercise 5.3 (L5)

Degenerate case: you dvc add an empty file (0 bytes). Predict the pointer's size, whether a hash is still computed, and whether two different empty files dedupe.

Recall Solution 5.3

size: 0 in the pointer.
A hash is still computed — MD5 of the empty byte string is a well-defined, fixed fingerprint (d41d8cd98f00b204e9800998ecf8427e).
Since all empty files share identical (zero) content, they share that one hash → they dedupe to a single cache entry. Content addressing handles the zero-size edge cleanly: same content (even "no content") → same address.

Recall check

Recall One-byte edit cost, in one line

Whole-file hashing → one changed byte flips the whole hash → a full new copy in cache. Dedup helps only when content repeats.

Recall The consistency rule between Git and DVC

git checkout moves the pointer; dvc checkout moves the data. Both must run to land in a consistent past state. dvc status catches the mismatch.

Connections

Data versioning (DVC) — the parent concepts these drills exercise.
Git version control — supplies the pointer-tracking half of every workflow.
Content-addressable storage — the hash-as-address principle behind dedup exercises 3.1–3.2 and 5.3.
Reproducibility in ML — the goal of L4 synthesis: code hash + data hash in one commit.
MLOps pipelines — dvc.yaml DAGs and dvc repro from Exercise 4.1.
Experiment tracking (MLflow) — pairs with pinned data for full experiment recall.
Object storage (S3/GCS) — the remote holding pushed bytes in 2.1–2.2 and 5.2.

Exercises — Data versioning (DVC)

L1 — Recognition

Exercise 1.1 (L1)