Data versioning (DVC)

WHY does this problem even exist?

WHY not just use Git LFS? Git LFS also uses pointers, but it's tied to Git hosting quotas, is storage-provider-limited, and has no notion of ML pipelines or experiments. DVC decouples storage (any S3/GCS/SSH/local) from Git and adds pipeline reproducibility.

HOW DVC works — derive it from scratch

Let's build DVC ourselves conceptually. Suppose you have data.csv (2 GB).

Step 1 — Content addressing. Compute a hash of the file's content: $h = \text{MD5}(\text{bytes of file})$

Step 2 — Move the heavy file to a cache, renamed by its hash:

.dvc/cache/a1/b2c3...   (the real 2 GB)

Step 3 — Leave a tiny pointer under Git. DVC creates data.csv.dvc:

outs:
- md5: a1b2c3...
  size: 2147483648
  path: data.csv

This .dvc file is ~100 bytes → Git tracks it happily.

Step 4 — Restore. Given a .dvc pointer, look up the hash in cache (or dvc pull it from remote), and re-create data.csv in your working directory.

The essential commands (the 20% you use 80% of the time)

Worked example A full workflow

dvc init                      # create .dvc/ (like git init)
dvc remote add -d store s3://mybucket/dvcstore
dvc add data/train.csv        # -> creates train.csv.dvc, moves file to cache
git add data/train.csv.dvc data/.gitignore
git commit -m "add v1 of training data"
dvc push                      # upload actual data to S3 remote

Why git add the .gitignore? dvc add auto-writes data/.gitignore so Git ignores the real file and only tracks the pointer — no accidental 2 GB commit.

Worked example Restoring an old dataset version

git checkout v1-commit-hash   # restores the .dvc pointer (old hash)
dvc checkout                  # reads pointer, restores matching data from cache/remote

Why two steps? Git moves the pointer; DVC moves the data the pointer points to. Separation of concerns = light Git, heavy storage anywhere.

DVC pipelines (bonus: reproducing the whole flow)

Definition

dvc.yaml stage A stage declares cmd, its deps (inputs), and outs (outputs). DVC hashes deps; if none changed, it skips re-running (like make).

stages:
  train:
    cmd: python train.py
    deps: [train.py, data/train.csv]
    outs: [model.pkl]

Run with dvc repro. Why? Turns "which data + code produced this model?" into a tracked DAG, not tribal memory.

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

What problem does DVC solve that plain Git cannot?

Versioning large binary data/models — Git stores full copies of every version and can't diff binaries efficiently; DVC keeps tiny hash pointers in Git and heavy data in remote storage.

What does dvc add data.csv actually create and move?

Creates a small data.csv.dvc pointer (containing the MD5 hash + size + path), moves the real file into .dvc/cache/, and updates .gitignore so Git ignores the real file.

Why must you run dvc push after committing?

The Git commit only stores the ~100-byte pointer; the actual bytes stay in your local cache until dvc push uploads them to the remote.

What two commands restore an old dataset version?

git checkout <commit> (restores the pointer) then dvc checkout (restores the matching data from cache/remote).

Why does DVC hash file content instead of using the filename?

Identical content → identical hash → automatic deduplication; any change flips the hash so new versions are detected reliably.

What is a DVC pipeline stage made of?

A cmd, its deps (inputs), and outs (outputs); DVC skips re-running if deps are unchanged (make-like), via dvc repro.

Reproducibility in DVC = ?

Binding code hash (Git) + data hash (.dvc pointer) in one commit, so checking out the commit restores the exact code AND data.

Fresh clone, dvc checkout restores nothing — why and fix?

Cache is empty; run dvc pull to fetch data bytes from the remote first.

Recall Feynman: explain to a 12-year-old

Imagine your homework folder. The instructions (code) are one small sheet of paper — easy to photocopy every day. But your giant box of LEGO (the data) is too heavy to photocopy each time you build something. So instead, you write a little sticky note: "Box #A1B2 — the red-and-blue set." You keep the sticky note in your folder (Git), and the actual LEGO boxes go in the garage (remote storage). Anytime you want to rebuild an old project, you read the old sticky note, go grab that exact box from the garage, and you're back to the exact same LEGO you had that day. DVC = the sticky-note system for heavy data.

Connections

Git version control — DVC layers on top of Git, reusing its content-hashing idea.
Reproducibility in ML — data versioning is one leg; the others are code + environment.
MLOps pipelines — dvc.yaml DAGs feed CI/CD retraining.
Experiment tracking (MLflow) — pairs with DVC: MLflow logs metrics, DVC pins data/artifacts.
Object storage (S3/GCS) — the "remote" that actually holds the heavy bytes.
Content-addressable storage — the hashing principle underneath Git, DVC, and Docker layers.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Git ek badhiya tool hai lekin wo sirf chhoti text files (code) ke liye bana hai. Agar tum apna 2 GB ka dataset ya model directly git add kar doge, toh Git uski har version ki poori copy .git/ ke andar rakhega — repo phool ke gubbara ban jayega aur clone karna mushkil. Yahi problem DVC solve karta hai. DVC file ke content ka hash (MD5) nikaalta hai, asli heavy file ko ek cache/remote (jaise S3) me daal deta hai, aur Git me sirf ek chhoti si pointer file (data.csv.dvc) rakhta hai jisme bas wo hash likha hota hai.

Iska matlab ye hua ki tumhare Git commit me code + data dono ka hash ek saath bind ho jaata hai. Reproducibility ka asli meaning yahi hai — koi bhi purana experiment dobara chalana ho toh git checkout <commit> karo (pointer wapas aa gaya) aur phir dvc checkout karo (us hash wala exact data cache/remote se restore ho gaya). Code aur data hamesha match karte rahenge. Yaad rakho: "Git tracks the note, DVC tracks the boat" — chhoti note Git me, bhaari cargo DVC me.

Ek common galti: log dvc add + git commit karke soch lete hain ki teammate ko data mil gaya. Nahi! Commit me sirf pointer gaya, asli bytes tumhare local cache me pade hain. Isliye hamesha dvc push karo taaki data remote pe chala jaaye; teammate git pull phir dvc pull karega. Aur agar data file edit ki hai toh dobara dvc add karna zaroori hai warna DVC ko naya hash pata nahi chalega.

80/20 rule: bas ye yaad rakho — dvc add, git commit, dvc push (save karne ke liye) aur git checkout + dvc checkout (purana version laane ke liye). Bonus me dvc.yaml pipelines hoti hain jo make ki tarah kaam karti hain — agar input (deps) nahi badle toh stage dobara nahi chalti. Isse pura training flow reproducible ho jaata hai.

WHY does this problem even exist?

HOW DVC works — derive it from scratch

The essential commands (the 20% you use 80% of the time)

DVC pipelines (bonus: reproducing the whole flow)

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections