Data versioning (DVC) — Hinglish notes

5.3.4 · AI-ML › MLOps & Deployment

YEH problem exist hi kyun karti hai?

Git LFS kyun nahi use karte? Git LFS bhi pointers use karta hai, lekin yeh Git hosting quotas se tied hai, storage-provider-limited hai, aur iska ML pipelines ya experiments ka koi notion nahi hai. DVC storage (koi bhi S3/GCS/SSH/local) ko Git se decouple karta hai aur pipeline reproducibility add karta hai.

DVC KAISE kaam karta hai — scratch se derive karo

Chalo conceptually DVC khud banate hain. Maano tumhare paas data.csv (2 GB) hai.

Step 1 — Content addressing. File ke content ka hash compute karo: $h = MD5 (bytes of file)$

Step 2 — Heavy file ko cache mein move karo, uske hash ke naam se rename karke:

.dvc/cache/a1/b2c3...   (the real 2 GB)

Step 3 — Git ke under ek tiny pointer chhodo. DVC data.csv.dvc create karta hai:

outs:
- md5: a1b2c3...
  size: 2147483648
  path: data.csv

Yeh .dvc file ~100 bytes ki hai → Git ise khushi se track karta hai.

Step 4 — Restore karo. Ek .dvc pointer diya ho toh hash cache mein dhundo (ya dvc pull karo remote se), aur data.csv ko tumhari working directory mein re-create karo.

Essential commands (woh 20% jo 80% time use hote hain)

Worked example Ek poora workflow

dvc init                      # create .dvc/ (like git init)
dvc remote add -d store s3://mybucket/dvcstore
dvc add data/train.csv        # -> creates train.csv.dvc, moves file to cache
git add data/train.csv.dvc data/.gitignore
git commit -m "add v1 of training data"
dvc push                      # upload actual data to S3 remote

git add mein .gitignore kyun? dvc add automatically data/.gitignore likhta hai taaki Git asli file ko ignore kare aur sirf pointer track kare — galti se 2 GB commit na ho jaaye.

Worked example Purana dataset version restore karna

git checkout v1-commit-hash   # restores the .dvc pointer (old hash)
dvc checkout                  # reads pointer, restores matching data from cache/remote

Do steps kyun? Git pointer ko move karta hai; DVC woh data move karta hai jisko pointer point karta hai. Separation of concerns = halka Git, heavy storage kahin bhi.

DVC pipelines (bonus: poora flow reproduce karna)

Definition

dvc.yaml stage Ek stage cmd, uske deps (inputs), aur outs (outputs) declare karta hai. DVC deps ko hash karta hai; agar kuch nahi badla, toh yeh re-run skip karta hai (make ki tarah).

stages:
  train:
    cmd: python train.py
    deps: [train.py, data/train.csv]
    outs: [model.pkl]

dvc repro se run karo. Kyun? "Kaun sa data + code is model ko produce kiya?" — is sawaal ko tribal memory se nikaalkaar ek tracked DAG mein daalta hai.

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Woh kaun si problem hai jo DVC solve karta hai jo plain Git nahi kar sakta?

Large binary data/models ko version karna — Git har version ki full copies store karta hai aur binaries ko efficiently diff nahi kar sakta; DVC Git mein tiny hash pointers rakhta hai aur heavy data remote storage mein.

dvc add data.csv actually kya create aur move karta hai?

Ek chhota data.csv.dvc pointer banata hai (jisme MD5 hash + size + path hota hai), asli file ko .dvc/cache/ mein move karta hai, aur .gitignore update karta hai taaki Git asli file ignore kare.

Commit karne ke baad dvc push kyun run karna padta hai?

Git commit sirf ~100-byte pointer store karta hai; actual bytes tumhari local cache mein rehti hain jab tak dvc push unhe remote par upload nahi karta.

Purana dataset version restore karne ke liye kaun se do commands chahiye?

git checkout <commit> (pointer restore karta hai) phir dvc checkout (matching data cache/remote se restore karta hai).

DVC filename ki jagah file content ko hash kyun karta hai?

Identical content → identical hash → automatic deduplication; koi bhi change hash flip kar deta hai toh naye versions reliably detect ho jaate hain.

DVC pipeline stage kis cheez se bana hota hai?

Ek cmd, uske deps (inputs), aur outs (outputs); DVC re-running skip karta hai agar deps unchanged hain (make-jaisa), dvc repro ke zariye.

DVC mein Reproducibility = ?

Code hash (Git) + data hash (.dvc pointer) ko ek hi commit mein bind karna, taaki commit checkout karne par exact code AUR data dono restore ho jayein.

Fresh clone, dvc checkout kuch restore nahi karta — kyun aur fix?

Cache empty hai; pehle dvc pull run karo remote se data bytes fetch karne ke liye.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tumhara homework folder hai. Instructions (code) ek chhoti si sheet of paper hai — har roz photocopy karna easy hai. Lekin tumhara LEGO ka bada box (data) itna heavy hai ki har baar photocopy nahi ho sakta jab tum kuch banate ho. Toh instead, tum ek chhota sticky note likhte ho: "Box #A1B2 — the red-and-blue set." Yeh sticky note apne folder mein rakhte ho (Git), aur asli LEGO boxes garage mein jaate hain (remote storage). Jab bhi koi purana project rebuild karna ho, purana sticky note padho, garage se woh exact box uthao, aur tum bilkul wahi LEGO ke saath waapis aa jate ho jo us din tha. DVC = heavy data ke liye sticky-note system.

Connections

Git version control — DVC Git ke upar layer karta hai, uski content-hashing idea reuse karta hai.
Reproducibility in ML — data versioning ek leg hai; baaki code + environment hain.
MLOps pipelines — dvc.yaml DAGs CI/CD retraining ko feed karte hain.
Experiment tracking (MLflow) — DVC ke saath pair karta hai: MLflow metrics log karta hai, DVC data/artifacts pin karta hai.
Object storage (S3/GCS) — woh "remote" jo actually heavy bytes hold karta hai.
Content-addressable storage — Git, DVC, aur Docker layers ke neeche ka hashing principle.