Pretraining data curation and cleaning

4.3.4 · AI-ML › Pretraining & Fine-Tuning LLMs

The pipeline (WHAT the stages are)

Figure — Pretraining data curation and cleaning

HOW each stage works (derived from first principles)

2. Quality filtering — deriving a heuristic score

3. Deduplication — deriving why and how

Exact dedup (WHAT): har document ko hash karo; repeats drop karo. $O (N)$ , lekin sirf byte-identical copies pakad sakta hai.

Near-dedup via MinHash + LSH (HOW, from scratch):

Formula MinHash estimator (derivation)

Shingles par ek random hash $h$ lo. Define karo $minhash_{h} (A) = min_{x \in A} h (x)$ . Key fact: $Pr [minhash_{h} (A) = minhash_{h} (B)] = J (A, B) .$ Why? $A \cup B$ par minimum hash uniformly likely hai kisi bhi element ke liye (hash uniform-random ordering hai). Yeh $A \cap B$ mein land karta hai exactly tab jab dono mins agree karein, aur $A \cap B$ , $A \cup B$ ka $J (A, B)$ fraction hai. Isliye $m$ independent hashes ke saath, matches ka fraction $J$ ka unbiased estimator hai, jiska variance $\to 0$ hota hai jaise $m$ barhta hai.

LSH to avoid $O (N^{2})$ : $m$ hashes ko $b$ bands mein split karo, har band mein $r$ rows ( $m = b r$ ). Koi bhi poora band share karne wale docs candidate pairs ban jaate hain. Do docs jo similarity $s$ rakhte hain unke candidates banne ki probability: $P (s) = 1 - (1 - s^{r})^{b} .$ Yeh ek S-curve hai. 50% crossover similarity $s_{0}$ (jahan $P (s_{0}) = \frac{1}{2}$ ) satisfy karta hai $s_{0} = (1 - 2^{- 1/ b})^{1/ r},$ jo $1 - (1 - s_{0}^{r})^{b} = \frac{1}{2} \Rightarrow (1 - s_{0}^{r})^{b} = \frac{1}{2} \Rightarrow s_{0}^{r} = 1 - 2^{- 1/ b}$ set karke derive hota hai. Curve ke steep point ke liye ek common rule-of-thumb approximation $s_{0} \approx (1/ b)^{1/ r}$ hai, lekin exact crossover $(1 - 2^{- 1/ b})^{1/ r}$ hai — $b, r$ tune karo taaki yeh wall aapke target $θ$ par baithe.

4. Content & PII filtering

5. Decontamination

6. Domain mixing / upsampling

Worked examples

Common mistakes (Steel-manned)

Active recall

Recall Feynman: explain to a 12-year-old

Socho tum flashcards bana rahe ho padhne ke liye, lekin tumhare papers ke dher mein bahut saare duplicate cards hain, kuch cards sirf ads hain, aur kuch mein actual test ke answers kisi ne daal diye hain. Agar tum yeh sara dher memorize karo toh duplicates par time waste hoga, ad-slogans seekhoge, aur test answers memorize karke cheat hoga (toh real test mein actually kuch nahi kar paoge). Data cleaning duplicate cards, ad cards, aur sneaked-in test answers ko nikaal dena hai, taaki jo bacha woh actually sochna sikhaye. "Thoda alag likha same card" pakadne ke liye hum ek clever trick use karte hain (MinHash): hum har card ko ek random lottery number dete hain aur check karte hain ki kya do cards same winning number baar baar draw karte hain — jitna zyada karte hain, utne zyada same hain.

Flashcards

Curation ko "token-budget allocation" kyun kaha jaata hai?

Fixed compute = fixed tokens; ek junk token remove karne se ek achha token add ho sakta hai, isliye cleaning directly budget ko useful data ki taraf reallocate karti hai.

Symbol-to-word ratio define karo aur uska filter rule batao.

r_{sy m} = S / W

;

r_{sy m} > 0.1

wale docs drop karo kyunki junk/SEO pages symbols spam karte hain.

MinHash identity state karo.

Pr [minhash (A) = minhash (B)] = J (A, B)

, yani Jaccard similarity.

MinHash identity hold kyun karta hai?

The global minimum hash over

A \cup B

uniformly likely hai kisi bhi element ke liye; yeh

A \cap B

mein land karta hai (mins ko agree karata hua) probability

∣ A \cap B ∣/∣ A \cup B ∣ = J

ke saath.

LSH candidate probability formula kya hai?

P (s) = 1 - (1 - s^{r})^{b}

for

b

bands of

r

rows; yeh ek S-curve hai.

LSH ka exact 50% crossover similarity kya hai?

s_{0} = (1 - 2^{- 1/ b})^{1/ r}

1 - (1 - s_{0}^{r})^{b} = \frac{1}{2}

solve karke. Rule

(1/ b)^{1/ r}

sirf ek approximation hai.

Near-dedup kyun use karo, exact dedup hi kyun nahi?

Zyaatar web duplication mein ads/ek word ka fark hota hai; exact hashing near-duplicates miss kar deta hai jo MinHash/LSH pakad leta hai.

Decontamination kya hai aur ek typical rule kya hai?

Pretraining docs jo eval benchmarks ke saath overlap karte hain unhe remove karna; jo docs kisi benchmark example ke saath

\geq

13-gram share karein unhe drop karo.

Kisi domain par effective epochs ka formula.

e_{d} = w_{d} T / N_{d}

; upsampling karte waqt memorization se bachne ke liye ise ~4 se neeche rakho.

Steel-man: "filter harder hamesha better hai" kyun galat hai?

Yeh narrow style ki taraf bias karta hai, diversity khatam karta hai, aur token budget ko scaling-law needs se neeche shrink karta hai; soft filters use karo jo downstream validate hoon.

Dedup privacy mein kaise help karta hai?

Kam repeated strings ka matlab hai model specific documents/PII memorize aur regurgitate karne ki bahut kam probability rakhta hai.

m

hashes ke saath MinHash Jaccard estimate ki variance kya hai?

Approximately

J (1 - J) / m

;

m

badhne se kam hoti hai, isliye hundreds of hashes use karo.

Language ID mein argmax ki jagah threshold kyun?

Ek threshold target language ke liye recall vs precision trade karta hai aur low-confidence, code-switched, ya short docs ko discard karta hai.

Connections

Scaling Laws for LLMs — curation decide karta hai ki budget mein kitne useful tokens fit honge.
Tokenization — cleaning text par hoti hai; tokenizer stats cleaned corpus par depend karte hain.
Deduplication and Memorization — dedup verbatim memorization/privacy leakage reduce karta hai.
Benchmark Contamination — decontamination eval validity protect karta hai.
Data Mixture and Domain Weighting — upsampling weights $w_{d}$ .
MinHash and LSH — near-dedup ki machinery.
Fine-Tuning and Instruction Data — baad ka, chhota, higher-quality curation stage.

WHY curate at all?