An LLM is a compression of its training data . Garbage in, garbage out — but at web scale the "garbage" isn't obvious: it's duplicates, boilerplate, machine-generated spam, and toxic text hiding among trillions of tokens. The 20% that matters most: aggressive quality filtering + deduplication. These two steps drive most of the downstream quality gain per compute dollar.
Intuition Why raw web scraping failsA raw dump like Common Crawl is ~90% junk for language modeling: HTML boilerplate, navigation menus, SEO spam, adult content, and near-identical copies of the same page. If you train on this:
The model wastes capacity memorizing duplicates instead of learning general patterns.
It learns boilerplate priors ("Click here to subscribe") instead of reasoning.
Benchmark contamination inflates scores that don't reflect real ability.
WHY it works: a fixed compute budget = a fixed number of tokens you can afford to train on. Every junk token you remove is a good token you can add . Curation is really token-budget allocation .
WHAT: classify each document's language. HOW: a fast classifier (e.g. fastText) outputs p ( lang ∣ doc ) p(\text{lang}\mid \text{doc}) p ( lang ∣ doc ) p ( en ) > τ p(\text{en}) > \tau p ( en ) > τ τ = 0.65 \tau=0.65 τ = 0.65 recall vs precision of the target language.
Intuition Why heuristics workHuman-written prose has statistical regularities: sane word lengths, some punctuation, low symbol ratio, few repeated lines. Junk violates these. So we build cheap red-flag detectors .
Intuition Model-based quality (Feynman version)Instead of hand rules, train a classifier: label a known-good set (e.g. Wikipedia, curated books) as positive and random web as negative. The classifier learns p ( high quality ∣ doc ) p(\text{high quality}\mid \text{doc}) p ( high quality ∣ doc ) sample with probability ∝ \propto ∝
Intuition Why dedup matters so muchDuplicates cause train/test leakage and make the model over-weight repeated strings (it can regurgitate them verbatim, a memorization/privacy risk). Removing them improves quality and privacy at once.
Exact dedup (WHAT): hash each document; drop repeats. O ( N ) O(N) O ( N )
Near-dedup via MinHash + LSH (HOW, from scratch):
Block-lists (URL/domain), toxicity classifiers, and regexes for emails/phone numbers. WHY regex-then-classifier: cheap regex removes obvious PII; a classifier catches contextual toxicity. Trade-off: over-filtering removes legitimate medical/legal text, hurting capability. Filter with a threshold you can defend .
WHAT: remove pretraining docs that overlap with eval sets (MMLU, etc.). HOW: n-gram overlap — if a doc shares a ≥ 13 \ge 13 ≥ 13 inflates benchmark scores fraudulently .
Worked example Example 1 — symbol filter decisionDoc: 300 words, 45 symbol chars.
Step: r sym = 45 / 300 = 0.15 r_{\text{sym}} = 45/300 = 0.15 r sym = 45/300 = 0.15 Why this step? Normalize by length so threshold is universal.
Step: 0.15 > 0.1 ⇒ 0.15 > 0.1 \Rightarrow 0.15 > 0.1 ⇒ drop . Why? Symbol density this high almost always means SEO/menu spam, not prose.
Worked example Example 2 — MinHash match probabilityTwo docs with true J = 0.8 J=0.8 J = 0.8 m = 200 m=200 m = 200 = 200 × 0.8 = 160 = 200\times0.8 = 160 = 200 × 0.8 = 160 Why? Each hash matches independently with probability J J J J ^ = 160 / 200 = 0.8 \hat J = 160/200 = 0.8 J ^ = 160/200 = 0.8
Worked example Example 3 — tuning LSH bandsWant candidates when s ≥ 0.8 s\ge0.8 s ≥ 0.8 b = 20 b=20 b = 20 r = 5 r=5 r = 5 m = 100 m=100 m = 100 Step: P ( 0.8 ) = 1 − ( 1 − 0.8 5 ) 20 = 1 − ( 1 − 0.328 ) 20 = 1 − 0.672 20 ≈ 1 − 0.00034 ≈ 0.9997 P(0.8) = 1-(1-0.8^5)^{20} = 1-(1-0.328)^{20} = 1 - 0.672^{20} \approx 1 - 0.00034 \approx 0.9997 P ( 0.8 ) = 1 − ( 1 − 0. 8 5 ) 20 = 1 − ( 1 − 0.328 ) 20 = 1 − 0.67 2 20 ≈ 1 − 0.00034 ≈ 0.9997 Why? Near-certain to catch true dups.
Step: Check a dissimilar pair s = 0.3 s=0.3 s = 0.3 P ( 0.3 ) = 1 − ( 1 − 0.3 5 ) 20 = 1 − ( 1 − 0.00243 ) 20 ≈ 0.0475 P(0.3)=1-(1-0.3^5)^{20}=1-(1-0.00243)^{20}\approx 0.0475 P ( 0.3 ) = 1 − ( 1 − 0. 3 5 ) 20 = 1 − ( 1 − 0.00243 ) 20 ≈ 0.0475 Why? Low false-positive rate — most dissimilar pairs never get compared, saving compute.
Step: The exact 50% crossover for b = 20 , r = 5 b=20, r=5 b = 20 , r = 5 s 0 = ( 1 − 2 − 1 / 20 ) 1 / 5 ≈ 0.489 s_0=(1-2^{-1/20})^{1/5}\approx 0.489 s 0 = ( 1 − 2 − 1/20 ) 1/5 ≈ 0.489 Why? This is where the S-curve wall sits — well below our θ = 0.8 \theta=0.8 θ = 0.8
Worked example Example 4 — upsamplingCode domain has N code = 100 N_{\text{code}}=100 N code = 100 T = 1 T=1 T = 1 w = 0.15 w=0.15 w = 0.15 Step: e code = 0.15 × 1000 / 100 = 1.5 e_{\text{code}} = 0.15 \times 1000 / 100 = 1.5 e code = 0.15 × 1000/100 = 1.5 Why? 1.5 < 4 1.5<4 1.5 < 4
Common mistake "Just filter harder — more filtering = better model."Why it feels right: cleaner data should mean cleaner model. The flaw: aggressive filtering biases the corpus toward a narrow style (e.g. only Wikipedia-like text), losing dialects, informal language, and rare-but-valuable content. Also shrinks token budget below what scaling laws demand. Fix: filter to remove junk , not diversity ; prefer soft (probabilistic) filters and validate on held-out downstream tasks.
Common mistake "Exact dedup is enough."Why it feels right: duplicates are duplicates. The flaw: most web duplication is near -duplicate (one word changed, different ads). Exact hashing misses it entirely. Fix: MinHash/LSH near-dedup at document and substring level.
Common mistake "MinHash directly measures similarity, so I don't need many hashes."Why it feels right: the estimator is unbiased. The flaw: with few hashes the variance is huge — a single hash gives a 0/1 estimate. Fix: use m m m ≈ J ( 1 − J ) / m \approx J(1-J)/m ≈ J ( 1 − J ) / m
Common mistake "The LSH threshold is just( 1 / b ) 1 / r (1/b)^{1/r} ( 1/ b ) 1/ r Why it feels right: it's a common textbook rule-of-thumb and roughly locates the steep region. The flaw: it's only an approximation; the actual 50% crossover is s 0 = ( 1 − 2 − 1 / b ) 1 / r s_0=(1-2^{-1/b})^{1/r} s 0 = ( 1 − 2 − 1/ b ) 1/ r Fix: solve 1 − ( 1 − s 0 r ) b = 1 2 1-(1-s_0^{r})^b=\tfrac12 1 − ( 1 − s 0 r ) b = 2 1 b , r b,r b , r
Common mistake "Decontamination is optional; overlap is tiny."Why it feels right: web is huge, eval sets are small. The flaw: popular benchmarks are copied all over the web; even small leakage massively inflates scores. Fix: always run n-gram decontamination before reporting.
Why does deduplication improve both quality AND privacy?
Derive why Pr [ minhash ( A ) = minhash ( B ) ] = J ( A , B ) \Pr[\text{minhash}(A)=\text{minhash}(B)]=J(A,B) Pr [ minhash ( A ) = minhash ( B )] = J ( A , B )
What does raising the symbol-ratio threshold do to precision/recall?
What is the exact LSH 50% crossover similarity, and how is it derived?
When is upsampling a domain dangerous?
Recall Feynman: explain to a 12-year-oldImagine you're making flashcards to study, but your pile of paper has tons of duplicate cards, some cards are just ads, and a few are the actual test answers someone slipped in. If you memorize this pile you'll waste time on duplicates, learn ad-slogans, and cheat by memorizing test answers (so you can't really do it on the real test). Data cleaning is throwing out the duplicate cards, the ad cards, and the sneaked-in test answers, so what's left actually teaches you to think. To spot "same card written slightly differently," we use a clever trick (MinHash): we give every card a random lottery number and check if two cards keep drawing the same winning number — the more they do, the more alike they are.
"Little Quiet Dogs Can't Deceive Managers"
L anguage → Q uality → D edup → C ontent/PII → D econtamination → M ixing.
Why is curation really "token-budget allocation"? Fixed compute = fixed tokens; removing a junk token lets you add a good one, so cleaning directly reallocates the budget toward useful data.
Define the symbol-to-word ratio and its filter rule. r s y m = S / W r_{sym}=S/W r sy m = S / W ; drop docs with
r s y m > 0.1 r_{sym}>0.1 r sy m > 0.1 since junk/SEO pages spam symbols.
State the MinHash identity. Pr [ minhash ( A ) = minhash ( B ) ] = J ( A , B ) \Pr[\text{minhash}(A)=\text{minhash}(B)]=J(A,B) Pr [ minhash ( A ) = minhash ( B )] = J ( A , B ) , the Jaccard similarity.
Why does the MinHash identity hold? The global minimum hash over
A ∪ B A\cup B A ∪ B is uniformly likely to be any element; it lands in
A ∩ B A\cap B A ∩ B (making mins agree) with probability
∣ A ∩ B ∣ / ∣ A ∪ B ∣ = J |A\cap B|/|A\cup B|=J ∣ A ∩ B ∣/∣ A ∪ B ∣ = J .
What is the LSH candidate probability formula? P ( s ) = 1 − ( 1 − s r ) b P(s)=1-(1-s^{r})^{b} P ( s ) = 1 − ( 1 − s r ) b for
b b b bands of
r r r rows; an S-curve.
What is the exact LSH 50% crossover similarity? s 0 = ( 1 − 2 − 1 / b ) 1 / r s_0=(1-2^{-1/b})^{1/r} s 0 = ( 1 − 2 − 1/ b ) 1/ r , from solving
1 − ( 1 − s 0 r ) b = 1 2 1-(1-s_0^{r})^b=\tfrac12 1 − ( 1 − s 0 r ) b = 2 1 . The rough rule
( 1 / b ) 1 / r (1/b)^{1/r} ( 1/ b ) 1/ r is only an approximation.
Why use near-dedup, not just exact dedup? Most web duplication differs by ads/one word; exact hashing misses near-duplicates that MinHash/LSH catch.
What is decontamination and a typical rule? Removing pretraining docs overlapping eval benchmarks; drop docs sharing a ≥13-gram with a benchmark example.
Formula for effective epochs on a domain. e d = w d T / N d e_d = w_d T / N_d e d = w d T / N d ; keep it below ~4 to avoid memorization when upsampling.
Steel-man: why "filter harder is always better" is wrong. It biases toward narrow style, kills diversity, and shrinks the token budget below scaling-law needs; use soft filters validated downstream.
Why does dedup help privacy? Fewer repeated strings means the model is far less likely to memorize and regurgitate specific documents/PII.
Variance of the MinHash Jaccard estimate with m m m Approximately
J ( 1 − J ) / m J(1-J)/m J ( 1 − J ) / m ; decreases as
m m m grows, so use hundreds of hashes.
Why threshold instead of argmax in language ID? A threshold trades recall vs precision for the target language and discards low-confidence, code-switched, or short docs.
Scaling Laws for LLMs — curation decides how many useful tokens fill the budget.Tokenization — cleaning happens on text; tokenizer stats depend on the cleaned corpus.Deduplication and Memorization — dedup reduces verbatim memorization/privacy leakage.Benchmark Contamination — decontamination protects eval validity.Data Mixture and Domain Weighting — upsampling weights w d w_d w d MinHash and LSH — the near-dedup machinery.Fine-Tuning and Instruction Data — a later, smaller, higher-quality curation stage.
remove junk token = add good token
Symbol-to-word ratio r_sym
Wasted capacity on duplicates
Intuition Hinglish mein samjhoDekho, ek LLM basically apne training data ka ek smart compression hai. Agar tum raw web data (jaise Common Crawl) seedha model me daal doge, to usme 90% kachra hota hai — duplicate pages, ads, "Add to cart" jaisa boilerplate, SEO spam. Model us kachre ko yaad karne me apni capacity waste kar deta hai, actual reasoning seekhne ke bajaye. Isliye curation matlab apne token-budget ko sahi jagah lagana — har junk token hatao, ek accha token add karo.
Pipeline simple hai: pehle language ID (target language rakho), phir quality filter (symbol-to-word ratio r = S / W r=S/W r = S / W r > 0.1 r>0.1 r > 0.1 dedup , phir content/PII filter , phir decontamination , aur last me domain mixing . Sabse zyada faayda quality filter aur dedup se aata hai — yahi tumhara 80/20 hai.
Dedup ka jaadu MinHash hai. Har document ko shingles (k-word tukde) ka set maano. Do docs ki similarity Jaccard J = ∣ A ∩ B ∣ / ∣ A ∪ B ∣ J=|A\cap B|/|A\cup B| J = ∣ A ∩ B ∣/∣ A ∪ B ∣ N 2 N^2 N 2 minimum number match karta hai exactly probability J J J J J J LSH se sirf similar-lagne wale pairs hi check karte hain — S-curve P ( s ) = 1 − ( 1 − s r ) b P(s)=1-(1-s^r)^b P ( s ) = 1 − ( 1 − s r ) b
