Pretraining data curation and cleaning: Question bank

Notation reminder, so no symbol is unearned here:

$r_{sym} = S / W$ = symbol chars divided by words (the symbol-to-word ratio).
$J (A, B)$ = Jaccard similarity, the fraction of shingles two docs share.
$P (s) = 1 - (1 - s^{r})^{b}$ = probability an MinHash and LSH scheme makes two docs of similarity $s$ into a candidate pair.
$e_{d} = w_{d} T / N_{d}$ = effective epochs a domain gets: its weight $w_{d}$ times total tokens $T$ , divided by how many tokens it actually has $N_{d}$ .

True or false — justify

More filtering always produces a better model

False — over-filtering narrows style and deletes rare-but-legitimate text (medical, legal, non-standard dialects), reducing capability even as surface "cleanliness" rises. Curation is token-budget allocation, not maximal deletion.

Deduplication is purely a quality trick, not a safety concern

False — it does both at once: fewer verbatim repeats means the model can't regurgitate training strings, which is a genuine privacy/memorization win alongside the quality gain.

Exact deduplication makes near-dedup unnecessary

False — exact dedup only catches byte-identical copies; the web is full of near-identical pages (same article, different ads/timestamps) that only Jaccard/MinHash catches.

A raw Common Crawl dump is roughly usable as-is with light cleaning

False — it is ~90% junk (boilerplate, SEO spam, duplicates), so light cleaning leaves the model memorizing "Click here to subscribe" instead of learning to reason.

The symbol ratio

r_{sym} = S / W

is better than a raw symbol count

S

True — the ratio is length-invariant, so a tweet and a long article are judged on the same scale; a raw count would unfairly flag every long document.

Upsampling a scarce high-value domain is always safe if its weight is small

False — what matters is effective epochs

e_{d} = w_{d} T / N_{d}

, not

w_{d}

alone; a small domain with a modest weight can still be repeated

e_{d} ≫ 4

times and overfit.

Language ID should keep the single highest-probability language (argmax)

False — argmax has no confidence floor, so short/code-switched docs get force-classified; a threshold

p (en) > τ

lets you tune recall vs precision instead.

Decontamination inflates your benchmark scores

False — skipping decontamination inflates them fraudulently; removing benchmark-overlapping docs makes the reported scores honest reflections of real ability.

Spot the error

J = 0.8

means 80% of the two documents' words are identical."

Error —

J

is over shingle sets (k-word sequences), and it's intersection over union, not a raw word-overlap percentage; identical word bags in different order can still have low

J

"MinHash needs the actual hash values to agree in size to estimate similarity."

Error — only the event

minhash (A) = minhash (B)

(a match/no-match) matters; the collision probability equals

J

, so we just count matching hashes, not compare magnitudes.

"With

b

bands and

r

rows the LSH crossover is

s_{0} \approx (1/ b)^{1/ r}

Error — that's only a rough approximation; the exact 50% crossover is

s_{0} = (1 - 2^{- 1/ b})^{1/ r}

, and you tune

b, r

so this wall sits at your target

θ

"A soft (probabilistic) quality filter is strictly worse than a hard threshold — it lets junk through."

Error — sampling with probability

\propto

score deliberately keeps some lower-scored docs to preserve stylistic diversity; a hard cutoff can over-narrow the model to Wikipedia-like prose.

"Regex PII removal makes the toxicity classifier redundant."

Error — regex catches obvious patterns (emails, phones) but not context; toxicity is meaning-dependent and needs a classifier. They cover different failure modes.

"A 5-gram decontamination check is safer than a 13-gram check because it's stricter."

Error — too-short n-grams cause huge accidental overlap (common phrases), deleting good data; 13-grams are long enough that coincidental overlap is astronomically unlikely, so they flag real leakage only.

"Removing junk tokens shrinks the training set, so it must hurt a scaling-law model."

Error — the compute budget fixes how many tokens you can afford; every junk token removed frees room for a good token added. Quality per token rises, not falls.

Why questions

Why divide symbols by words instead of thresholding raw symbol count?

Because the ratio is length-invariant — the same threshold (

r_{sym} > 0.1

) correctly judges a short and a long document; a raw count would flag all long documents regardless of quality.

Why does

Pr [minhash (A) = minhash (B)]

equal

J (A, B)

exactly?

A uniform-random hash makes every element of

A \cup B

equally likely to be the minimum; the mins agree exactly when that minimum lies in

A \cap B

, and

A \cap B

is a

J

fraction of

A \cup B

Why does LSH split

m

hashes into

b

bands instead of comparing all

m

at once?

Requiring all

m

to match is far too strict; banding makes docs candidates if they match any whole band, producing a tunable S-curve so true dups cross the wall while dissimilar docs almost never do — dodging the

O (N^{2})

all-pairs cost.

Why upsample code and math domains above their natural web share?

They are scarce but information-dense, so raising

w_{d}

gives the model more exposure per token; you just keep

e_{d} = w_{d} T / N_{d}

below ~4 so repetition doesn't tip into memorization.

Why use a threshold in language ID rather than always keeping the top language?

Because confidence is meaningful — short or code-switched docs have low

p (en)

; a threshold trades recall against precision, whereas argmax silently keeps garbage-labeled documents.

Why does deduplication improve quality and privacy simultaneously?

Repeated strings both waste model capacity (over-weighting duplicates) and make verbatim memorization easy; deleting them frees capacity for general patterns and removes the very strings that could be leaked.

Why is model-based quality filtering trained with Wikipedia as "positive" and random web as "negative"?

It teaches a cheap classifier

p (high quality ∣ doc)

to recognize the statistical texture of curated prose, so you can score a trillion documents without hand-writing a rule for every junk pattern.

Edge cases

A document has 0 words but 12 symbol characters — what does the symbol filter do?

r_{sym} = 12/0

is undefined (division by zero); a robust pipeline drops empty/near-empty docs before computing the ratio, or treats undefined as "drop."

A legitimate scientific table is dense in symbols and short lines — is the heuristic filter fair to it?

No — heuristic red-flag detectors will falsely flag it; this is exactly why model-based filters or per-domain rules exist, and why "filter harder" isn't free.

Two docs are byte-identical but scraped from different URLs — which stage catches them, exact or near dedup?

Exact dedup (

O (N)

hashing) catches them immediately; near-dedup would too, but you run cheap exact dedup first so MinHash only handles the harder near-duplicate cases.

A domain's weight

w_{d}

is high but it has more tokens than the budget can use — what happens to

e_{d}

e_{d} = w_{d} T / N_{d}

falls below 1, meaning the model sees only a fraction of that domain in one pass — undertraining it, the opposite risk from the

e_{d} ≫ 4

memorization case.

A benchmark example is only 8 words long — can 13-gram decontamination protect against its leakage?

No — there's no 13-gram to match, so short benchmark items slip through n-gram checks; they need shorter-n or exact-match handling with tighter accidental-overlap safeguards.

LSH with

b = 1, r = m

(one band, all rows) — what does the S-curve become?

P (s) = 1 - (1 - s^{m})^{1} = s^{m}

, a very strict curve requiring near-identical docs; almost no false positives but you miss moderate near-duplicates — the extreme rigidity end of the

b, r

trade-off.

A doc scores just below the quality threshold under a hard filter but is genuine niche writing — what's lost?

It's deleted entirely, eroding stylistic and topical diversity; a soft filter that samples

\propto

score would keep it with some probability, protecting coverage of rare-but-real text.

Recall Self-test before moving on

Give the exact LSH crossover formula and say why the " $(1/ b)^{1/ r}$ " version is only approximate. Exact crossover ::: $s_{0} = (1 - 2^{- 1/ b})^{1/ r}$ ; the $(1/ b)^{1/ r}$ form drops the $2^{- 1/ b}$ correction and only tracks the curve's steep region roughly. State when deduplication under-serves you and you'd still want near-dedup. When ::: When copies aren't byte-identical (ads/timestamps differ) — exact hashing misses them, so MinHash + LSH on shingles is required.

See also: Data Mixture and Domain Weighting, Fine-Tuning and Instruction Data, Tokenization.