Pretraining data curation and cleaning: Exercises

Before we start, two symbols we will reuse everywhere. A ratio like $r = S / W$ just means "how many of one thing per one of another" — dividing makes it length-independent, so a tweet and an essay are judged on the same scale. A probability $P$ is a number between $0$ and $1$ telling you how often something happens if you repeated the experiment forever ( $0$ = never, $1$ = always). Keep those two pictures in your head.

The whole page follows one climb. The picture below is the map — glance at it now, and we will return to it as we move up each rung.

Figure — Pretraining data curation and cleaning

Read it left to right: the dots are the five levels, and the arrows call out what changes about your job at each rung — L2 is "plug into a formula," L3 is "reason about trade-offs," L5 is "handle the broken inputs nobody tested." Every exercise below is one dot on that line.

Level 1 — Recognition

Goal: read a document's stats and fire the correct filter rule. No cleverness, just "which rule and does it trip?" — the leftmost blue dot on the ladder figure above.

Recall Solution 1.1

WHAT we do: divide symbols by words. $r_{sym} = \frac{S}{W} = \frac{40}{500} = 0.08.$ WHY divide, not just count 40: a raw count of 40 would flag every long article. The ratio asks "per word, how symbol-heavy is this?" — length-invariant. Decision: $0.08 \leq 0.1$ , so the rule does not trip → KEEP. This looks like ordinary prose.

Recall Solution 1.2

WHAT the symbol means: $f_{dup-line}$ is just "duplicate lines $D$ over total lines $L$ " — a ratio, so it is length-invariant exactly like $r_{sym}$ . $f_{dup-line} = \frac{D}{L} = \frac{18}{50} = 0.36.$ WHY this rule exists: repeated lines ("Add to cart / Add to cart / …") are the fingerprint of a templated junk page, not human writing. $0.36 > 0.3$ → DROP.

Recall Solution 1.3

WHAT $p (en ∣ doc)$ means: the classifier's confidence, between $0$ and $1$ , that the document is English. $0.58 < 0.65$ → DROP (below threshold). WHY a threshold, not argmax: even if English is the top guess, a low-confidence doc is often code-switched or too short to be clean prose. The threshold trades recall for precision.

Level 2 — Application

Goal: plug numbers into the pipeline formulas — MinHash, LSH, mixture epochs — and get a number out. This is the green dot on the ladder figure: the job is mechanical substitution.

Recall Solution 2.1

Key fact (from parent): for each hash, $Pr [minhash match] = J$ . So matches follow "flip a $J$ -biased coin $m$ times." WHY expected matches $= m \cdot J$ (linearity of expectation): write the total matches as a sum $X = X_{1} + X_{2} + \dots + X_{m}$ , where $X_{i} = 1$ if hash $i$ matches and $0$ if not. The expected value of any sum equals the sum of expected values — this is true even if the terms were dependent, and it needs no independence at all. Each $X_{i}$ is a coin that is $1$ with probability $J$ , so $E [X_{i}] = J$ . Adding $m$ of them: $E [X] = m \cdot J$ . Expected matches $= m \cdot J = 240 \times 0.75 = 180.$ Estimator: $\hat{J} = \frac{matches}{m} = \frac{180}{240} = 0.75$ — unbiased, recovering the truth. WHY MinHash and not compute $J$ directly? Direct $J$ over all pairs is $O (N^{2})$ ; MinHash turns a set-overlap question into cheap coin-flip counting.

Recall Solution 2.2

Step 1 — one band matches: a band of $r = 4$ rows matches only if all 4 hashes agree, probability $s^{r} = 0. 9^{4} = 0.6561$ . Step 2 — one band fails: $1 - 0.6561 = 0.3439$ . Step 3 — all $b = 25$ bands fail: $0.343 9^{25} \approx 2.7 \times 1 0^{- 12}$ . Step 4 — at least one band matches (that's what we want): $P (0.9) = 1 - 0.343 9^{25} \approx 0.99999999999.$ WHY this shape (an S-curve): "share any whole band" makes catching easy for high $s$ and hard for low $s$ — exactly the wall we want. You will see this curve plotted in the Level 3 figure.

Recall Solution 2.3

$e_{math} = \frac{0.10 \times 800}{20} = \frac{80}{20} = 4.0 epochs .$ WHY watch $e_{d}$ : repeating a scarce domain more than ~4 times (memorization risk) starts overfitting. Here $e = 4.0$ sits right on the caution line — acceptable but do not push $w$ higher.

Level 3 — Analysis

Goal: reason about trade-offs — where does the wall sit, what does a threshold cost you? The yellow dot on the ladder.

The figure below plots the LSH S-curve $P (s) = 1 - (1 - s^{r})^{b}$ for three configurations. The x-axis is true similarity $s$ ; the y-axis is the probability the pair becomes a candidate. The dashed horizontal line is $P = 0.5$ , and each dot marks that config's wall $s_{0}$ — the similarity where the curve crosses $50%$ . Notice how adding rows $r$ (green vs blue) slides the wall to the right. The yellow dotted vertical line is our near-dup target $θ = 0.8$ ; we want the wall to sit below it so real duplicates land in the flat near- $1$ top.

Recall Solution 3.1

Step 1: $2^{- 1/20} = 2^{- 0.05} \approx 0.96594$ , so $1 - 2^{- 1/20} \approx 0.03406$ . Step 2: $s_{0} = 0.0340 6^{1/5} = 0.0340 6^{0.2} \approx 0.5065$ . Interpretation (read it off the figure): the S-curve crosses 50% at $s_{0} \approx 0.51$ , left of the yellow $θ = 0.8$ line. Since $0.51 < 0.8$ , by the time similarity reaches your target the curve is already near 1 — true dups are safely captured. WHY the exact form beats the rule-of-thumb $(1/ b)^{1/ r}$ : the approximation gives $(1/20)^{1/5} = 0.0 5^{0.2} \approx 0.549$ , close but slightly off. When you're tuning a wall precisely against $θ$ , use the exact $(1 - 2^{- 1/ b})^{1/ r}$ .

Recall Solution 3.2

(a) Precision ↑ — a stricter bar means the docs you do keep are more surely English (fewer false English).
(b) Recall ↓ — genuine but short/code-switched English docs scoring $0.65$ – $0.85$ are now wrongly dropped (false negatives rise).
(c) Total tokens ↓ — you discard more, shrinking the corpus. WHY this matters for training: fewer tokens can hurt an LLM that is token-starved (scaling laws say more good tokens help). Threshold choice is a precision–recall–budget triangle, not a "higher is better" knob.

Recall Solution 3.3

$r_{sym} = \frac{60}{200} = 0.30 > 0.1 \Rightarrow rule says DROP .$ But this is good content! WHY it mis-fires: the heuristic assumes symbols signal SEO spam, an assumption that fails for math/code where symbols are the content. Fix in practice: route symbol-heavy docs through a model-based quality classifier, or set domain-specific thresholds — a single global rule is too blunt for code/math domains.

Level 4 — Synthesis

Goal: combine multiple stages / design a config to hit a spec. The red dot on the ladder — no single formula hands you the answer.

Recall Solution 4.1

Use $s_{0} = (1 - 2^{- 1/ b})^{1/ r}$ :

$(16, 8)$ : $1 - 2^{- 1/16} = 0.04239$ ; $s_{0} = 0.0423 9^{1/8} \approx 0.6733$ .
$(8, 16)$ : $1 - 2^{- 1/8} = 0.08300$ ; $s_{0} = 0.0830 0^{1/16} \approx 0.8557$ .
$(32, 4)$ : $1 - 2^{- 1/32} = 0.02143$ ; $s_{0} = 0.0214 3^{1/4} \approx 0.3826$ . Distances to $0.8$ : $∣0.673 - 0.8∣ = 0.127$ , $∣0.856 - 0.8∣ = 0.056$ , $∣0.383 - 0.8∣ = 0.417$ . Winner: $(b, r) = (8, 16)$ , wall at $\approx 0.856$ — closest to target. WHY the pattern: more rows per band (larger $r$ ) = each band is stricter = wall moves right (toward high similarity). Fewer, longer bands catch only very-similar pairs.

Recall Solution 4.2

Multiply survival fractions in sequence: $1000 \times 0.60 \times 0.50 \times (1 - 0.30) = 1000 \times 0.60 \times 0.50 \times 0.70 = 210 B .$ WHY multiply (not add) the losses: each stage acts on the survivors of the previous stage, so fractions compound. 79% of the raw crawl was junk — a realistic figure.

Recall Solution 4.3

Invert $e_{d} = w T / N \Rightarrow w = e N / T$ : $w_{code} = \frac{3 \times 15}{300} = \frac{45}{300} = 0.15.$ Check: $e = 0.15 \times 300/15 = 3.0$ ✓, and $3 < 4$ so memorization risk stays low. See Data Mixture and Domain Weighting for how the remaining $0.85$ splits across other domains.

Level 5 — Mastery

Goal: edge cases, degenerate inputs, and why-it-breaks reasoning. The white dot at the top of the ladder — the scenarios nobody tested until production.

Recall Solution 5.1

(a) $J = 0$ : match probability $0$ → expected matches $= 0$ . Disjoint docs never collide (before hashing collisions, ignored). (b) $J = 1$ : probability $1$ → all $m$ minhashes match; $\hat{J} = 1$ exactly. WHY: identical shingle sets have $A \cap B = A \cup B$ , so the two minima always land on the same element — every hash agrees. (c) Each hash is a Bernoulli( $J$ ) trial; $\hat{J}$ is their mean, so $Var (\hat{J}) = \frac{J ( 1 - J )}{m} = \frac{0.6 \times 0.4}{50} = \frac{0.24}{50} = 0.0048.$ WHY variance shrinks with $m$ : averaging more independent coin flips squeezes the estimate toward the truth — the $1/ m$ is the whole reason we use many hashes.

Recall Solution 5.2

(a) $s = 0$ : $s^{r} = 0^{r} = 0$ , so $P (0) = 1 - (1 - 0)^{b} = 1 - 1^{b} = 1 - 1 = 0$ . A totally dissimilar pair is never a candidate — the curve starts pinned at the floor. (b) $s = 1$ : $s^{r} = 1^{r} = 1$ , so $P (1) = 1 - (1 - 1)^{b} = 1 - 0^{b} = 1 - 0 = 1$ . An identical pair is always a candidate — the curve ends pinned at the ceiling. (c) So the whole S-curve is trapped between the two endpoints $(0, 0)$ and $(1, 1)$ you just computed; every real config only bends the curve between those anchors. This mirrors the MinHash endpoints $J = 0 \Rightarrow$ 0 matches and $J = 1 \Rightarrow$ all matches from Exercise 5.1 — the two tools agree at the extremes, as they must. WHY it matters: if a config ever gave $P (0) > 0$ or $P (1) < 1$ you would know the formula was mis-coded — these endpoints are a free correctness check.

Recall Solution 5.3

$r_{sym} = S /0$ is division by zero — undefined. A naive implementation crashes or yields $\infty$ . Correct handling: treat $W = 0$ (or $W$ below a minimum-length floor, e.g. 50 words) as an automatic DROP before computing any ratio. Empty/near-empty docs carry no language-modeling signal anyway. WHY guard first: every ratio filter has a hidden precondition (denominator $> 0$ ). Covering the zero case is not optional — a real crawl has millions of empty scrapes.

Recall Solution 5.4

First, fix the unit: here an $n$ -gram means $n$ consecutive words, not characters. A 3-word gram like "in the morning" is a short common phrase; a 13-word gram is a specific 13-word string. A 3-word gram appears in millions of innocent docs — a match flags tons of clean text (huge false-positive rate). A 13-word gram is so specific that the chance two unrelated documents share it by accident is astronomically small. The trade-off: longer $n$ → fewer false alarms but risks missing paraphrased leakage; shorter $n$ → catches more but drowns in coincidences. $13$ is the empirical sweet spot: long enough that a match almost certainly means real overlap. See Benchmark Contamination. WHY it matters: skipping decontamination inflates eval scores fraudulently — the model "knew the answers."

Recall Solution 5.5

$0. 9^{20} \approx 0.1216 \Rightarrow only \approx 12.2% survives .$ Lesson: independent strict filters compound multiplicatively, so "one more safe-looking filter" can quietly destroy your token budget. Beyond a point you starve the model of data (Scaling Laws for LLMs) and narrow its style. Curation is budget allocation, not maximal deletion — the parent note's core message.

Recall Self-test checklist

Ratio filters need a non-zero denominator ::: True — guard empty docs upstream LSH candidate prob for similarity $s$ , bands $b$ , rows $r$ ::: $P (s) = 1 - (1 - s^{r})^{b}$ LSH endpoints ::: $P (0) = 0$ , $P (1) = 1$ — every curve is anchored there Exact 50% crossover $s_{0}$ ::: $(1 - 2^{- 1/ b})^{1/ r}$ Effective epochs on a domain ::: $e_{d} = w_{d} T / N_{d}$ , keep $≲ 4$ Pipeline survival across stages ::: multiply survival fractions, never add losses MinHash estimator variance ::: $J (1 - J) / m$ , shrinks with more hashes