Pretraining data curation and cleaning: Worked examples

The scenario matrix

Before working examples, let's map out every cell a curation problem can land in. Each row is a "situation class"; every example below is tagged with the cell it fills.

Cell	Stage	Situation class	What makes it tricky
A	Quality	Clean prose, well under threshold	Baseline "keep"
B	Quality	Symbol spam, over threshold	Baseline "drop"
C	Quality	Degenerate: zero words ( $W = 0$ )	Division by zero!
D	Quality	Repetition penalty, boilerplate	Second heuristic
E	Dedup	MinHash estimator, mid-similarity	Probability = similarity
F	Dedup	Limiting case: identical ( $J = 1$ ), disjoint ( $J = 0$ ), boundary ( $J = θ$ ), empty shingle set	Every extreme
G	LSH	Tuning $b, r$ so the wall sits at $θ$	S-curve crossover
H	LSH	Exam twist: wrong rule-of-thumb vs exact crossover	Catch the trap
I	Mixing	Upsample a scarce domain safely	$e_{d} \leq 4$
J	Mixing	Real-world word problem: full token budget across 3 domains	Weights must sum to 1

Every symbol used below is defined in the parent note; we re-anchor each as it appears.

Definition The exact boundary convention (read this first)

Every threshold rule needs an unambiguous $\leq$ vs $<$ . We fix it once, for the whole page:

Symbol filter: drop iff $r_{sym} > 0.1$ . So exactly $r_{sym} = 0.1$ is kept (the wall belongs to the keep side).
Repetition filter: drop iff $f_{dup-line} > 0.3$ . So exactly $f_{dup-line} = 0.3$ is kept.
Deduplication: flag as near-duplicate iff $J \geq θ$ . So exactly $J = θ$ is a duplicate (the wall belongs to the drop side).
Mixing guardrail: memorization-safe iff $e_{d} \leq 4$ . So exactly $e_{d} = 4$ is safe.

Why the asymmetry? For quality filters we prefer to keep borderline text (throwing away good tokens is costly), so the wall sits on the keep side. For deduplication we prefer to remove borderline pairs (leaked duplicates are costly), so the wall sits on the drop side.

Quality-filter examples (cells A–D)

Recall the symbol-to-word ratio. A document has $W$ words (space-separated chunks of letters) and $S$ symbol characters (things like #, …, @). We defined $r_{sym} = \frac{S}{W},$ which measures "how many weird characters per word." Prose keeps this tiny; junk inflates it. Rule from the parent (with our convention): drop iff $r_{sym} > 0.1$ ; keep if $r_{sym} \leq 0.1$ .

Figure — Pretraining data curation and cleaning

The symbol-ratio number line: one wall at $0.10$ splits keep from drop. The teal keep-zone sits at or left of the wall; the orange drop-zone is strictly right. Doc A ( $0.04$ , teal) keeps; Doc B ( $0.15$ , orange) drops. Doc C is the empty page: its ratio $S /0$ is off the chart to the right — we draw a plum arrow running past the right edge with the label " $\to \infty$ " to show it is not a real point but a runaway limit, always landing deep in the drop-zone.

Worked example Cell B — symbol spam, drop (with a boundary sub-case)

Doc: $W = 300$ words, $S = 45$ symbol characters (the parent's Example 1).

Forecast: keep or drop?

Compute. $r_{sym} = 45/300 = 0.15$ . Why this step? Same normalization — density, not raw count, is what generalizes.
Compare. $0.15 > 0.1 \Rightarrow$ drop. Why? Fifteen symbols per hundred words is SEO/menu spam territory (»»» BUY NOW ### »»»), not sentences.
Boundary sub-case. Suppose instead $W = 300, S = 30$ , so $r_{sym} = 30/300 = 0.10$ exactly. By our convention ( $>$ , not $\geq$ ) this doc is kept. Why? We deliberately put the wall on the keep side for quality filters — a doc sitting exactly at the threshold is given the benefit of the doubt.

Verify: $0.15 = 1.5 \times 0.1 > 0.1 \Rightarrow$ drop (decisive). The boundary value $0.10 \neq > 0.10 \Rightarrow$ keep, consistent with the convention. ✓

Worked example Cell C — DEGENERATE, zero words

Doc: $W = 0$ words, $S = 12$ symbol characters (e.g. a page that is nothing but ···###···).

Forecast: what does $r_{sym} = 12/0$ even mean?

Spot the division by zero. $r_{sym} = S / W$ is undefined when $W = 0$ . Why this step? Every ratio filter must handle empty input before dividing, or the pipeline crashes on real web garbage (blank pages, image-only pages).
Decide by the spirit of the rule. A doc with no words is not language — we drop it (treat $W = 0$ as "infinite symbol density", the worst possible case). Why? Language identification / quality filtering both want text; zero words carries zero learning signal, so it can only waste token budget.

Verify: As $W \to 0^{+}$ with $S$ fixed, $r_{sym} = S / W \to + \infty > 0.1$ . The limit agrees with the "drop" decision — no special case needed if we define the empty doc as the limiting drop. ✓

Worked example Cell D — repetition penalty (boilerplate)

Recall the duplicate-line fraction: with $L$ total lines and $D$ of them being copies of an earlier line, $f_{dup-line} = D / L$ ; drop iff $> 0.3$ ; keep if $\leq 0.3$ . Doc: $L = 20$ lines, of which $8$ are repeats of "Add to cart".

Forecast: keep or drop?

Compute the fraction. $f_{dup-line} = 8/20 = 0.4$ . Why this step? Repeated identical lines signal a template (shopping cart, comment spam), not original prose.
Compare. $0.4 > 0.3 \Rightarrow$ drop. Why? Even if symbols look fine, templated repetition wastes model capacity on memorizing boilerplate.
Boundary sub-case. If instead $D = 6$ of $L = 20$ , then $f_{dup-line} = 6/20 = 0.30$ exactly — by convention ( $>$ , not $\geq$ ) this is kept. Why? Same benefit-of-the-doubt as the symbol filter: quality walls sit on the keep side.

Verify: $0.4 > 0.3 \Rightarrow$ drop; boundary $0.30 \neq > 0.30 \Rightarrow$ keep. Consistent with the convention. ✓

Deduplication examples (cells E–F)

We represent a document as its set of $k$ -shingles $A$ (overlapping $k$ -word windows). Two docs' overlap is the Jaccard similarity $J (A, B) = \frac{∣ A \cap B ∣}{∣ A \cup B ∣},$ "shared shingles over total distinct shingles." Now the MinHash machinery. A hash function $h$ is a rule that maps each shingle to a random-looking number; $minhash_{h} (A) = min_{x \in A} h (x)$ is the smallest such number over set $A$ . The subscript $h$ just names which random hash we used — we will use $m$ different independent ones, $h_{1}, \dots, h_{m}$ . MinHash's magic fact: $Pr [minhash_{h} (A) = minhash_{h} (B)] = J (A, B),$ so counting matches across $m$ random hashes estimates $J$ without comparing full documents. Our dedup threshold is $θ = 0.8$ : flag as near-duplicate iff $J \geq 0.8$ .

Two shingle sets drawn as overlapping circles: the teal crescent is $A$ -only, the orange crescent is $B$ -only, and the plum lens in the middle is $A \cap B$ . The Jaccard similarity $J$ is the plum lens measured as a fraction of the whole two-circle union. The picture makes the "magic fact" visible: throw one random hash, look at whichever shingle wins the minimum over the union — it is equally likely to be any shingle, so the chance it lands inside the plum lens (making the two minhashes agree) is exactly that lens-fraction $J$ .

Worked example Cell E — MinHash estimator, mid-similarity (with variance)

Two docs with true $J = 0.8$ , using $m = 200$ hashes.

Forecast: how many of the 200 minhashes should match, and how tight is the estimate?

Expected matches = $m \times J$ . $200 \times 0.8 = 160$ . Why this step? Each of the $m$ independent hashes matches with probability $J$ ; the count of matches is a Binomial $(m, J)$ , whose mean is $m J$ .
Turn matches back into an estimate. $\hat{J} = 160/200 = 0.8$ . Why? This is the unbiased estimator: $E [\hat{J}] = J$ .
How confident? Compute the variance. For a Binomial fraction, $Var (\hat{J}) = \frac{J ( 1 - J )}{m} = \frac{0.8 \times 0.2}{200} = 0.0008$ , so the standard error is $0.0008 \approx 0.0283$ . Why? This tells us $\hat{J}$ typically lands within $\pm 0.028$ of $0.8$ ; a rough $95%$ interval is $0.8 \pm 2 (0.028) \approx [0.744, 0.856]$ . More hashes ( $m ↑$ ) shrink the error like $1/ m$ .

Verify: $160/200 = 0.8 = J$ (unbiased), and $0.8 \cdot 0.2/200 = 0.0008$ so SE $\approx 0.0283$ — small enough that $200$ hashes reliably separates "duplicate" from "not". ✓

Worked example Cell F — LIMITING cases: identical, disjoint, boundary, and empty shingles

Case F1: docs are byte-identical, $A = B$ . Case F2: docs share no shingles, $A \cap B = \emptyset$ . Case F3: docs land exactly on the wall, $J = θ = 0.8$ . Case F4: a doc is too short to form any $k$ -shingle, so $A = \emptyset$ .

Forecast: what is $J$ , and drop-or-keep, at each of the four cases?

Identical (F1). $A \cap B = A$ , $A \cup B = A$ , so $J = ∣ A ∣/∣ A ∣ = 1$ . Since $1 \geq 0.8$ ⇒ flagged as duplicate (drop one). Why this step? This is the ceiling — every random minimum agrees, so all $m$ minhashes match (probability $1$ ). Exact duplicates must go.
Disjoint (F2). $A \cap B = \emptyset$ , so $J = 0/∣ A \cup B ∣ = 0$ . Since $0 < 0.8$ ⇒ keep both. Why? This is the floor — the shared minimum can never fall in an empty intersection, so zero minhashes match (probability $0$ ). Nothing shared, nothing to dedup.
Boundary (F3). Suppose a concrete pair with $∣ A \cap B ∣ = 8$ , $∣ A \cup B ∣ = 10$ , so $J = 8/10 = 0.8 = θ$ exactly. By our dedup convention ( $\geq$ , not $>$ ) this pair is a near-duplicate ⇒ drop one. Why? Unlike quality filters, the dedup wall sits on the drop side — a pair sitting exactly at $θ$ is treated as a duplicate, because letting borderline leaks through is the costlier error.
Empty shingles (F4). If a doc has fewer than $k$ words it produces no $k$ -shingles: $A = \emptyset$ . Then $∣ A \cup B ∣$ could be $0$ (if both empty) and $J = 0/0$ is undefined. Why? Just like the $W = 0$ case in the quality filter, the pipeline must catch empty shingle sets before computing $J$ . Convention: a doc with $A = \emptyset$ has no comparable content, so we treat its $J$ with anything as $0$ (never a duplicate) and, if it is also too short to be useful prose, drop it at the earlier quality stage instead.

Verify: $J = 1$ ⇒ all $m$ match ⇒ drop; $J = 0$ ⇒ none match ⇒ keep; $J = 0.8 = θ$ ⇒ $\geq θ$ ⇒ drop; $A = \emptyset$ ⇒ $J$ undefined, patched to $0$ ⇒ keep. Variance vanishes at the pure extremes: $Var = J (1 - J) / m = 0$ when $J \in {0, 1}$ . Every corner of the identity $Pr = J$ is covered. ✓

LSH tuning examples (cells G–H)

To dodge the $O (N^{2})$ all-pairs comparison, split $m = b r$ hashes into $b$ bands of $r$ rows. Docs matching a whole band become candidate pairs. Let's build the S-curve from scratch, one probability at a time, before quoting it.

Formula Deriving the S-curve

$P (s)$ step by step Fix two docs with true similarity $s$ . Recall the magic fact: a single minhash matches with probability $s$ . Now:

One row matches with probability $s$ . Why? A "row" is one minhash slot; matching it is exactly one minhash agreeing, which happens with probability $s$ .
A whole band (all $r$ rows) matches with probability $s^{r}$ . Why? A band is a candidate only if all $r$ of its rows agree. The rows use independent hashes, so we multiply: $s \times s \times \dots = s^{r}$ .
One band FAILS to match with probability $1 - s^{r}$ . Why? The complement of "all $r$ agree."
ALL $b$ bands fail with probability $(1 - s^{r})^{b}$ . Why? Bands are independent, so we multiply the fail-probabilities across all $b$ of them.
At least one band matches (⇒ the pair becomes a candidate) with probability $P (s) = 1 - (1 - s^{r})^{b} .$ Why the " $1 -$ "? "Candidate" means at least one band matched; the easiest route to "at least one" is $1$ minus "none matched." That is the whole trick — AND within a band gives $s^{r}$ ; OR across bands gives $1 - (\dots)^{b}$ .

The LSH S-curve $P (s) = 1 - (1 - s^{r})^{b}$ for $b = 20, r = 5$ , drawn in orange. It starts almost flat near zero, then rises steeply through a "wall" and saturates near one — that flatness-then-cliff shape is exactly what makes LSH cheap. The dotted ink line marks $P = 0.5$ ; the plum dashed vertical marks the exact wall $s_{0} \approx 0.509$ from the derivation. The teal dash-dot line is our target $θ = 0.8$ , where the curve is already at $P \approx 0.9997$ (true duplicates are almost surely caught). The ink dot at $s = 0.3$ shows $P \approx 0.047$ (dissimilar pairs almost never become candidates, saving compute).

Worked example Cell H — EXAM TWIST, rule-of-thumb vs exact crossover

Same $b = 20, r = 5$ . A classmate estimates the crossover with the popular rule-of-thumb $s_{0} \approx (1/ b)^{1/ r}$ . Is that where the wall really sits?

Forecast: will the rough estimate match the exact formula?

Rule-of-thumb. $(1/ b)^{1/ r} = (1/20)^{1/5} = 0.0 5^{0.2} \approx 0.549$ . Why this step? This is the approximation many texts quote for the steep point.
Exact crossover. $s_{0} = (1 - 2^{- 1/20})^{1/5} = (1 - 0.96594)^{1/5} = 0.0340 6^{0.2} \approx 0.5089$ . Why? This is the true 50% point, from the derivation above (invert $P$ at $\frac{1}{2}$ ). Trust this one.
Compare. $0.549$ vs $0.509$ — the rule-of-thumb is off by $\sim 0.04$ , always overestimating the wall. Why it matters? If you tuned by the rough number you'd set the wall higher than intended and could miss borderline duplicates just below your target.

Verify: Both sit well below $θ = 0.8$ , so our $b, r$ still safely catches true dups; but the two formulas disagree by $\approx 0.04$ , confirming you must use the exact $(1 - 2^{- 1/ b})^{1/ r}$ . ✓

Domain-mixing examples (cells I–J)

Given domains $d$ with token counts $N_{d}$ and target weights $w_{d}$ (with $\sum_{d} w_{d} = 1$ ), over a training budget of $T$ tokens each domain is seen for $e_{d} = \frac{w _{d} T}{N _{d}} ("effective epochs") .$ Keep $e_{d}$ modest: by our convention, memorization-safe iff $e_{d} \leq 4$ . See Data Mixture and Domain Weighting for the reweighting theory.

Worked example Cell J — REAL-WORLD word problem, full budget

You train on $T = 1$ T tokens across three domains: web ( $N = 800$ B, $w = 0.60$ ), books ( $N = 50$ B, $w = 0.25$ ), code ( $N = 100$ B, $w = 0.15$ ).

Forecast: which domain risks memorization first?

Sanity-check the weights sum to 1. $0.60 + 0.25 + 0.15 = 1.00$ . ✓ Why this step? If weights don't sum to 1 you're implicitly re-normalizing and $T$ no longer means what you think.
Web epochs. $e_{web} = 0.60 \times 1000/800 = 0.75$ . Why? Web is huge; even at 60% weight we don't finish one pass — no memorization risk.
Books epochs. $e_{books} = 0.25 \times 1000/50 = 5.0$ . Why? Books are scarce; 25% weight forces 5 passes — above the guardrail of 4! Books are the memorization risk.
Code epochs. $e_{code} = 0.15 \times 1000/100 = 1.5$ (matches Cell I).

Verify: Total tokens drawn $= 600 B + 250 B + 150 B = 1000 B = T$ . ✓ And $e_{books} = 5 > 4$ flags the fix: lower $w_{books}$ (e.g. to $0.20 \Rightarrow e = 4.0$ , which is $\leq 4$ and safe). ✓

Recall

Recall Empty document (

$W = 0$ ) in the symbol filter — keep or drop, and why? Drop. $S /0$ is undefined; as $W \to 0^{+}$ the ratio $\to + \infty > 0.1$ , so the limiting decision is "drop" — a text-free page carries no learning signal.

Recall A doc has

$r_{sym} = 0.10$ exactly. Keep or drop under our convention? Keep. The symbol rule drops iff $r_{sym} > 0.1$ , so the wall belongs to the keep side (benefit of the doubt for quality filters).

Recall A pair has

$J = θ = 0.8$ exactly. Duplicate or not, and why is this the opposite convention to the quality filters? Duplicate — dedup flags iff $J \geq θ$ , so the wall sits on the drop side. Letting borderline leaks through is the costlier error, so we err toward removing.

Recall A doc is too short to form any

$k$ -shingle. What is $J$ with another doc, and what do we do? $A = \emptyset$ makes $J$ undefined ( $0/0$ if both empty). We patch it to $J = 0$ (never a duplicate) and drop the too-short doc at the earlier quality stage.

Recall Why is a whole-band match probability

$s^{r}$ and the candidate probability $1 - (1 - s^{r})^{b}$ ? A band is a candidate only if all $r$ rows match (AND ⇒ multiply ⇒ $s^{r}$ ). The pair is a candidate if at least one of $b$ bands matches (OR ⇒ $1$ minus "all fail" ⇒ $1 - (1 - s^{r})^{b}$ ).

Recall For

$b = 20, r = 5$ , is the exact LSH crossover $s_{0}$ closer to $0.51$ or $0.55$ ? About $0.51$ — the exact $(1 - 2^{- 1/ b})^{1/ r} \approx 0.509$ . The rule-of-thumb $(1/ b)^{1/ r} \approx 0.549$ overestimates.

Recall With

$m = 200$ hashes and $J = 0.8$ , what is $Var (\hat{J})$ ? $J (1 - J) / m = 0.8 \times 0.2/200 = 0.0008$ , so standard error $\approx 0.028$ .

Recall In Cell J, which domain crosses the

$e_{d} \leq 4$ guardrail, and at what value? Books, at $e_{books} = 5.0$ epochs — a scarce pool ( $50$ B) paired with a high weight ( $0.25$ ).

Parent: 4.3.4 curation & cleaning. Related: MinHash and LSH, Deduplication and Memorization, Data Mixture and Domain Weighting, Benchmark Contamination, Scaling Laws for LLMs.