Foundations — Pretraining data curation and cleaning

This page assumes nothing. We build every letter, slash, and squiggle the parent note (Pretraining Data Curation and Cleaning) throws at you, in an order where each idea leans on the one before it.

0. The picture behind everything: a pile of documents

Imagine a giant bag of scraps of text — web pages, comment threads, menus, articles. Curation is the act of sorting this bag: throw out the junk, throw out the copies, keep the good stuff. Two questions repeat everywhere:

• How junky is this one scrap? → answered with ratios.
• Are these two scraps secretly the same? → answered with sets and probabilities.

Hold that bag in mind; every symbol below is a tool for one of those two questions. Figure 1 draws this bag and the two questions it splits into.

Figure 1 — The corpus as a bag of coloured scraps. The magenta arrow leads to Q1 ("how junky?", answered by ratios); the violet arrow leads to Q2 ("are two the same?", answered by sets and probability).

1. Counts: $W$, $S$, $L$, $D$, $N$

Why does the topic need counts? Because the only cheap thing you can do to a trillion tokens is tally features of them. You never "understand" a page; you count its ingredients and judge by the recipe.

2. Ratios: turning two counts into a fair judgment

Two of the parent's ratios use only counts you already have (§1), so we can read them now:

$r_{\text{sym}}=\frac{S}{W},f_{\text{dup-line}}=\frac{D}{L}.$

• $r_{\text{sym}}$ ("symbols per word"): small for prose ($<0.1$), large for symbol-spam.
• $f_{\text{dup-line}}$ ("fraction of lines that are repeats"): a number between $0$ and $1$.

A third ratio, $e_d$ ("effective epochs"), also has this bad-count-over-total shape, but its pieces ($w_d$, $T$, $N_d$) aren't built yet — we assemble it in §7. The subscript (the little text below, like the "sym" in $r_{\text{sym}}$) is just a name tag so you know which ratio; it changes nothing about the maths. Figure 2 shows why the ratio, not the raw count, is the fair judge.

Figure 2 — A short doc and a long doc with different raw symbol counts land on the same value $r=0.15$ because dividing by word count cancels length. The violet wall at $0.1$ then judges both fairly.

3. Thresholds and the symbols $>$, $<$, $\ge$, $\le$, $=$, $\tau$, $\theta$

Two Greek letters name thresholds throughout the topic:

• $\tau$ (tau) — the language-keep wall (a probability threshold we meet properly in §4), with a typical value $\tau =0.65$.
• $\theta$ (theta) — the near-duplicate wall, e.g. call docs duplicates if $J\ge \theta$ with $\theta =0.8$.

4. Probability: the symbol $p(\cdot )$ and $\mathrm{Pr}[\cdot ]$

A probability is itself a ratio: (favourable outcomes) / (all outcomes). So it fits the §2 mental model perfectly — it's a fraction that always lands in $[0,1]$.

Now the language rule from the pipeline reads cleanly: keep a doc if $p(\text{en}\mid \text{doc})>\tau$. In words — keep it if the classifier's confidence that the doc is English clears the language wall $\tau$ (from §3).

5. Sets, and the symbols $A$, $\cap$, $\cup$, $|A|$

A $k$-shingle is how we fill the circles. First, what is $k$? It is just a whole number you pick, at least $1$ (an integer $\ge 1$) — it sets how long each chunk is. To shingle a document: chop it into every overlapping run of $k$ consecutive words. With $k=2$, the text "the cat sat on" becomes the set $\{\text{"the cat"},\text{"cat sat"},\text{"sat on"}\}$. Small $k$ (like $1$) makes almost everything overlap (too loose); large $k$ (like $10$) demands long identical runs (too strict) — typical near-dedup uses $k$ around $5$ to $10$ words. Two docs sharing many shingles overlap a lot. Figure 3 draws the two circles and names each region.

Figure 3 — Two document sets $A$ and $B$ as overlapping circles. The central lens is $A\cap B$ (shared shingles); the whole blob is $A\cup B$; Jaccard is the lens divided by the blob.

6. Functions and the "min", "hash", "argmax" tools

7. Putting the last ratio together: $e_d$, $w_d$, $N_d$, $T$, and $\sum$

Now that counts and ratios are solid, the mixture formula reads cleanly.

8. Big-O: the symbols $O(N)$ and $O(N^2)$

The prerequisite map — how these foundations feed the topic

The diagram below is a teaching aid, not decoration: read it top-down as a dependency chain. Counts (top) are the raw material; they branch into ratios (the left path, feeding quality filtering) and into sets → Jaccard → probability → minhash (the right path, feeding deduplication). Big-O joins in to explain why dedup must be clever. All three coloured paths converge on the parent's curation pipeline at the bottom — so nothing on the parent page floats free of these roots.

Ready to go deeper into the moving parts? See Tokenization (how words become the shingles and tokens we count), MinHash and LSH (the dedup engine), Deduplication and Memorization, Benchmark Contamination, Data Mixture and Domain Weighting, and Scaling Laws for LLMs (why token budgets are fixed).

Equipment checklist

Read the item as a question, then reveal to check yourself.