This page assumes you have seen the parent note's symbols flash by — P(x), log, ΔL, PMI, ##, ▁, and the training loop called Expectation–Maximization — and felt lost. Here we build every one of them from the ground up, in an order where each brick rests on the one before it. Nothing is used before it is drawn.
Picture it. Think of a box of alphabet-stamps. Each stamp is a token; the whole box is the vocabulary. To print any sentence you must build it out of stamps you actually own — you can't invent a stamp mid-sentence. Figure s01 draws this box beside the two extremes we reject.
Figure s01 — the subword box (centre) has a small, fixed size yet can still spell unbreakable; the word-level box on the upper right is infinite, the char-level box on the lower right is tiny but forces very long print jobs.
Why the topic needs it. Word-level stamps mean an infinite box (every rare word needs its own stamp). Character-level stamps mean a tiny box but very long print jobs. Subword stamps are the compromise — that compromise is the topic.
The symbol ∣V∣ (two vertical bars around V) just means "how many things are in the set V" — the count of stamps in the box. Bars-around-a-thing always means "size of".
Picture it. Take the word dogs. You could cut it as dog | s, or do | g | s, or leave it whole as dogs. Each set of scissor-cuts is a different segmentation. The word is a strip of paper; a segmentation is a choice of where to snip. Figure s02 lines up four different cuts of the very same word.
Figure s02 — four valid segmentations of dogs, from all-characters (top) to the whole word (bottom). Every row rebuilds the identical strip of paper; the algorithms differ only in which row they prefer.
Why the topic needs it. Every algorithm here is secretly answering "of all the ways to cut this word, which cut do we prefer?". WordPiece answers greedily (one hard cut); SentencePiece-Unigram answers softly (a probability over all cuts, using the training loop of §9). You cannot understand either without first seeing that many cuts exist.
Picture it. Imagine a spinner (a pie chart). Each token gets a slice; the slice's area is P(x). Frequent tokens get fat slices, rare tokens get thin slivers. Spin once — the slice you land on is the token you "drew".
Why the topic needs it. Both algorithms score a cut by multiplying the probabilities of its pieces. So we need a per-piece probability first. The little x is just a placeholder name for "some token" — like using a blank to write "P(blank)".
Here n is how many pieces this cut has, and xi is the i-th piece. The little i under ∏ is a counter that walks 1,2,…,n.
Picture it. Spin the spinner once per piece. The chance of getting exactly this sequence of pieces is each slice-area multiplied together (independent events multiply). More pieces → more multiplications by small numbers → smaller product. This is why long, over-chopped cuts score badly.
Why the topic needs it. This single product is the scoring rule inside SentencePiece-Unigram's E-step (parent's Worked Example 3: dog×s = 0.40×0.20 = 0.08). "Independent" is doing real work here — it is exactly what lets us multiply, and exactly what "no context" means in the parent note.
Picture it. Plot log: it climbs steeply near 0 and flattens for large inputs. A probability of 0.0001 becomes log10(0.0001)=−4 — a comfortable negative number instead of a vanishing decimal. Because probabilities are ≤1, their logs are always ≤0; a bigger (closer to zero) log-likelihood means a more probable corpus. Figure s03 plots this squashing.
So the product from §4 becomes the sum the parent writes:
L=∑i=1NlogP(xi)
The script letter L is just a name — "the log-likelihood", a single number scoring how well our vocabulary + probabilities explain the whole corpus. N is the total number of token-slots in the corpus.
Figure s03 — the log curve. Three marked points show how probabilities as small as 0.0001 (pink dot) become the tame value −4; every log sits at or below the horizontal line because P(x)≤1.
Why the topic needs it. "Maximize the likelihood" is the shared goal of both algorithms. Every merge or prune is judged by how much it moves L. Logs make that arithmetic possible and additive.
Picture it.L is your height on a hill. Take one step (merge a pair, prune a token). ΔL is how much higher or lower you now stand. WordPiece always takes the step that climbs most.
Why the topic needs it. WordPiece's merge rule is literally "pick the pair with the biggest ΔL". A positive ΔL = a good merge; negative = a bad one. This is also how Unigram decides which tokens to prune (remove the ones whose loss hurts L least).
First two pieces of counting notation we are about to use:
Merging pair (a,b) into one token ab replaces logP(a)+logP(b) with logP(ab) everywhere that pair occurs — that is, count(ab) times. Subtracting, and using loga−logb=logba:
ΔL(a,b)=count(ab)⋅logP(a)P(b)P(ab)
The fraction inside the log is exactly Pointwise Mutual Information (PMI).
Picture it.P(a)P(b) is the "if they were strangers" expectation; P(ab) is what really happens. PMI is the surprise that they show up as a couple.
Why the topic needs it. This is the exact reason the parent stresses WordPiece is not frequency-based. A pair like the+cat can be frequent yet low-PMI (both words are common on their own, so meeting is no surprise). Raw-frequency Byte-Pair Encoding (BPE) would merge them; PMI-weighted WordPiece won't.
Picture it.## is a left-facing hook linking a fragment back onto its neighbour. ▁ is a visible ghost of a space that rides along inside a token (▁w = "space then w").
Why the topic needs it. These markers are how each system recovers word boundaries. WordPiece needs whitespace pre-split, so it marks interior pieces with ##. SentencePiece refuses to assume spaces exist (Chinese, Thai, Japanese don't use them), so it encodes every space as ▁ and stays perfectly reversible — vital for T5 and mT5. This directly addresses the Out-of-Vocabulary Problem and mirrors how humans read morphemes.
Two tributaries feed the topic: the likelihood chain (token → probability → product → log → L → ΔL → PMI) and the bookkeeping markers. PMI (§7) is what WordPiece maximizes; EM soft counts and pruning (§9) are what SentencePiece-Unigram runs. Both need Tokenization Fundamentals and both attack the Out-of-Vocabulary Problem.
