WordPiece and SentencePiece: Question bank — AI-ML deep dive

This page is a question bank for WordPiece and SentencePiece. It is not about crunching numbers — it is about the ideas that are easy to get backwards. Each line below hides its answer. Cover the right-hand side, commit to a guess and a reason, then reveal.

Before you start, here is the vocabulary of symbols we lean on, so nothing appears un-earned:

The two pictures behind every trap

Almost every misconception below is a confusion between two mental models. Build them visually first — then the Q&A is just checking you kept them straight.

Picture 1 — WordPiece is a ladder that only climbs

Look at figure s01. The vertical axis is $L$ , the corpus log-likelihood — "how happy the model is". The horizontal axis counts merges WordPiece has accepted so far.

Read the orange dots left to right. Each dot is the state after accepting one merge. Notice they never dip: the curve is a staircase going only up.
Read a purple arrow. Each arrow is one merge, labelled with its gain (like +0.90). WordPiece computes $count (ab) lo g \frac{P ( ab )}{P ( a ) P ( b )}$ for every candidate pair, then takes the single largest positive one — that arrow. A pair whose arrow would point down (negative PMI) is simply never climbed.
Watch the gains shrink (+0.90, +0.70, +0.50, …). Early merges grab the tightest-bound, highest-PMI pairs; later merges have slimmer pickings. This falling step-size is why WordPiece is greedy-but-safe: it always spends its next merge on the biggest remaining likelihood win.

Hold this in mind for every "True/False" about merging: WordPiece is one arrow chosen per step, always the biggest climb, never a descent.

Picture 2 — Unigram is a spray, not a single dart

Now look at figure s02 — the same word dogs, a completely different spirit. Each horizontal bar is one possible way to cut the word, and its length is the posterior soft weight that segmentation receives.

The bars sum to 1. The E-step does not pick a split; it distributes a total weight of $1$ across all splits, in proportion to each split's probability under current $P (x)$ .
Follow the orange bar [dog, s] (weight $\approx 0.605$ ) and the teal bar [dogs] (weight $\approx 0.378$ ). Both are alive at once. The tiny plum bars ([do,g,s], [d,o,gs], [d,o,g,s]) get slivers of weight — down-weighted, never deleted.
Read the ink arrow: it points at the orange bar and says "greedy WordPiece would pick just this one." That contrast is the whole lesson — WordPiece throws one dart (longest-match), unigram sprays weight, and the M-step later turns those soft weights into new token probabilities.

Every "soft vs hard" trap below is really asking: did you remember the spray, or did you collapse it to a single dart?

True or false — justify

WordPiece merges whichever pair of tokens appears most frequently in the corpus.

False. That is Byte-Pair Encoding (BPE). WordPiece merges the pair with the largest likelihood gain

count (ab) lo g \frac{P ( ab )}{P ( a ) P ( b )}

, which is frequency weighted by PMI — a pair can be common yet score low if

a

and

b

are just independently popular. (In figure s01: it climbs the biggest arrow, not the fattest raw count.)

A pair that occurs 10,000 times will always be merged before a pair that occurs 500 times, in WordPiece.

False. The 500-pair can win if its PMI is far higher (the two pieces almost never appear apart). High raw count with low PMI (e.g. the+##dog) loses to a rarer but tightly-bound pair.

SentencePiece requires the input to be split on spaces before tokenizing.

False. That is the whole point of SentencePiece — it eats raw text and treats space itself as a token ▁, so it works on space-less languages (Chinese, Thai). WordPiece is the one needing pre-tokenized, space-separated input.

The ## prefix is used by both WordPiece and SentencePiece unigram to mark non-initial pieces.

False. Only WordPiece uses ##. SentencePiece unigram uses plain substrings and recovers word boundaries purely from the ▁ (space) marker.

SentencePiece tokenization is exactly reversible back to the original text.

True. Because ▁ encodes every space (including leading/trailing ones), joining the tokens and swapping ▁→space reconstructs the original stream character-for-character. WordPiece is not guaranteed reversible — the ##/space rejoining can lose spacing info.

The unigram model used in both algorithms conditions each token on the previous token.

False. "Unigram" means no context:

P (sentence) = \prod_{i} P (x_{i})

, each token independent. That independence is exactly what makes the merge/pruning math tractable.

WordPiece uses a hard, greedy decision at inference; SentencePiece unigram uses a soft, probabilistic one during training.

True. WordPiece throws one dart (greedy longest-match, one winner per position). Unigram's E-step is the spray of figure s02 — soft weight over all segmentations at once.

Adding a merge in WordPiece can never decrease the corpus likelihood.

True. WordPiece only ever performs the merge with the largest positive likelihood gain (a merge with negative gain is simply never chosen). So every merge it actually carries out strictly increases

L

— the staircase of figure s01 only climbs.

Both algorithms produce the same tokenization for the same corpus and vocabulary size.

False. Different objective (PMI-gain merge vs. likelihood-maximising prune), different boundary handling (## vs ▁), and different inference (greedy vs probabilistic) generally give different token sets and different splits.

Spot the error

"unhappiness → WordPiece → [un, ##happiness], so the second piece always carries ##."

Error: the ## is not something you decide at output — it is baked into the vocab entry. Two cases: (a) if the vocabulary stores the entry as happiness (no prefix), the output is [un, happiness]; (b) if it stores it as ##happiness, the output is [un, ##happiness]. The prefix marks how the token was learned (word-initial vs non-initial), so you must copy the stored form exactly — you cannot add or drop the ## yourself.

"SentencePiece scores segmentation [dog, s] as

P (dog) + P (s)

Error: it is a product, not a sum. Under independence

P (seg) = \prod P (x_{i}) = 0.40 \times 0.20 = 0.08

— that is the length of the orange bar in figure s02. The sum appears only after taking logs (

lo g \prod = \sum lo g

"The likelihood gain formula

count (ab) lo g \frac{P ( ab )}{P ( a ) P ( b )}

is just the raw frequency of the pair."

Error: the log ratio is the essential part. Strip it away and you get BPE. That ratio is PMI; it is what turns "often" into "often and meaningfully bound".

"SentencePiece unigram grows its vocabulary by merging, like BPE."

Error: it prunes. Unigram starts with a large candidate vocabulary and removes the tokens whose deletion hurts

L

least, shrinking down to the target size — the opposite direction from merging.

"Character-level tokenization solves OOV, so subwords are unnecessary."

Error: it solves OOV but at ruinous cost. Characters make sequences very long (slow, hard for the model to see morphology). Subword is the Goldilocks middle — coverage of characters, brevity closer to words. See Out-of-Vocabulary Problem.

Why questions

Why does WordPiece pick likelihood gain instead of raw count?

Because it wants the tokenization that makes the corpus most probable under a unigram model. Merging

a, b

into

ab

changes

L

, and choosing the largest change rewards meaningful units (high PMI), not just common ones — the tallest arrow in figure s01.

Why does high PMI signal a "meaningful unit"?

PMI = lo g \frac{P ( ab )}{P ( a ) P ( b )}

compares "seen together" against "chance if independent". PMI

> 0

means

a

and

b

appear together more than luck predicts — evidence they belong to one morpheme like un+happy. See Morphology and Compositionality.

Why does SentencePiece use the special symbol ▁ instead of just deleting spaces?

Deleting spaces loses information — you could never tell hotdog from hot dog. ▁ encodes every space as data, making detokenization exact and lossless.

Why is the unigram model trained with Expectation-Maximization rather than a direct formula?

Each word has many possible segmentations (all the bars in figure s02), and we don't know which one produced it (a hidden/latent choice). EM handles hidden variables: the E-step guesses soft segmentation weights from current

P (x)

, the M-step re-estimates

P (x)

from those weights, iterating up. See Expectation Maximization.

Why does SentencePiece use forward-backward in the E-step instead of listing every segmentation?

The number of segmentations grows exponentially with word length. Forward-backward computes the expected token counts (those bar lengths) in linear time by summing over shared sub-paths, never enumerating them all.

Why can WordPiece never split across a word boundary, but SentencePiece can span them?

WordPiece requires pre-tokenized, whitespace-split input, so a token lives inside one word by construction. SentencePiece treats the whole stream (spaces as ▁) as one sequence, so a token may include a ▁ and effectively straddle words.

Why is WordPiece a natural fit for BERT Architecture while SentencePiece suits T5 and mT5?

BERT is trained on space-delimited English/multilingual text where pre-tokenization is easy, so ## subwords work. T5/mT5 target many languages including space-less ones, needing SentencePiece's raw-stream, language-agnostic design.

Edge cases

What does WordPiece output when no subword in the vocabulary matches at the current position?

The greedy match fails and it emits the [UNK] token — the fallback that guarantees termination but signals lost information. Good vocabularies keep this rare by including all single characters.

If a vocabulary contains every single character, can WordPiece ever produce [UNK] for known-alphabet text?

No — worst case it falls back to one-character tokens, so any string over the known character set is fully coverable. [UNK] then only appears for truly unseen characters (e.g. an emoji absent from vocab).

In the unigram E-step, what happens to the segmentation [dogs] (the whole word as one token) if P(dogs) is tiny?

It still gets a posterior weight — the teal bar in figure s02 — just a small one; soft assignment never zeroes a valid segmentation, it only down-weights it relative to higher-probability ones like [dog, s].

What is the tokenization of an empty string, and of a string of only spaces, in SentencePiece?

Empty string → no tokens (or just boundary markers, none of content). A run of

k

spaces →

k

copies of ▁ (each space is real data), so detokenization restores every space exactly.

At the extreme, if the target vocabulary size equals the number of distinct characters, what does each algorithm reduce to?

Both collapse toward character-level tokenization — WordPiece never merges beyond characters, unigram prunes back to characters. Sequences get long; you've left the Goldilocks zone. See Tokenization Fundamentals.

At the other extreme, if the vocabulary is large enough to hold every whole word, what happens on a new rare word at inference?

You still fall back to subword/character pieces for the unseen word — a giant vocabulary reduces splitting for common words but cannot pre-store words it never saw, which is exactly the Out-of-Vocabulary Problem subwords exist to soften.

What if two different segmentations of one word have equal probability under the unigram model?

The E-step gives them equal soft weight — two bars of equal length in figure s02 — no arbitrary tie-break needed, since it sums over all of them anyway. (WordPiece's greedy match would, by contrast, deterministically pick the longest-first path.)

Can a WordPiece merge ever have negative likelihood gain, and would it be added?

Yes, negative gain occurs when PMI

< 0

(the pair co-occurs less than chance). It would not be added — WordPiece only picks positive, largest-gain merges, so an anti-correlated arrow (pointing down in figure s01) is never climbed.

Recall One-line self-check before you leave

WordPiece merges by PMI-weighted likelihood gain (a ladder of upward arrows, hard greedy dart at inference, uses ##); SentencePiece unigram prunes a large vocab by likelihood loss (a spray of soft EM weights, plain substrings, ▁ for spaces).