Byte-Pair Encoding (BPE): Question bank — AI-ML deep dive

Quick vocabulary reminder so every question below is readable from line one:

A merge rule is a learned instruction "replace this adjacent pair with one token", stored in the order it was learned.
A token is one symbol in the vocabulary — it can be a single character, a subword like est, or a whole word like bug</w>.
</w> is the end-of-word marker: a special symbol glued to the last character of every word so the tokenizer can tell "word-final" pieces from "word-internal" ones.

True or false — justify

BPE at training time picks the pair that gives the best final compression.

False — it is greedy: each round it picks the most frequent pair right now, with no lookahead. Locally-best merges need not give the globally-optimal vocabulary.

BPE can produce an <UNK> (unknown) token for a word it never saw in training.

False — the base vocabulary contains every single character, so any string can always be broken down to characters at worst. There is no such thing as an out-of-vocabulary word.

Two identical corpora with the same target vocab size always yield the same merge rules.

True (given a fixed tie-break rule) — BPE is deterministic: same counts, same greedy choices, same alphabetical tie-breaks → identical vocabulary. This is why models ship a frozen merges.txt.

The order of merge rules doesn't matter as long as you apply all of them.

False — later rules are built on top of earlier merges. es t → est is meaningless if e s → es hasn't run first; applying out of order changes the tokenization.

A larger vocabulary always makes sequences dramatically shorter.

False — sequence length shrinks with diminishing returns. The first merges (very frequent pairs) shorten a lot; going 10k → 50k barely raises the average token length

k

, so the extra shortening is tiny.

BPE and WordPiece learn the vocabulary using the exact same selection criterion.

False — BPE picks the most frequent pair; 4.2.03-WordPiece-Tokenization picks the pair that most increases corpus likelihood (a probability score), not raw count.

Adding </w> is just cosmetic and changes nothing important.

False — </w> lets ing</w> (word-ending) be a different token from ing in the middle of a word, so BPE can learn genuine morphology like suffixes.

Spot the error

"I'll tokenize by matching the longest subword in the vocab first, ignoring merge order."

Wrong for classic BPE — standard BPE applies merges by rule index (earliest learned first), not by longest match. Longest-match-first is the SentencePiece/WordPiece style; mixing them gives different tokens.

"Pairs (e,s) and (s,t) both had count 9, so I merged both this round."

Error — exactly one merge happens per round. Ties are broken (e.g. alphabetically) and the loser is re-counted next round, because merging es changes the counts around st.

"The vocab size equals the number of merge rules."

Error — vocab size = base characters plus number of merges. With 11 base characters and 4 merges you have 15 tokens, not 4.

"After training, encoding must re-count pair frequencies in the new text."

Error — encoding uses only the fixed learned rules; no counting happens at inference. Frequency counting is a training-only step.

"I merged (u,g)→ug, so now the standalone u and g tokens can be deleted from the vocab."

Error — old tokens stay in the vocab. A future word like gum still needs the bare g; BPE only adds, never removes.

"BPE guarantees each token is a real morpheme like a prefix or suffix."

Error — merges follow frequency, not linguistics. Frequent pieces often line up with morphemes, but tokens like th or low are statistical, not grammatical, units.

Why questions

Why start from characters instead of from whole words?

Because characters are the smallest guaranteed-representable units, so a bottom-up build can reconstruct any input word, eliminating unknowns while still discovering frequent chunks.

Why is the pair (e,s) chosen over the rarer (e,r) even though both help lower?

BPE optimizes compression — replacing the most frequent pair removes the most symbols corpus-wide, so high count wins regardless of which words it helps.

Why must es exist before est can be learned?

est is defined as the merge es + t; the token es is a building block. Without it, the pair (es, t) never appears, so the rule can't fire — the dependency is baked into the ordering.

Why does BPE feed shorter sequences into the model, and why does that help?

Fewer tokens per sentence means the model's self-attention (which scales with sequence length) does less work and fewer positions to relate — see 6.2.02-Inference-Optimization. Each token also carries more meaning than a lone character.

Why can BPE handle a brand-new made-up word like "cryptobros"?

It falls back to whatever known subwords cover it (crypto, bros, or down to characters), so novel words decompose gracefully instead of hitting <UNK>.

Why do the merged tokens get their own vectors in the 3.1.05-Embedding-Layer?

Each vocabulary entry — character, subword, or word — is one row of the embedding table, so the model learns a dedicated meaning-vector for every BPE token it can emit.

Edge cases

What does BPE do with a single unseen character (e.g. an emoji) never in the base vocab?

True char-level BPE would fail, which is why byte-level BPE encodes text as raw bytes first — then all 256 byte values are the base, and nothing is ever unrepresentable.

If a target word's pair never becomes adjacent (like (ug,</w>) in "hugs"), what happens?

The rule simply doesn't fire — an intervening character (s) blocks adjacency, so "hugs" tokenizes as [h, ug, s, </w>] even though ug</w> exists as a token.

What is the tokenization of a word made of one repeated character, say "aaaa"?

It merges greedily by rule order: if a a → aa was learned, aaaa → aa aa; otherwise it stays as four a characters. There's no infinite loop — merges only apply where a rule matches.

If the target vocab size is smaller than the number of unique base characters, what happens?

Training can't shrink below the base vocabulary — you get zero merges and the vocab is just the characters. The target is effectively a floor, not a hard ceiling below base size.

What happens at the very first round if all pair counts are equal (a perfectly uniform corpus)?

The tie-break rule decides (e.g. alphabetical order), keeping the process deterministic; BPE never stalls on ties.

Does removing the most frequent word from the corpus change all the merge rules?

Not necessarily all — it changes the counts, which can flip the early greedy choices, and because later merges depend on earlier ones, the effect can cascade. Small corpus changes can reshape the whole vocabulary.

Recall Self-check before you leave

Greedy or optimal? ::: Greedy, most-frequent-pair each round. Can BPE emit <UNK>? ::: No (byte-level guarantees full coverage). Vocab size formula in words? ::: Base characters plus number of merges. Why apply merges in learned order at inference? ::: Later rules depend on tokens created by earlier rules.