Question bank — Tokenization fundamentals

False. A larger $V$ shortens sequences but scatters training signal across many rare tokens, so their embeddings are undertrained — there is a sweet spot near 30K–100K (see the trade-off figure above), not "bigger is better".

True — every string is made of characters, so nothing is unrepresentable. The cost is paid elsewhere: sequences balloon and the $O(n^2)$ attention in the Transformer becomes expensive.

In practice no for byte-level BPE (GPT-2 style), because every byte is a base token so any string decomposes. Classic character-BPE can emit <unk> — see the step-by-step in the edge-case section below.

Not always. Splitting and numericalization are reversible, but the normalization step (lowercasing, accent removal) can destroy information, so $\text{decode}(\text{encode}(s))$ may not equal $s$.

False. Different tokenizations can concatenate to the same string (e.g. ["low","est"] vs ["l","o","w","est"]), so decoding is many-to-one; only encoding is deterministic per tokenizer.

It should not — padding only fills batches to equal length so all sequences share a shape, and it is masked out, so attention must ignore it. If it does affect output, the mask is broken. (The padding mechanism itself is covered in sequence length & padding, where a figure shows short sentences topped up with <pad> to a common length.)

False, and this is the classic trap. It merges the most frequent pair in the current tokenization, which changes after every merge — you must recount pairs each iteration, exactly as the merge-loop figure shows.

False. A token may be a full word, a subword ("ing"), a single character, or a byte. Word-tokens are just one convenient case that emerges when a word is frequent.

Wrong — BPE greedily applies its learned merge rules in order to the character sequence; there is no nearest-word search. It composes the word from subwords, never matches to a whole word.

Overgeneralized. Lowercasing loses case information (US the country vs us the pronoun, proper nouns), which many modern tokenizers deliberately keep — it is optional, not mandatory.

Misleading. Fertility 1.0 means one token per word, which requires a huge word-level vocabulary and reintroduces the OOV and rare-word problems — the goal is balance (~1.3–1.5 for BPE), not minimum fertility.

No — Ġ encodes the leading space so whitespace position survives round-tripping. Spaces are perfectly legal; Ġ exists for reversibility and to distinguish "word" from " word".

98% coverage means 1 in 50 tokens is <unk>, which badly degrades meaning. The parent note's rule of thumb is $>99.5\%$ — 98% is a red flag, not a pass.

False. BPE merges by raw frequency; WordPiece (see WordPiece & SentencePiece) merges the pair that most increases likelihood, so their merge orders and resulting tokens differ.

They are distinct. Pre-tokenization sets boundaries words cannot cross (splitting on whitespace/punctuation); BPE then operates within each of those pieces. Confusing them causes merges across word boundaries.

So the base vocabulary is tiny and closed — every possible input can be built up from it. Starting from words would reintroduce OOV for anything unseen. See the BPE algorithm.

Because BPE merges high-count pairs first, so common sequences get "absorbed" into one token early, while rare sequences never reach the top of the count list and remain as their subword pieces — exactly the word-vs-subword picture above.

Because a shared suffix like "est" becomes one reusable token, so "newest" and "widest" share structure and the model can generalize — word tokenization treats them as unrelated atomic symbols.

Attention cost grows as $O(n^2)$ in sequence length $n$; a tokenizer with high fertility produces longer sequences, quadratically increasing compute. (The padding note sequence length shows the same quadratic bite when short sentences are padded up to the batch maximum.)

A BPE learned mostly on English gives non-Latin scripts high fertility (many tokens per word), so those languages get longer sequences and worse coverage — the motivation for multilingual tokenization.

So special tokens have stable, reserved slots independent of the learned vocabulary, keeping the ID scheme consistent across models and simplifying masking and lookup (vocabulary & OOV discusses how these reserved slots bound the OOV rate).

Byte-level BPE falls back to the raw UTF-8 bytes (all 256 bytes are base tokens), so it still encodes it — possibly as several byte-tokens. Character-BPE would emit <unk> instead (walked through next).

Its base vocabulary is only the characters seen during training. When encoding new text it first maps each character to a base token — but a never-seen character (say 𝔸) has no base token, so there is nothing to start merging from; the tokenizer substitutes the fallback <unk> for it. Byte-BPE avoids this because all 256 byte values are always present.

Zero content tokens, though many pipelines still wrap it with the <bos> (start) and <eos> (end) markers defined at the top, giving a length of 2. The tokenizer never crashes on empty input — it just returns an empty (or special-token-only) list.

Almost never — a single occurrence is unlikely to make any of its pairs the most frequent, so it survives as separate subwords. This is exactly the desired anti-rare-word behavior.

BPE breaks the tie by a fixed deterministic rule (commonly lexicographic order of the pair), so here ("e","s") — which sorts before ("s","t") — is merged first; then counts are recomputed and ("s","t")/("es","t") is handled on the next round. The outcome is reproducible, but a different implementation's tie rule can pick the other pair and produce a slightly different vocabulary.

Yes if space handling is imperfect. GPT-2 avoids this with Ġ so spaces round-trip exactly; a tokenizer that strips leading/trailing whitespace in normalization will not restore it, so the recovered string differs from $s$.

Very high — it equals the number of characters (plus any byte-splits). A spike in fertility on new text is a symptom that the domain drifted from the training data.