Foundations — Tokenization fundamentals

Before you can read a single formula in the parent note, you need to know what each squiggle means and what picture lives behind it. We build them in order, each one using only the ones before it. Nothing is assumed.

1. Text as a stream of characters

The very first mental picture: a sentence is not a neat list of words to a computer. It is a ribbon of characters laid end to end, with no gaps except the space characters that are themselves just characters on the ribbon.

Look at the ribbon in the figure: H e l l o , w o r l d !. The space (drawn as ␣) is a real tile on the ribbon, not empty air. Why the topic needs this: every later idea — normalization, splitting, counting pairs — is an operation on this ribbon. If you don't see text as a physical sequence of tiles, phrases like "consecutive pair" or "sequence length" have nothing to point at.

2. The set — writing "the collection of allowed things"

The parent note writes the starting BPE vocabulary as $V_0=\{\text{all unique characters in corpus}\}.$

Read this slowly, symbol by symbol:

• $V$ is just a name we chose for the bag (V for vocabulary).
• The subscript $0$ means "the version before any merges" — a snapshot label, like "frame 0 of a movie."
• The braces $\{\dots \}$ say "here is the bag's contents."

The picture: a jar of unique tiles. Throwing a second l into the jar changes nothing, because the jar only ever keeps one of each.

3. Vocabulary and its size $|V|$

The vertical bars are the same "how many?" idea you already use for the length of a list. Whenever you see $|\cdot |$ in this chapter, translate it in your head to the plain English word "count of."

Why the topic needs it: the whole BPE loop stops when $|V|$ reaches a target like 50 000. Without a symbol for "size of the bag," we couldn't even state the stopping rule.

4. Token — the tile we actually work with

Picture a token as a single domino tile. The ribbon of characters gets glued into a shorter ribbon of these domino tiles. A word like newest might become the two tiles new and est.

The figure shows the same word cut three different ways: as 6 character-tiles, as 2 subword-tiles, and as 1 word-tile. This is the central trade-off of the entire topic — more tiles means longer sequences but a smaller jar; fewer tiles means a shorter sequence but a bigger, sparser jar. Everything the parent note says about "sequence length explosion" and "vocabulary explosion" is a comment on this one picture.

5. Counting and the "most frequent pair"

Compare two cousins:

• ${\max }_{x}f(x)$ → the biggest value of $f$.
• $\arg {\max }_{x}f(x)$ → the $x$ that produced that biggest value.

The parent's rule ${\text{pair}}^{\ast }=\arg {\max }_{\text{pair}}\text{count}(\text{pair})$ reads in plain English: "Look at every adjacent pair of tiles, count how often each occurs, and hand me back the pair that occurred most." The little star ${}^{\ast }$ is just a decoration meaning "the chosen/winning one."

The picture: slide a two-tile window across the whole corpus, tally marks under each pair you see, then circle the pair with the most tally marks.

6. Union — growing the jar with $\cup$

The merge rule $V_{t+1}=V_t\cup \{{\text{pair}}^{\ast }\}$ now reads cleanly: "The next vocabulary is the current jar plus the one new merged tile." The subscripts $t$ and $t+1$ are just step numbers — "this iteration" and "the next iteration," like page numbers of the algorithm's diary.

Why the topic needs it: BPE literally is a sequence of jars $V_0,V_1,V_2,\dots$ each one tile larger than the last. The $\cup$ is the act of adding that tile.

7. Functions, arrows, and "maps to"

Think of a function as a vending machine: put in a string, out comes a list of numbers. The notation $\text{vocab}[t]$ (square brackets) is a dictionary lookup — "find token $t$ in the vocab and read off its ID," exactly like looking up a word in a phone book to get its number.

The subscript numbering in $[t_1,t_2,\dots ,t_n]$ just says "the 1st tile, 2nd tile, …, up to the $n$-th (last) tile." The letter $n$ stands for however many tiles there happen to be — a placeholder for a count, not a fixed number.

8. Averages and the $\sum$ (sigma) symbol

So the average sequence length $\bar{n}=\frac{1}{|C|}{\sum }_{s\in C}|\text{encode}(s)|$ translates piece by piece:

• ${\sum }_{s\in C}$ → "for every sentence $s$ in the corpus $C$, add up…"
• $|\text{encode}(s)|$ → "…the number of tokens that sentence became" (bars = count, from §3).
• $\frac{1}{|C|}$ → "divide by how many sentences there are."
• The bar over $\bar{n}$ → the common notation for an average.

Result: "the average number of tiles per sentence." Nothing mysterious — it is the ordinary classroom average, dressed in symbols so it fits on one line.

9. Putting the symbols back into the pipeline

Now every arrow in the parent's pipeline is readable: $\text{Raw Text}\stackrel{\text{Normalization}}{\to }\text{Clean Text}\stackrel{\text{Tokenization}}{\to }\text{Tokens}\stackrel{\text{Numericalization}}{\to }\text{Token IDs}$ Each arrow is a function (§7). The words above the arrows name which vending machine is running. The endpoints move from ribbon (§1) → cleaner ribbon → domino tiles (§4) → integers (the phone-book numbers).

Once these atoms are in place you are ready for the algorithms that stack on top — 4.2.02-BPE-algorithm, 4.2.03-WordPiece-and-SentencePiece, and the fixed-size batching in 3.4.03-Sequence-length-and-padding. The integers you produce here are exactly what 4.3.01-Word-embedings turns into vectors, and what 5.1.02-Transformer-architecture consumes. The size of the jar and its unknown-token fallback are studied in 4.1.05-Vocabulary-and-OV, and the cross-language version in 4.2.08-Multilingual-tokenization.

Equipment checklist

Test yourself — you should be able to answer each without peeking.