Tokenization fundamentals: Visual walkthrough

Our single running example, the same tiny corpus the parent used:

low   low   low   lower   lower   newest  newest  widest

We will watch this pile of letters slowly clump into meaningful chunks.

Step 1 — Start with the smallest possible pieces: characters

WHAT. Take every word and cut it into individual characters. Nothing is joined yet. We also write a special end-of-word mark _ after the last letter of each word so the algorithm knows where a word stops (otherwise it might glue the end of one word to the start of the next).

WHY. We need a guaranteed starting vocabulary. If our first tokens are single characters, then any word — even one we have never seen — can be spelled out. This is the promise that kills the out-of-vocabulary problem from the parent note: worst case, you fall back to letters. A vocabulary here just means "the set of token strings the model is allowed to use".

PICTURE. Look at the figure: each word is a row of separated letter-tiles. The teal tiles on the right are the _ end marks. The box below lists every distinct tile — that is $V_{0}$ , the starting vocabulary.

Step 2 — Define a "pair" and learn to count it

WHAT. A pair is two tokens that sit next to each other inside one word. In l o w _, the pairs are (l,o), (o,w), (w,_). We never pair across the _ boundary, so words stay separate.

WHY. BPE's whole idea is: the two pieces that keep showing up side by side probably belong together. To find "keeps showing up", we must be able to count how often each pair occurs across the entire corpus. Counting is the tool here because frequency is our only signal about what matters — a common pair is common because the language uses it a lot.

PICTURE. The figure sweeps a window of width 2 across low, lower, and newest, highlighting each pair in orange as it slides. The tally on the right sums identical pairs over all eight words (remember low appears 3 times, lower 2 times).

Here is the full tally for our corpus, so no tie is hidden. Recall the word frequencies: low×3, lower×2, newest×2, widest×1.

pair	where it comes from	count
$(l, o)$	low×3 + lower×2	5
$(o, w)$	low×3 + lower×2	5
$(s, t)$	newest×2 + widest×1	3
$(t,_)$	newest×2 + widest×1	3
$(e, s)$	newest×2 + widest×1	3
$(w,_)$	low×3	3
$(w, e)$	lower×2 + newest×2	4

Why is $(l, o) = 5$ ? It appears once in each low (×3) and once in each lower (×2): $3 + 2 = 5$ . Notice several pairs — $(s, t)$ , $(t,_)$ , $(e, s)$ , $(w,_)$ — all sit at $3$ . That cluster of ties is exactly why we need a precise tiebreak rule, which we pin down in Step 3.

Step 3 — Pick the winner: the $ar g max$

WHAT. Choose the pair with the highest count. Here $(l, o)$ and $(o, w)$ tie at $5$ . When two pairs tie we break the tie deterministically: scan the pairs in the fixed order they are first produced by reading the corpus left-to-right, word by word, in the order the words are listed — and among tied pairs keep the one encountered earliest in that scan. Reading low left-to-right, (l,o) is seen before (o,w), so $(l, o)$ wins.

WHY. We merge the most frequent pair first because merging it removes the most redundancy — it shortens the corpus the most in one move, and it captures the strongest "these two belong together" signal. The symbol for "the input that makes something biggest" is $ar g max$ . The tiebreak must be spelled out so that the same corpus always yields the same tokenizer — training has to be reproducible.

PICTURE. A bar chart of pair counts. The tallest bars (orange) are the tie; a plum crown sits on the chosen $(l, o)$ .

Step 4 — Merge: glue the winner into one new token

WHAT. Wherever l o appears next to each other, replace the two tiles with a single tile lo. Add lo to the vocabulary. Also record the merge rule l + o → lo in an ordered list — this list is the tokenizer.

WHY. The merge is the payoff of the count. By fusing the frequent pair, every future step sees lo as an atom, so we can now discover pairs of pairs (like lo + w). The ordered rule list matters because at encode time we must replay the merges in the same order to reproduce the training result.

PICTURE. Before/after strip: the two separate tiles l,o on top collapse into one wider lo tile below, with an arrow. The growing vocabulary box gains a plum lo.

After this merge:

lo w _ | lo w _ | lo w _ | lo w e r _ | lo w e r _ | n e w e s t _ | ... | w i d e s t _

Step 5 — Loop: the pieces snowball into words

WHAT. Go back to Step 2 and repeat: recount pairs, pick the champion, merge. Now lo w wins (5 times) → low. After that, the pairs inside newest/widest come into play: e s and s t are tied at $3$ ; by our left-to-right tiebreak e s is seen first in newest, so e s → es merges first. Then the new pair es t occurs $3$ times → est.

WHY. Each pass builds one level higher. Small clumps become bigger clumps: l+o→lo, then lo+w→low. This is compositionality emerging for free — we never told it "low is a word"; frequency discovered it. Notice est was learned as a suffix, so newest and widest now share it. That is exactly the morphology win the parent promised.

PICTURE. A vertical ladder: each rung is one merge round, showing the corpus getting shorter and the tiles getting wider as we climb. The est tile lights up in two different words to show sharing.

After these merges the corpus reads:

low _ | low _ | low _ | low e r _ | low e r _ | n e w est _ | n e w est _ | w i d est _

Step 6 — When do we stop? The termination knob

WHAT. Keep merging until the vocabulary reaches a target size we choose in advance, written $V_{target}$ (say 30K–50K tokens), or until no pair repeats.

WHY. More merges = longer tokens = shorter sequences, but a bigger table of embeddings to learn. Fewer merges = short tokens = long sequences. Long sequences are expensive because Transformer attention has cost that grows like $O (n^{2})$ — read that as: if the sequence has $n$ tokens, the work grows roughly in proportion to $n$ times $n$ (double the tokens, roughly quadruple the work). That is the "big-O" way of saying "how the cost scales with the input size $n$ ". See 3.4.03-Sequence-length-and-padding and 5.1.02-Transformer-architecture. The target size $V_{target}$ is the dial that trades these off, tying into 4.1.05-Vocabulary-and-OV.

PICTURE. Two curves against number of merges: sequence length falling (teal), vocabulary size rising (orange). A dashed plum line marks the chosen stopping point where they balance.

Step 7 — The degenerate cases (never leave a gap)

WHAT & WHY & PICTURE, three edge cases the loop must survive:

A brand-new word at encode time, e.g. slowest. No merge for sl exists, so those letters stay as single-character tokens — but nothing breaks, because Step 1 guaranteed every character is in $V_{0}$ . No <unk> needed.
A tie in Step 3. Two pairs share the top count. We resolve it with the exact rule from Step 3: among tied pairs, keep the one first encountered when reading the corpus left-to-right, word by word, in listed order. Same corpus always yields the same tokenizer.
A single-character word or an emoji, e.g. a or 🌍. It has no internal pair to merge, so it simply stays one token forever. (Byte-level BPE spells unseen bytes out — see 4.2.08-Multilingual-tokenization.)

The figure shows all three side by side: the fallback spelling of slowest, the tiebreak arrow, and the untouched lone emoji tile.

The one-picture summary

The whole algorithm is a single loop: count pairs → crown the top → merge it → repeat until full. Characters at the bottom, words at the top, frequency as the elevator.

Recall Feynman retelling — say it to a 12-year-old

Imagine a bag of loose letters spelling out lots of words. You play a game: find the two letters that most often sit next to each other, and glue them with tape into one bigger tile. If two pairs are tied, you pick the one you bumped into first while reading left-to-right. Do it again — now bigger tiles can get taped to letters, or to each other. Keep taping the most common neighbours until your box of tiles is as big as you decided it should be (that target size). Common words like "low" end up as one tile; common endings like "est" become a tile you can reuse in "newest" and "widest". And because you started from single letters, you can always spell any weird new word by taping tiles back together — you never get stuck. That taping-order list you wrote down is the tokenizer: replay it and any text turns into tiles, replay it backwards and the tiles turn back into text.