Byte-Pair Encoding (BPE): Visual walkthrough — AI-ML deep dive

This page rebuilds the central result of Byte-Pair Encoding (BPE) — how a handful of "merge rules" turn raw characters into meaningful subword tokens — entirely in pictures. We start from a bag of letters and end holding a finished vocabulary, watching the corpus shrink at every step.

If a word here feels unexplained, we build it. Let's start from zero.

Step 0 — What are we even looking at?

WHAT. Before any merging, a "corpus" is just text. We chop every word into its individual letters (called characters) and glue a special marker </w> to the end. </w> is not a letter you type — it is a made-up symbol meaning "a word ends right here." We need it so BPE can tell est sitting inside a word from est that finishes a word.

WHY. BPE has to start from something that can spell any possible input, even a word it has never seen. The only thing guaranteed to do that is the raw alphabet. So the starting vocabulary — call it $V_{0}$ — is simply "every character that appears, plus </w>."

PICTURE. Each word is a row of boxes. The number on the left is how often that word appears (its count). We keep counts so that a pair inside a word that appears 6 times is worth 6, not 1.

We use the tiny corpus from Byte-Pair Encoding (BPE): 5× low, 2× lower, 6× newest, 3× widest.

Step 1 — Count every adjacent pair

WHAT. Slide a two-box window along each row. Every time two neighbouring tokens sit side by side, that ordered pair scores points equal to the word's count.

WHY. BPE's whole strategy is compression by frequency: if the same two symbols keep touching, storing them as one symbol saves space everywhere they occur. So the very first question is "which neighbour-pair is the most popular?" — nothing else can be decided until we know that.

PICTURE. Below, the pair (e,s) is highlighted everywhere it appears. It lives inside newest (worth 6) and inside widest (worth 3), so its score is $6 + 3 = 9$ .

Top scores: (e,s) $= 9$ and (s,t) $= 9$ tie for first.

Step 2 — Merge the winner, break ties by rule

WHAT. Two pairs tie at 9. We break the tie with a fixed, boring rule — alphabetical order — and merge (e,s) into a brand-new single token es. Then we rewrite the corpus, replacing every adjacent e s with the single box es.

WHY. We need the algorithm to be deterministic: same corpus in, same vocabulary out, every time. A tie-break that depends on luck would break reproducibility. Alphabetical is arbitrary but consistent — that is all we require.

PICTURE. Watch newest and widest: two boxes e s fuse into one box es. The rows get shorter. That shortening is the entire point.

Step 3 — Recount, because the board changed

WHAT. After a merge, some old pairs vanished and new ones appeared. So we count again from scratch on the new corpus.

WHY. This is the subtle heart of BPE. Merging e s created the new neighbour es t that did not exist as a single window before (previously the window only ever saw e s or s t). We cannot reuse old counts — the geometry of the rows changed.

PICTURE. The new pair (es, t) now appears in newest (6) and widest (3) → score $9$ , the new champion.

Merge (es, t) → est, giving $V_{2} = V_{1} \cup {est}$ , size $13$ .

Step 4 — Keep going until the vocabulary is full

WHAT. Repeat "count → merge winner → rewrite" over and over. The next merges are (w,e) → we (score 8), then (l,o) → lo (score 7), then (lo,w) → low, and so on until $∣ V ∣$ hits the target.

WHY. Each round buys us one more frequent subword. Stopping condition: the vocabulary reaches the chosen size $V$ , or no pair is frequent enough to bother. The result is an ordered list — the merge rules — that is the actual output of training.

PICTURE. The merge rules stack up like a numbered recipe. The order they were learned is baked in.

Step 5 — Encoding a new word: apply rules by priority

WHAT. Now training is done. Give the model a fresh word, lowest. Split it into characters, then apply the merge rules in learned order, lowest index first, merging every matching pair before moving to the next rule.

WHY. Encoding must reproduce the same decomposition the model was trained on, so it replays the exact recipe. Using priority order guarantees es forms before est, exactly as during training. Any other order could give inconsistent, unseen tokenizations.

PICTURE. Follow lowest collapsing rule by rule until it settles as [low, est, </w>].

Input:            l o w e s t </w>
Rule 1 (e s→es):  l o w es t </w>
Rule 2 (es t→est):l o w est </w>
Rule 3 (l o→lo):  lo w est </w>
Rule 4 (lo w→low):low est </w>
Output: [low, est, </w>]

Step 6 — The degenerate cases (never leave the reader stranded)

WHAT. Three things can go "wrong". We show all of them.

WHY. The contract: the reader must never meet a scenario we didn't draw. Real inputs contain unseen words, blocked merges, and letters that never got their own merge.

PICTURE.

Case A — a blocker interrupts a merge. Encoding hugs with rules learned from hug/pug/bug: after u g → ug we would love ug </w>, but the letter s sits between ug and </w>. The pair never becomes adjacent, so the merge cannot fire. Result: [h, ug, s, </w>]. No merge across a non-matching neighbour — ever.
Case B — an unseen character. If a word contains a symbol not in $V_{0}$ (say an emoji), byte-level BPE falls back to raw bytes so there is still zero <UNK> — every input is spellable. This is why it is called Byte-Pair Encoding. (See 4.2.04-SentencePiece for how this is handled without spaces.)
Case C — a lone character with no rule. A letter that was never merged simply stays a single-character token. That is fine: it is still a valid vocabulary entry, just an inefficient one.

The one-picture summary

The whole loop in one frame: count pairs → merge the most frequent → rewrite the corpus → repeat, producing an ordered rulebook; then encoding replays that rulebook in priority order on any new word.

Recall Feynman retelling — say it like a story

Imagine a page of words cut into single letters. I look for the two letters that touch most often across the whole page — remembering that a word appearing five times counts five times. I glue that winning pair into one new tile and add it to my box of tiles. The page got a little shorter. I look again — and now new pairings exist that couldn't before, because the tile I just made can touch its neighbours. I keep gluing the most-popular pair, writing down each glue-move as a numbered rule, until my tile-box is as big as I wanted. That numbered list of glue-moves is the trained tokenizer. To read a brand-new word, I cut it into letters and replay my glue-moves in the exact order I learned them — rule 1, then rule 2, and so on — merging wherever a rule matches. If some stray letter sits between two tiles that a rule wants to join, the glue can't reach across it, so that merge just doesn't happen. That is why common words end up as one tile (fast), rare words break into sensible pieces (no "unknowns"), and shared endings like "est" get learned only once.

Recall

What does the </w> marker let BPE distinguish? ::: est ending a word versus est sitting inside a word — end-of-word context. Why must merge rules be applied in the order they were learned? ::: Later rules are built from earlier merges (e.g. est = es+t), so es must exist first; wrong order gives inconsistent tokenizations. How is a pair's score computed? ::: Sum of the word-counts of every word where those two tokens are adjacent (weighted by frequency). Why does adding an s change how hug tokenizes? ::: The s is a blocker between ug and </w>, so the ug </w> merge can never fire (adjacency is broken). Why does BPE produce zero <UNK> tokens? ::: It starts from all characters/bytes, so any input is always spellable from the base vocabulary.

Related: 4.2.01-Tokenization-Overview · 4.2.03-WordPiece-Tokenization (a probabilistic cousin) · 6.2.02-Inference-Optimization (shorter sequences = faster inference).