Vocabulary size tradeoffs: Visual walkthrough

This is the visual companion to Vocabulary size tradeoffs. Prerequisites we will lean on but re-explain: how subwords are formed (4.2.02-Byte-pair-encoding), and why attention costs what it costs (4.3.01-Transformer-attention-complexity).

Step 1 — What a "token" even is, and what "vocabulary size" counts

WHAT. Text is a long ribbon of characters. A tokenizer is a rule that chops that ribbon into pieces called tokens. The vocabulary is the fixed list of every distinct piece the tokenizer is allowed to output. Its length is $V$ — the vocabulary size.

WHY start here. Every later symbol ( $n$ , the sequence length; $V$ , the table height) is counted off this picture. If we don't nail down what one token is, nothing downstream means anything.

PICTURE. Below, the same 30-character sentence is cut two ways. A small vocabulary knows few pieces, so it must cut into many small tokens (long ribbon of boxes). A large vocabulary knows whole words, so it cuts into few big tokens (short ribbon).

Step 2 — The two piles of cost: a table and a ribbon

WHAT. Every token in the vocabulary needs a private list of $d$ numbers — its embedding vector — so the model can tell tokens apart. Stack all $V$ of these vectors and you get the embedding table: a grid $V$ rows tall, $d$ columns wide.

$Embedding params = one row per token V \times numbers per row d$

WHY these two symbols multiply. Each row is independent (every token gets its own vector), and each row has exactly $d$ slots, so total slots $= V \times d$ . Linear in $V$ : add a token, add a row, add $d$ numbers.

WHY it matters that this is a table. The table is permanent memory — it sits in the model whether you process one sentence or a million. Contrast that with the ribbon of Step 1, whose cost only appears while you're processing text. That distinction is the pivot of the whole derivation.

PICTURE. Left: the tall table (grows downward as $V$ grows). Right: the ribbon of $n$ boxes (grows sideways as $V$ shrinks). One is stored forever; one is paid per use.

See 4.5.02-Embedding-layer-design for how this table is actually laid out in memory.

Step 3 — Why the ribbon's cost is $n^{2}$ , not $n$

WHAT. Inside a transformer, every token looks at every other token to decide what matters. If there are $n$ tokens, that is $n \times n$ pairwise "attention scores". We store those scores in a square grid: the attention score matrix, size $n \times n$ .

WHY it's a square. "Every token looks at every token" is literally a grid: row = the token doing the looking, column = the token being looked at. $n$ rows $\times$ $n$ columns $= n^{2}$ cells.

$score matrix size = rows (lookers) n \times cols (looked-at) n = n^{2}$

WHY this is the killer. Halve the ribbon (make $n$ smaller) and the square shrinks by the square of the change: $n \to 0.7 n$ gives $0. 7^{2} = 0.49$ — nearly half the grid. This nonlinearity is exactly why shortening sequences is so powerful.

PICTURE. A short ribbon ( $n = 4$ ) makes a tiny $4 \times 4$ grid; a long ribbon ( $n = 8$ ) makes a $8 \times 8$ grid — four times as many cells for only twice the tokens.

Step 4 — Turning "score grid" into actual megabytes

WHAT. To compare piles we need one currency: bytes. The score grid appears once per layer, once per item in the batch. So multiply the grid by how many copies exist and by bytes-per-number.

score memory = batch B \times layers L \times cells/grid n^{2} \times bytes/cell b

WHY each factor. $B$ items processed together each need their own grid; $L$ layers each redo attention; $n^{2}$ cells per grid (Step 3); $b$ bytes per cell ( $b = 2$ for float16). All are multiplicative because they are independent copies.

WHY this is the "ribbon" cost made concrete. This is the number that shrinks when a bigger vocabulary shortens the ribbon. It is temporary (per batch) but paid over and over.

PICTURE. A stack of $B \times L$ identical grids, each $n \times n$ , each cell tagged "2 bytes" — the physical pile of activation memory.

Step 5 — How a bigger vocabulary shortens the ribbon (sublinearly)

WHAT. Bigger $V$ means the tokenizer can afford to remember longer character-chunks, so each token swallows more characters. Call the average characters-per-token $\overset{ˉ}{ℓ}$ . Then for a text of $C$ characters:

n \approx \frac{C}{ℓ ˉ}, \overset{ˉ}{ℓ} grows as V grows, but slowly

WHY it's slow (sublinear). 4.2.02-Byte-pair-encoding merges the most frequent pairs first. The first merges (like t+h→th) pay off hugely; later merges (rare medical words) fire rarely. So doubling $V$ might lengthen $\overset{ˉ}{ℓ}$ by only 20–30%, not double it. Diminishing returns.

PICTURE. A curve: horizontal axis $V$ , vertical axis $\overset{ˉ}{ℓ}$ . It rises fast then flattens — the classic diminishing-returns bend. The dashed drop shows how a modest $\overset{ˉ}{ℓ}$ gain (4 → 5 chars) gives a shorter ribbon.

Step 6 — Laying the two costs on one balance scale

WHAT. Compare a small vocab ( $V_{s}$ , giving ribbon $n_{s}$ ) against a large one ( $V_{l}$ , giving shorter ribbon $n_{l}$ ). We put both costs into the same currency: bytes, so they can actually be weighed against each other.

Extra permanent cost of going large — the new table rows. Each new row is $d$ numbers, and each number takes $b$ bytes: $(V_{l} - V_{s}) d b bytes, paid ONCE$
Per-batch memory saved — the score grid shrinks from $n_{s}^{2}$ cells to $n_{l}^{2}$ cells. Each cell is one scalar (Step 3), so no factor of $d$ appears here; just $b$ bytes per cell, over $B$ batch items and $L$ layers: $(n_{s}^{2} - n_{l}^{2}) B L b bytes saved, EVERY batch$

WHY no $d$ on the savings side. Step 3's [!mistake] warned that score-grid cells are scalars, not $d$ -vectors. So the table side carries $d$ (each row genuinely has $d$ numbers) but the attention-savings side does not — the earlier draft's $d$ there was the very error we flagged. Both sides now carry the same bytes-per-number factor $b$ , and neither can be cancelled away against the other's mismatched units.

WHY introduce a batch symbol. The table is stored once no matter what; the score grid is rebuilt on every batch. Let $N$ = the number of batches you will process over the model's life. The savings accumulate $N$ times, so the right side is multiplied by $N$ .

WHAT the inequality looks like. Large vocabulary is worth it when accumulated savings beat the one-time table cost:

extra table, once, in bytes (V_{l} - V_{s}) d b < attention saved over N batches, in bytes (n_{s}^{2} - n_{l}^{2}) B L b \cdot N

PICTURE. A balance scale: left pan = one heavy fixed weight (extra table); right pan = many small weights (per-batch savings) piling up as $N$ grows. As $N$ climbs, the right pan sinks.

Step 7 — The degenerate cases (where the rule quietly breaks)

WHAT. A rule is only trustworthy if we test its edges. Three break-points:

(a) Sequences don't shrink: $n_{l} = n_{s}$ . Then $n_{s}^{2} - n_{l}^{2} = 0$ — the right pan is empty. No savings ever, so the extra table is pure loss. This is exactly GPT-3: it kept $V = 50, 257$ because at $d = 12288$ new rows cost billions of parameters while its already-optimal tokenizer barely shortened sequences.

(b) Tiny vocab, huge ribbon: $V_{s} \to 256$ (byte-level). Left pan almost weightless (few rows) but $n_{s}$ explodes, so $n_{s}^{2}$ is enormous — the large vocab wins easily. This is why byte-level is chosen only when universality (no unknown tokens, all languages — see 6.2.01-Multilingual-models) matters more than speed.

(c) One batch, forever: $N = 1$ . The savings lose their multiplier; the fixed table almost always dominates. Small vocab wins. So the breakeven is a throughput argument, not a single-inference one.

PICTURE. Three mini-scales, one per case, showing the pan that dominates.

The one-picture summary

Everything above collapses into a single tug-of-war: a fixed table weight versus an accumulating pile of per-batch savings, where the pile grows by $(n_{s}^{2} - n_{l}^{2})$ each batch and $n$ itself bends sublinearly with $V$ .

Recall Feynman retelling — say it like a story

Imagine two ways to write the same book. In the small-alphabet way you only know letters, so the book is thousands of tiny symbols long. In the big-dictionary way you know whole words, so the book is short — but you must carry a giant dictionary everywhere. The dictionary is heavy and you carry it always (that's $V \times d$ numbers). The book's reading effort is worse than its length suggests, because to understand each word you glance at every other word — that's the $n \times n$ grid, and it grows like the square of the length. A shorter book saves a lot of glancing. Now: is the big dictionary worth it? Only if you'll read many books. One reading, and the heavy dictionary isn't worth it. A thousand readings, and the saved glancing adds up past the dictionary's weight. And there's a catch — past a point, adding more words to the dictionary barely shortens the book (the common words were added long ago), so you stop getting shorter and just get heavier. The sweet spot is where the shrinking book still pays for the growing dictionary. To weigh the two fairly you turn everything into the same unit — bytes — so the dictionary's weight and the glancing you saved can sit on the same scale.

Recall Quick self-test

Why is attention cost $n^{2}$ and not $n$ ? ::: Every token attends to every token, forming an $n \times n$ pairwise grid; cells $= n^{2}$ . Which cost is permanent and which is per-batch? ::: Embedding table $V \times d$ is permanent; attention score grid $n^{2}$ is per-batch. When does a larger vocabulary give zero benefit? ::: When it doesn't shorten sequences ( $n_{l} = n_{s}$ ), so $n_{s}^{2} - n_{l}^{2} = 0$ . Why does the batch count $N$ multiply only the savings side? ::: The table is stored once; the score grid is rebuilt every batch, so its savings accumulate $N$ times. Why does the savings side carry $b$ but not $d$ ? ::: Score-grid cells are single scalars (so no $d$ ), but they are still real numbers taking $b$ bytes each. Why is $\overset{ˉ}{ℓ}$ vs $V$ sublinear? ::: BPE merges frequent pairs first, so later vocabulary additions fire rarely and shorten sequences less.