Foundations — Vocabulary size tradeoffs

Before you can read Vocabulary size tradeoffs, you need to own every symbol it throws at you. This page builds each one from absolute zero: plain words → the picture → why the topic needs it. We go in order, so each symbol leans only on the ones before it.

1. Text, characters, and tokens

Look at the top of the first figure: the same sentence is cut three different ways. The scissors show where each tokenizer decides to make a break.

2. Vocabulary and its size $V$

Picture a numbered list: entry 0, entry 1, ..., entry $V-1$. Every token that comes out of the tokenizer must be one of these rows — nothing off-menu is allowed.

The topic needs $V$ because it is the first of the two dial settings we trade off. Turning $V$ up makes the menu longer (costs memory) but the sequences shorter (saves compute). See 4.2.02-Byte-pair-encoding and 4.2.03-WordPiece-and-SentencePiece for how the menu is actually built.

3. Sequence length $n$

The second figure shows the same fixed text becoming different $n$ as $V$ changes. Notice: bigger $V$ → smaller $n$ for the same text, because each fatter token swallows more characters.

The topic needs $n$ because it is the second dial — and it appears squared in the compute cost, which is why it matters so much.

4. Average token length $\bar{\ell }$

This links $n$ and $V$ together:

The picture: imagine laying tiles of width $\bar{\ell }$ over a strip of fixed length (the text). Wider tiles → fewer tiles → smaller $n$. Bigger $V$ empirically makes $\bar{\ell }$ grow (more common groups get their own menu entry), but only slowly — doubling $V$ nudges $\bar{\ell }$ up ~20–30%, not double.

5. Embedding dimension $d$

Picture each token as an arrow floating in a space with $d$ directions. $d$ is how many independent "meaning-knobs" each token gets.

6. The embedding table: $V\times d$

Now the first cost falls out for free.

Third figure: the grid literally, with the row-count $V$ and column-count $d$ labelled.

7. Big-O notation and the $n^2$ cost

The fourth figure draws the $n\times n$ grid of "who-looks-at-whom" and shows it exploding as $n$ grows.

8. Batch and layer counts: $L$ and batch size $b$

Why the topic needs them: the memory saved by a shorter sequence gets multiplied by $b\times L$ and by every batch you ever run. Small per-batch savings compound into large totals over a training run — that is what tips the breakeven toward large $V$. See 5.1.03-Memory-optimization-techniques.

Prerequisite map

The two arrows into the topic node T are the whole story: the memory branch ($V\times d$) and the compute branch ($n^{2}d$) meet, and choosing $V$ is choosing where on that seesaw to sit. This carries straight into 6.2.01-Multilingual-models, where many languages force $V$ upward.

Equipment checklist

Cover the right side and answer each before opening the parent note.