Vocabulary size tradeoffs: Question bank — AI-ML deep dive

Before we start, three symbols recur — make sure you own them:

Keep the two costs straight, because half of all traps confuse them:

Embedding table = $V \times d$ — a permanent parameter cost, paid once, sits in memory forever.
Attention = $\propto n^{2} d$ per layer — a per-batch activation cost, paid every forward pass.

True or false — justify

True or false: increasing

V

always increases the total parameter count of the model.

Only the embedding (and tied output) rows grow with

V

; attention and feedforward weights depend on

d

and layer count, not

V

— but yes, more

V

means strictly more embedding rows, so total params do rise, just not everywhere.

True or false: a larger vocabulary always makes each forward pass cheaper.

False — it shortens

n

(cheaper attention activations) but enlarges the embedding table (more parameters to store and load); the net effect depends on how many batches you run, so it can go either way.

True or false: byte-level tokenization (

V \approx 256

) can produce an out-of-vocabulary token.

False — every possible byte is in the vocabulary, so any input decomposes into known tokens; that universality is exactly why byte-level has "no UNK" as its headline advantage.

True or false: doubling the vocabulary size roughly halves the sequence length.

False — the relationship is strongly sublinear; doubling

V

merges only somewhat longer sequences, typically lengthening average token length

\overset{ˉ}{ℓ}

by 20–30%, not 100%.

True or false: the embedding table cost scales with sequence length

n

False — it is

V \times d

and has no

n

in it; sequence length affects activation memory (attention scores), not the parameter count of the embedding table.

True or false: two models with the same

V

but different

d

have the same embedding table size.

False — table size is

V \times d

, so GPT-3's

d = 12288

makes its table ~16× bigger than GPT-2's

d = 768

despite identical

V = 50257

True or false: character-level tokenization gives every language the same sequence length for the same "amount of meaning."

False — information density per character differs wildly; a 100-character Chinese passage carries far more meaning than 100 English characters, so equal character counts are not equal content.

True or false: if a token appears only 3 times in training, giving it its own vocabulary slot guarantees a good embedding for it.

False — a rarely-seen token gets few gradient updates, so its embedding stays noisy; a bigger vocabulary of rare tokens can hurt by wasting slots on poorly-trained vectors.

Spot the error

"Attention memory for a batch is

8 \times 12 \times n^{2} \times d \times 2

bytes." — what's wrong?

The extra

\times d

is the error: attention scores are

n \times n

scalars, not

d

-vectors, so the correct size is

8 \times 12 \times n^{2} \times 2

; multiplying by

d

inflates the answer by ~500× into nonsense (hundreds of GB).

"Switching from

V = 32 K

V = 128 K

made attention 2× faster, so we halved sequence length." — spot the flaw.

Attention cost scales with

n^{2}

, so a 2× speedup comes from only a ~30% reduction in

n

(since

1/0. 7^{2} \approx 2.04

), not a halving; the person confused a quadratic effect with a linear one.

"Larger vocab is pure win because 'king − man + woman = queen' needs 'king' as one token." — where is this overstated?

The analogy benefits from common words staying whole, but that gain saturates; beyond a point, extra vocabulary slots go to rare tokens with noisy embeddings, so it's not pure win — it's diminishing and then negative returns.

"GPT-3 kept

V = 50257

because it couldn't afford any more parameters." — correct the reasoning.

It's not that it couldn't afford them — at 175B params, adding ~1.8B for a 200K vocab is tolerable; the real reason is embedding cost scales with the large

d = 12288

making each new token expensive, and other optimizations offered better returns first.

"We use character-level for Chinese to save memory since

V

is tiny." — what does this miss?

The tiny

V

saves embedding memory, but Chinese subword merges shorten sequences 2.5–3×, and since attention is

n^{2}

, that's ~6–9× less attention cost — so subword usually wins overall despite the bigger table.

"Feedforward and attention both quadruple when we double sequence length." — fix it.

Attention (

\propto n^{2} d

) quadruples, but feedforward (

\propto n d^{2}

) only doubles because it's linear in

n

; only attention has the quadratic term.

"A 100K vocabulary always beats 32K for a medical chatbot because it captures jargon." — what's the hidden condition?

It only wins if the specialized merges actually reduce UNK/splitting and you run enough batches for activation savings to outweigh the ~49M extra permanent parameters; without those conditions the bigger table is dead weight.

Why questions

Why does attention cost, not embedding cost, dominate the argument for shortening sequences on long documents?

Attention grows as

n^{2}

while embedding params are fixed regardless of

n

; for long documents the

n^{2}

term explodes, so even a modest drop in

n

yields a large activation saving — see 4.3.01-Transformer-attention-complexity.

Why is the embedding table called a "permanent" cost while attention memory is "per-batch"?

Embedding parameters are stored weights loaded once and reused for every input; attention scores are activations recomputed and re-allocated for each batch, so their cost recurs and accumulates over training/serving.

Why does average token length

\overset{ˉ}{ℓ}

rise only sublinearly with

V

New vocabulary slots get spent on progressively rarer character sequences, which appear less often, so each added token shaves few characters off the average — merges follow frequency, and frequency has a long thin tail.

Why can a vocabulary be "too large" from a learning standpoint, not just a memory one?

Splitting the same text into many rare tokens means each token's embedding is updated by few examples, producing noisy, under-trained vectors — so generalization on rare words can worsen even as the table grows. This is why 4.5.02-Embedding-layer-design cares about frequency.

Why do byte-pair encoding and WordPiece land near 32K–50K rather than 256 or 250K?

It's the goldilocks zone — big enough that common words are single tokens (short sequences, good semantics) yet small enough that the embedding table and rare-token noise stay manageable; see 4.2.02-Byte-pair-encoding and 4.2.03-WordPiece-and-SentencePiece.

Why does the breakeven for a larger vocabulary depend on how many batches you process?

The parameter overhead

(V_{l} - V_{s}) d

is paid once, but the activation saving

(n_{s}^{2} - n_{l}^{2}) d

recurs per batch; summed over many batches the recurring saving can overtake the one-time overhead, so throughput volume flips the decision.

Why is a shared/tied vocabulary a bigger design lever for multilingual models than for monolingual ones?

Each language competes for slots in one

V

, so a fixed budget spread across many scripts means shorter coverage per language and more splitting — the tradeoff tightens as languages multiply, a central concern in 6.2.01-Multilingual-models.

Edge cases

V = 256

(pure byte-level), what happens to UNK tokens and to sequence length?

UNK tokens vanish entirely (every byte is known), but sequence length hits its maximum — roughly one token per character/byte — making attention cost the worst case.

For a language where every character is already a meaningful word (highly information-dense script), does character-level lose much to subword?

Less than for English — the sequences are already short and the per-token meaning is high, so the subword advantage shrinks, though merging still helps for common multi-character words.

If two vocabularies produce the same sequence length

n

on your corpus, which do you pick?

The smaller

V

— with equal

n

the attention costs match, so you take the smaller embedding table (fewer parameters, less memory), gaining nothing by going larger.

What happens to the breakeven analysis if you only ever run one inference pass (e.g., a one-shot demo)?

The per-batch activation saving is collected just once, so it rarely overcomes the permanent embedding overhead — with a single pass, the smaller vocabulary almost always wins on total footprint.

In the degenerate limit

d \to

very large (like GPT-3 scale), how does the cost of adding vocabulary slots change?

Each new token now costs a huge

d

-length vector, so the marginal price of vocabulary rises sharply — this is precisely why large-

d

models are conservative about growing

V

; also relevant to 5.1.03-Memory-optimization-techniques.

If a domain has extreme jargon (e.g., chemistry formulae) that a standard 32K vocab shreds into many pieces, what boundary case favors a bigger vocab?

When shredding inflates

n

badly and produces poor semantics for key terms, dedicating slots to those terms both shortens sequences and improves representation — the rare case where a domain-specialized larger vocab clearly wins.

Recall Quick self-audit

Which cost is permanent, $V \times d$ or $n^{2} d$ ? ::: $V \times d$ (embedding parameters) is permanent; $n^{2} d$ (attention) is per-batch activation. Does doubling $n$ quadruple attention or feedforward? ::: Attention (quadratic in $n$ ); feedforward only doubles. Can byte-level tokenization emit UNK? ::: No — all 256 bytes are always in-vocabulary.

Read the Hinglish companion: 4.2.04 Vocabulary size tradeoffs (Hinglish).