Vocabulary size tradeoffs

Why vocabulary size matters

When we tokenize text, we're making a compression-granularity tradeoff. A small vocabulary (say, 256 tokens for byte-level) treats every character separately. A large vocabulary (100K+ tokens) can represent entire common words or phrases as single tokens.

WHY this matters:

Sequence length directly impacts transformer compute: $O(n^2 d)$ for self-attention where $n$ is sequence length
Embedding table size is $V \times d$ parameters where $V$ is vocabulary size
Token granularity affects how well the model can generalize to rare/unseen words

The mathematics of the tradeoff

Let's derive the resource costs from first principles.

Embedding table cost

For vocabulary size $V$ and embedding dimension $d$ :

$\text{Embedding params} = V \times d$

WHY: Each of the $V$ tokens needs a $d$ -dimensional dense vector representation.

WHAT this means: If $V = 5000$ and $d = 768$ (BERT-base), that's $5000 \times 768 = 3.84M$ parameters just for the embedding table. (If instead $V = 50000$ , that's $50000 \times 768 = 38.4M$ .) This grows linearly with $V$ .

Sequence processing cost

For sequence length $n$ , embedding dimension $d$ , and $L$ transformer layers:

Self-attention cost per layer: $\text{Attention cost} = O(n^2 d)$

WHY: The attention mechanism computes $Q K^T$ which is $(n \times d) \times (d \times n) = n \times n$ . Then attention-weighted values: $(n \times n) \times (n \times d) = n \times d$ . The $n^2$ term dominates.

Total forward pass cost: $\text{Total cost} \approx L \cdot n^2 d + L \cdot n d^2$

The $n^2 d$ term (attention) and $n d^2$ term (feedforward) both scale with sequence length.

WHAT this means: Doubling sequence length $n$ quadruples attention cost but only doubles feedforward cost. For long documents, attention dominates.

Formula Expected sequence length vs. vocabulary size

Given a text corpus with average word length $\bar{w}$ characters and vocabulary size $V$ : For character-level tokenization ( $V \approx 256$ ): $n_{\text{char}} \approx \text{char count in text}$

For subword tokenization (BPE/WordPiece) with average token length $\bar{\ell}$ characters: $n_{\text{subword}} \approx \frac{\text{char count}}{\bar{\ell}}$ where $\bar{\ell}$ increases with $V$ (empirically $\bar{\ell} \approx 4$ for $V=32K$ , $\bar{\ell} \approx 5$ for $V=100K$ ).

Derivation: Larger vocabularies merge more frequent sequences, so each token represents more characters on average. The relationship is sublinear: doubling $V$ might increase $\bar{\ell}$ by 20-30%, not 2x.

The breakeven analysis

When does a larger vocabulary save resources overall?

Memory cost comparison:

Small vocab ( $V_s$ ): Embedding cost $V_s d$ , sequence length $n_s$ , attention cost $\propto n_s^2 d$
Large vocab ( $V_l$ ): Embedding cost $V_l d$ , sequence length $n_l$ , attention cost $\propto n_l^2 d$

Large vocab wins if: $\underbrace{(V_l - V_s) d}_{\text{extra embedding params}} < \underbrace{(n_s^2 - n_l^2) d \cdot \text{batch count}}_{\text{attention memory savings}}$

WHY this formula: The left side is the permanent parameter overhead of the larger vocabulary. The right side is the per-batch activation memory savings from shorter sequences. If you process many batches (training, high-throughput inference), the right side accumulates and can outweigh the left.

Worked example GPT-2 vs. GPT-3 vocabulary choices

GPT-2: $V = 50257$ , $d = 768$ (small model)

Embedding table: $50257 \times 768 = 38.6M$ parameters
Uses byte-level BPE, average token ≈ 4 characters
For a 1000-character text: $\approx 250$ tokens

GPT-3: $V = 50257$ (same!), but the largest model has $d = 12288$ (175B model)

Embedding table: $50257 \times 12288 \approx 617.6M$ parameters
Same tokenization, same sequence lengths WHY GPT-3 kept the same vocab: The embedding table cost scales with $d$ . At GPT-3's scale, a larger vocab (say $V=200K$ ) would add $(200K - 50K) \times 12288 \approx 1.84B$ parameters. The model is already 175B, so this is tolerable, but OpenAI optimized other things first.

Alternative scenario — Chinese text:

Character-level ( $V \approx 1000$ for common Chinese chars): Each character is one token
Subword BPE ( $V = 50K$ ): Common words become single tokens (e.g., "学习" → one token)
For a 100-character Chinese text:
Character-level: 100 tokens
Subword: ~30-40 tokens (Chinese characters are more information-dense)
Tradeoff: Character-level has ~50x smaller embedding table ( $V \approx 1K$ vs $V = 50K$ ) but 2.5-3x longer sequences → attention cost $\approx 6-9x$ higher. Subword wins for Chinese.

Worked example Worked calculation: Choosing vocab size for a medical chatbot

Scenario: Building a medical Q&A system. Input documents average 2000 tokens (at $V=32K$ ). Embedding dim $d=512$ . Batch size 8, 12 transformer layers.

Option 1: $V = 32000$ (standard BPE)

Embedding params: $32000 \times 512 = 16.4M$
Sequence length: 2000 tokens
Attention score memory per batch: $8 \times 12 \times 2000 \times 2000 \times 2 \text{ bytes} \approx 768 \text{ MB}$
WHY this calculation: The attention score matrix is $n \times n$ scalars per head-group per layer (we simplify to per-layer here). Batch size 8, 12 layers, $n \times n = 2000^2$ scalars, each stored in float16 (2 bytes): $8 \times 12 \times 2000^2 \times 2 \approx 768\text{ MB}$ .
Common mis-step: Multiplying by $d=512$ as well (as if each score entry were a $d$ -vector) gives $\approx 393\text{ GB}$ — wildly wrong. The scores are scalars, not vectors. Only the $QKV$ projections and value-weighted outputs carry the $d$ dimension.

Option 2: $V = 128000$ (larger vocab, specialized medical terms)

Embedding params: $128000 \times 512 = 65.5M$
Sequence length: $2000 \times 0.7 = 1400$ tokens (estimate: 30% reduction from domain-specific merges)
Attention score memory per batch: $8 \times 12 \times 1400 \times 1400 \times 2 \text{ bytes} \approx 376 \text{ MB}$

Analysis:

Parameter cost: +49M parameters (Option 2)
Memory savings per batch: $768 - 376 = 392\text{ MB}$ (Option 2)
Throughput: $\left(\frac{2000}{1400}\right)^2 \approx 2.04x$ faster attention (Option 2)

Decision: Option 2 wins IF

You train/serve many batches (memory savings × batches > parameter cost)
The specialized vocab actually improves task performance (fewer "unk" tokens for medical jargon)
You have 49M parameters to spare in your model budget

WHY this step? The 30% sequence reduction from specialized vocabulary (medical terms like "hypertension" or "electrocardiogram" as single tokens) cuts attention cost by $(0.7)^{-2} \approx 2x$ . Over thousands of batches, the memory savings compound.

Common vocabulary size choices

Vocabulary Size	Typical Use Case	Pros	Cons
256–512	Byte-level, multilingual	Universal, no UNK tokens	Very long sequences
8K-16K	Early neural MT, smaller models	Balanced, manageable embedding size	Some rare words split
32K-50K	GPT-2, BERT, RoBERTa	Standard tradeoff, proven	Struggles with rare technical terms
100K-250K	Newer LMs, code models	Short sequences, domain coverage	Large embedding tables

Common mistake "Larger vocabulary is always better for performance"

Why this feels right: More tokens = more specific representations = better understanding, right?

Steel-man: This intuition comes from word2vec era: "king - man + woman = queen" works because "king" is a single token. If "king" were split into "ki" + "ng", that semantic relationship is harder to learn.

The fix: Larger vocab helps up to a point, then:

Rare tokens have poor embeddings: A token that appears 3 times in training gets a noisy embedding. You've wasted parameters.
Overfitting: The model memorizes specific rare tokens instead of learning compositional patterns.
Inference cost: A 500K vocab embedding table is 2.5GB+ at $d=1024$ . That's memory you could use for bigger batch sizes or longer context.

Better approach: Choose $V$ such that the bottom 10% of tokens (by frequency) still appear at least 50-100 times in training data. Use subword tokenization to handle the long tail.

Common mistake "Character-level models eliminate all vocab tradeoffs"

Why this feels right: With 256 byte-level tokens, you have no OOV (out-of-vocabulary) problem, support any language, and tiny embedding tables.

Steel-man: This is appealing because it's simple. No tokenization training, no weird BPE merge rules, just map bytes to tokens.

The fix: Character-level models pay the cost elsewhere:

Sequence length explosion: "Tokenization" (14 chars) → 14 tokens instead of 1. A 512-token limit becomes ~2000 characters (~400 words). GPT-3 would need ~8x longer sequences for the same text.
Harder to learn long-range dependencies: Transformer attention is $O(n^2)$ . When $n$ is 8x larger, you need 64x more attention compute to see the same context.
Slower inference: Autoregressive generation is $O(n)$ steps. 8x more tokens = 8x slower generation.

Empirical result: ByT5 (byte-level T5) performs comparably to subword T5 but requires longer training and larger models to compensate for sequence length. The "no vocab tradeoff" claim just moves the cost to compute.

Practical guidelines

Choosing vocabulary size for your use case:

General-domain text (web, books): 32K-50K subword vocab (BPE/WordPiece)
- WHY: Balances common word compression with rare word splitting
Code models: 50K-100K vocab
- WHY: Code has many unique identifiers, API names. Larger vocab reduces splitting of function names like get_user_profile → get, _user, _profile
Multilingual (100+ languages): 250K+ vocab OR byte-level
- WHY: Each language needs its own frequent tokens. Byte-level avoids the explosion but pays sequence-length cost.
Domain-specific (medical, legal): Train domain-specific vocab at 50K-100K
- WHY: "Myocardial infarction" should be 1-2 tokens, not 6. Improves both efficiency and performance.
Extremely long context (100K+ tokens): Smaller vocab (16K-32K) or hierarchical tokenization
- WHY: With $n^2$ attention, every token counts. Better to have shorter sequences and split rare words.

Connections

4.2.02-Byte-pair-encoding — BPE training produces different vocab sizes depending on number of merges
4.2.03-WordPiece-and-SentencePiece — SentencePiece allows direct control of vocab size hyperparameter
4.3.01-Transformer-attention-complexity — The $O(n^2)$ cost that vocabulary size indirectly impacts
4.5.02-Embedding-layer-design — How embedding tables are implemented and optimized
5.1.03-Memory-optimization-techniques — Flash Attention, gradient checkpointing reduce $n^2$ cost
6.2.01-Multilingual-models — Why massive vocab sizes are needed for 100+ languages

Flashcards

#flashcards/ai-ml

What is the vocabulary size tradeoff in tokenization?

Smaller vocabulary → longer sequences (more attention cost) but fewer embedding parameters. Larger vocabulary → shorter sequences (less attention cost) but more embedding parameters. It's a resource allocation decision.

Why does sequence length matter more than vocab size for attention cost?

Attention cost is

O(n^2 d)

where

n

is sequence length. Doubling sequence length quadruples attention cost. Embedding cost is only

O(V \times d)

, linear in vocab size

V

What is the embedding table parameter count formula?

Embedding parameters =

V \times d

where

V

is vocabulary size and

d

is embedding dimension. For GPT-2 small (

V=50257

d=768

), that's 38.6M parameters.

Why don't character-level models solve the vocabulary tradeoff?

They eliminate embedding table cost but explode sequence length (8-10x longer). Attention is

O(n^2)

, so this increases compute by ~64x. The tradeoff just moves from parameters to compute.

When does a larger vocabulary save resources?

When

(V_l - V_s) d < (n_s^2 - n_l^2) d \times \text{batch count}

. The extra embedding params must be outweighed by accumulated attention memory savings over many batches.

What's a typical vocabulary size for code models vs. general text?

Code models use 50K-100K (many unique identifiers). General text uses 32K-50K. Code needs larger vocab to avoid splitting function/variable names into many tokens.

Why is the bottom 10% of vocabulary by frequency important?

Rare tokens (bottom 10%) should still appear 50-100+ times in training for good embeddings. Otherwise you waste parameters on noisy embeddings that don't generalize. This guides vocab size choice.

Recall Explain to a 12-year-old

Imagine you're texting a friend, but you only have a limited "dictionary" of pre-approved text shortcuts.

Small dictionary (256 shortcuts): You can only send individual letters like "h", "e", "l", "o". To say "hello" you send 5 messages. That's slow!

Medium dictionary (32,000 shortcuts): You have shortcuts for common words. "hello" is one shortcut, "how" is another, "are" is another. Now "hello how are you" is just 4 shortcuts. Much faster!

Huge dictionary (200,000 shortcuts): You have shortcuts for entire phrases like "hello how are you doing today". Super fast! But now your phone needs to store 200,000 shortcuts in memory. That takes up space.

The tradeoff: Small dictionary = send LOTS of messages (slow). Huge dictionary = your phone is full of shortcuts (eats memory). You want the "goldilocks" size: enough shortcuts to be fast, but not so many that your phone runs out of space.

For AI models, "sending messages" is like the computer processing each token. More tokens = more work (especially because the computer has to compare every token with every other token, which gets really slow). But storing shortcuts (the vocabulary) also takes memory. So we pick a size that's "just right" for the task!

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, is note ka core intuition ye hai ki vocabulary size choose karna ek "goldilocks problem" hai — na bahut chhota, na bahut bada, balki bilkul theek hona chahiye. Jab hum text ko tokenize karte hain, toh basically ek tradeoff kar rahe hote hain. Agar vocabulary chhoti rakhein (jaise 256 tokens byte-level ke liye), toh har character alag token banega, matlab sequence lambi ho jayegi — aur transformer ka self-attention cost $O(n^2 d)$ hai, toh lambi sequence se compute bahut zyada badh jaata hai. Doosri taraf, agar vocabulary badi rakhein (100K+ tokens), toh sequence chhoti ho jayegi, par embedding table ka size $V \times d$ hone ki wajah se memory aur parameters bahut zyada kha jaate hain. Yahi fundamental tension hai.

Ab ye matter kyun karta hai? Kyunki resource allocation ki poori decision isi pe depend karti hai. Sochke dekho — $V = 5000$ aur $d = 768$ pe embedding table sirf 3.84M parameters ka hai, par $V = 50000$ pe wahi 38.4M ho jaata hai, yaani seedha linear growth. Waise hi sequence length double karo toh attention cost 4 guna badh jaata hai (kyunki $n^2$ term dominate karta hai), jabki feedforward sirf double hota hai. Interesting baat ye hai ki jab vocabulary badhaate hain, toh average token length sublinear tarike se badhti hai — $V$ ko double karne se $\bar{\ell}$ sirf 20-30% badhta hai, poore 2x nahi. Isiliye simple calculation se decide nahi hota.

Toh practically iska matlab ye hai ki bada vocabulary tabhi worthwhile hai jab uske extra embedding parameters ka permanent cost, shorter sequences se milne wale attention memory savings se kam ho. Real-world engineering mein — jaise koi model design karte time — yahi breakeven analysis batata hai ki apne use case ke hisaab se kaunsa vocabulary size lena chahiye. Agar tum long documents process kar rahe ho, toh attention cost dominate karega, isliye bada vocabulary faydemand ho sakta hai; par agar zyaadatar chhote texts hain, toh chhota vocabulary better hai. Ye samajhna important hai kyunki ye tumhe rattafication se aage le jaake, actual tradeoffs ke basis pe smart decisions lena sikhata hai.

Why vocabulary size matters

The mathematics of the tradeoff

Embedding table cost

Sequence processing cost

The breakeven analysis

Common vocabulary size choices

Practical guidelines

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Tokenization & Language Modeling

Connections