Vocabulary size tradeoffs — Hinglish notes

4.2.4 · AI-ML › Tokenization & Language Modeling

Why vocabulary size matters

Jab hum text ko tokenize karte hain, toh hum ek compression-granularity tradeoff kar rahe hote hain. Chhoti vocabulary (jaise, byte-level ke liye 256 tokens) har character ko alag se treat karti hai. Badi vocabulary (100K+ tokens) common words ya phrases poore ke poore ek hi token mein represent kar sakti hai.

YEH kyun matter karta hai:

Sequence length directly transformer compute ko impact karti hai: $O (n^{2} d)$ self-attention ke liye, jahan $n$ sequence length hai
Embedding table size $V \times d$ parameters hoti hai, jahan $V$ vocabulary size hai
Token granularity affect karti hai ki model rare/unseen words pe kitna generalize kar sakta hai

The mathematics of the tradeoff

Chalo resource costs ko first principles se derive karte hain.

Embedding table cost

Vocabulary size $V$ aur embedding dimension $d$ ke liye:

$Embedding params = V \times d$

KYU: $V$ tokens mein se har ek ko ek $d$ -dimensional dense vector representation chahiye.

ISKA matlab: Agar $V = 5000$ aur $d = 768$ (BERT-base), toh yeh $5000 \times 768 = 3.84 M$ parameters hain sirf embedding table ke liye. (Agar instead $V = 50000$ ho, toh yeh $50000 \times 768 = 38.4 M$ hoga.) Yeh $V$ ke saath linearly badhta hai.

Sequence processing cost

Sequence length $n$ , embedding dimension $d$ , aur $L$ transformer layers ke liye:

Self-attention cost per layer: $Attention cost = O (n^{2} d)$

KYU: Attention mechanism $Q K^{T}$ compute karta hai jo $(n \times d) \times (d \times n) = n \times n$ hai. Phir attention-weighted values: $(n \times n) \times (n \times d) = n \times d$ . $n^{2}$ term dominate karta hai.

Total forward pass cost: $Total cost \approx L \cdot n^{2} d + L \cdot n d^{2}$

$n^{2} d$ term (attention) aur $n d^{2}$ term (feedforward) dono sequence length ke saath scale karte hain.

ISKA matlab: Sequence length $n$ ko double karne se attention cost chaar guni ho jaati hai lekin feedforward cost sirf double hoti hai. Long documents ke liye, attention dominate karta hai.

The breakeven analysis

Kab ek badi vocabulary overall resources bachati hai?

Memory cost comparison:

Chhoti vocab ( $V_{s}$ ): Embedding cost $V_{s} d$ , sequence length $n_{s}$ , attention cost $\propto n_{s}^{2} d$
Badi vocab ( $V_{l}$ ): Embedding cost $V_{l} d$ , sequence length $n_{l}$ , attention cost $\propto n_{l}^{2} d$

Badi vocab jeetati hai agar: $extra embedding params (V_{l} - V_{s}) d < attention memory savings (n_{s}^{2} - n_{l}^{2}) d \cdot batch count$

YEH formula kyun: Left side badi vocabulary ka permanent parameter overhead hai. Right side chhoti sequences se per-batch activation memory savings hai. Agar aap bahut saare batches process karte ho (training, high-throughput inference), toh right side accumulate hoti hai aur left ko outweigh kar sakti hai.

Worked example GPT-2 vs. GPT-3 vocabulary choices

GPT-2: $V = 50257$ , $d = 768$ (small model)

Embedding table: $50257 \times 768 = 38.6 M$ parameters
Byte-level BPE use karta hai, average token ≈ 4 characters
Ek 1000-character text ke liye: $\approx 250$ tokens

GPT-3: $V = 50257$ (same!), lekin sabse bade model mein $d = 12288$ hai (175B model)

Embedding table: $50257 \times 12288 \approx 617.6 M$ parameters
Same tokenization, same sequence lengths GPT-3 ne same vocab kyun rakha: Embedding table cost $d$ ke saath scale karti hai. GPT-3 ke scale pe, ek badi vocab (maano $V = 200 K$ ) $(200 K - 50 K) \times 12288 \approx 1.84 B$ parameters add kar deti. Model already 175B ka hai, toh yeh tolerable hai, lekin OpenAI ne pehle doosri cheezein optimize kiin.

Alternative scenario — Chinese text:

Character-level ( $V \approx 1000$ common Chinese chars ke liye): Har character ek token hai
Subword BPE ( $V = 50 K$ ): Common words single tokens ban jaate hain (e.g., "学习" → ek token)
Ek 100-character Chinese text ke liye:
Character-level: 100 tokens
Subword: ~30-40 tokens (Chinese characters zyada information-dense hote hain)
Tradeoff: Character-level mein ~50x chhoti embedding table hai ( $V \approx 1 K$ vs $V = 50 K$ ) lekin 2.5-3x lambi sequences → attention cost $\approx 6 - 9 x$ zyada. Subword Chinese ke liye jeet jaata hai.

Worked example Worked calculation: Choosing vocab size for a medical chatbot

Scenario: Ek medical Q&A system bana rahe hain. Input documents average 2000 tokens hain (at $V = 32 K$ ). Embedding dim $d = 512$ . Batch size 8, 12 transformer layers.

Option 1: $V = 32000$ (standard BPE)

Embedding params: $32000 \times 512 = 16.4 M$
Sequence length: 2000 tokens
Attention score memory per batch: $8 \times 12 \times 2000 \times 2000 \times 2 bytes \approx 768 MB$
YEH calculation kyun: Attention score matrix $n \times n$ scalars hai per head-group per layer (hum yahan simplify karke per-layer le rahe hain). Batch size 8, 12 layers, $n \times n = 200 0^{2}$ scalars, har ek float16 mein stored (2 bytes): $8 \times 12 \times 200 0^{2} \times 2 \approx 768 MB$ .
Common mis-step: $d = 512$ se bhi multiply karna (jaise agar har score entry ek $d$ -vector ho) $\approx 393 GB$ deta hai — bilkul galat. Scores scalars hain, vectors nahi. Sirf $Q K V$ projections aur value-weighted outputs mein $d$ dimension hoti hai.

Option 2: $V = 128000$ (badi vocab, specialized medical terms)

Embedding params: $128000 \times 512 = 65.5 M$
Sequence length: $2000 \times 0.7 = 1400$ tokens (estimate: domain-specific merges se 30% reduction)
Attention score memory per batch: $8 \times 12 \times 1400 \times 1400 \times 2 bytes \approx 376 MB$

Analysis:

Parameter cost: +49M parameters (Option 2)
Memory savings per batch: $768 - 376 = 392 MB$ (Option 2)
Throughput: $(\frac{2000}{1400})^{2} \approx 2.04 x$ faster attention (Option 2)

Decision: Option 2 jeetata hai AGAR

Aap bahut saare batches train/serve karte ho (memory savings × batches > parameter cost)
Specialized vocab actually task performance improve kare (medical jargon ke liye kam "unk" tokens)
Aapke model budget mein 49M parameters spare hain

YEH step kyun? Specialized vocabulary se 30% sequence reduction (medical terms jaise "hypertension" ya "electrocardiogram" single tokens ke roop mein) attention cost ko $(0.7)^{- 2} \approx 2 x$ cut karta hai. Hazaron batches ke baad, memory savings compound ho jaati hain.

Common vocabulary size choices

Vocabulary Size	Typical Use Case	Pros	Cons
256–512	Byte-level, multilingual	Universal, koi UNK tokens nahi	Bahut lambi sequences
8K-16K	Early neural MT, chhote models	Balanced, manageable embedding size	Kuch rare words split ho jaate hain
32K-50K	GPT-2, BERT, RoBERTa	Standard tradeoff, proven	Rare technical terms se struggle
100K-250K	Newer LMs, code models	Chhoti sequences, domain coverage	Badi embedding tables

Common mistake "Larger vocabulary is always better for performance"

Yeh sahi kyun lagta hai: Zyada tokens = zyada specific representations = better understanding, hai na?

Steel-man: Yeh intuition word2vec era se aata hai: "king - man + woman = queen" isliye kaam karta hai kyunki "king" ek single token hai. Agar "king" ko "ki" + "ng" mein split kar diya jaaye, toh woh semantic relationship seekhna mushkil ho jaata hai.

The fix: Badi vocab ek haad tak help karti hai, phir:

Rare tokens ke embeddings kharab hote hain: Jo token training mein sirf 3 baar aaya, uska embedding noisy hoga. Aapne parameters waste kiye.
Overfitting: Model compositional patterns seekhne ki jagah specific rare tokens memorize kar leta hai.
Inference cost: Ek 500K vocab embedding table $d = 1024$ pe 2.5GB+ ki hoti hai. Yeh woh memory hai jo aap bade batch sizes ya lambe context ke liye use kar sakte the.

Better approach: $V$ itna chunna ki vocabulary ke bottom 10% tokens (frequency ke hisaab se) training data mein phir bhi kam se kam 50-100 baar aaye hoon. Rare words ke long tail ko handle karne ke liye subword tokenization use karo.

Common mistake "Character-level models eliminate all vocab tradeoffs"

Yeh sahi kyun lagta hai: 256 byte-level tokens ke saath, aapko koi OOV (out-of-vocabulary) problem nahi, koi bhi language support hoti hai, aur embedding tables tiny hoti hain.

Steel-man: Yeh appealing isliye hai kyunki yeh simple hai. Koi tokenization training nahi, koi weird BPE merge rules nahi, bas bytes ko tokens pe map karo.

The fix: Character-level models yeh cost kahin aur pay karte hain:

Sequence length explosion: "Tokenization" (14 chars) → 14 tokens instead of 1. Ek 512-token limit ~2000 characters (~400 words) ban jaati hai. GPT-3 ko same text ke liye ~8x lambi sequences chahiye hongi.
Long-range dependencies seekhna mushkil: Transformer attention $O (n^{2})$ hai. Jab $n$ 8x bada hota hai, toh same context dekhne ke liye 64x zyada attention compute chahiye.
Slower inference: Autoregressive generation $O (n)$ steps hai. 8x zyada tokens = 8x slow generation.

Empirical result: ByT5 (byte-level T5) subword T5 ke comparable perform karta hai lekin sequence length compensate karne ke liye zyada training aur bade models chahiye hote hain. "No vocab tradeoff" ka claim sirf cost ko compute pe shift kar deta hai.

Practical guidelines

Apne use case ke liye vocabulary size chunna:

General-domain text (web, books): 32K-50K subword vocab (BPE/WordPiece)
- KYU: Common word compression aur rare word splitting ke beech balance karta hai
Code models: 50K-100K vocab
- KYU: Code mein bahut saare unique identifiers, API names hote hain. Badi vocab get_user_profile → get, _user, _profile jaise function names ki splitting reduce karti hai
Multilingual (100+ languages): 250K+ vocab YA byte-level
- KYU: Har language ko apne frequent tokens chahiye. Byte-level explosion avoid karta hai lekin sequence-length cost pay karta hai.
Domain-specific (medical, legal): 50K-100K pe domain-specific vocab train karo
- KYU: "Myocardial infarction" 1-2 tokens hona chahiye, 6 nahi. Efficiency aur performance dono improve hoti hai.
Extremely long context (100K+ tokens): Chhoti vocab (16K-32K) ya hierarchical tokenization
- KYU: $n^{2}$ attention ke saath, har token count karta hai. Sequences chhoti rakhna aur rare words split karna better hai.

Connections

4.2.02-Byte-pair-encoding — BPE training merges ki number ke hisaab se alag vocab sizes produce karta hai
4.2.03-WordPiece-and-SentencePiece — SentencePiece vocab size hyperparameter ka direct control allow karta hai
4.3.01-Transformer-attention-complexity — $O (n^{2})$ cost jo vocabulary size indirectly impact karti hai
4.5.02-Embedding-layer-design — Embedding tables kaise implement aur optimize hoti hain
5.1.03-Memory-optimization-techniques — Flash Attention, gradient checkpointing $n^{2}$ cost reduce karte hain
6.2.01-Multilingual-models — 100+ languages ke liye massive vocab sizes kyun zaroori hain

Flashcards

#flashcards/ai-ml

Tokenization mein vocabulary size tradeoff kya hai?

Chhoti vocabulary → lambi sequences (zyada attention cost) lekin kam embedding parameters. Badi vocabulary → chhoti sequences (kam attention cost) lekin zyada embedding parameters. Yeh ek resource allocation decision hai.

Attention cost ke liye vocab size se sequence length zyada kyun matter karta hai?

Attention cost

O (n^{2} d)

hai jahan

n

sequence length hai. Sequence length double karne se attention cost chaar guni ho jaati hai. Embedding cost sirf

O (V \times d)

hai, vocab size

V

mein linear hai.

Embedding table parameter count ka formula kya hai?

Embedding parameters =

V \times d

jahan

V

vocabulary size hai aur

d

embedding dimension hai. GPT-2 small ke liye (

V = 50257

d = 768

), yeh 38.6M parameters hai.

Character-level models vocabulary tradeoff kyun solve nahi karte?

Woh embedding table cost eliminate karte hain lekin sequence length explode ho jaati hai (8-10x lambi). Attention

O (n^{2})

hai, isliye compute ~64x badh jaata hai. Tradeoff sirf parameters se compute pe shift ho jaata hai.

Badi vocabulary resources kab bachati hai?

Jab

(V_{l} - V_{s}) d < (n_{s}^{2} - n_{l}^{2}) d \times batch count

ho. Extra embedding params ko bahut saare batches ke accumulated attention memory savings se outweigh hona chahiye.

Code models vs. general text ke liye typical vocabulary size kya hai?

Code models 50K-100K use karte hain (bahut saare unique identifiers). General text 32K-50K use karta hai. Code ko function/variable names ko bahut saare tokens mein split hone se bachane ke liye badi vocab chahiye.

Vocabulary mein frequency ke hisaab se bottom 10% kyun important hai?

Rare tokens (bottom 10%) ko good embeddings ke liye training mein phir bhi 50-100+ baar aana chahiye. Warna aap noisy embeddings pe parameters waste karte ho jo generalize nahi karte. Yeh vocab size choice guide karta hai.

Recall Ek 12-saal ke bacche ko explain karo

Imagine karo tum ek dost ko text kar rahe ho, lekin tumhare paas sirf ek limited "dictionary" of pre-approved text shortcuts hai.

Chhoti dictionary (256 shortcuts): Tum sirf individual letters bhej sakte ho jaise "h", "e", "l", "o". "hello" kehne ke liye tum 5 messages bhejte ho. Yeh slow hai!

Medium dictionary (32,000 shortcuts): Tumhare paas common words ke shortcuts hain. "hello" ek shortcut hai, "how" doosra hai, "are" teesra hai. Ab "hello how are you" sirf 4 shortcuts hai. Kitna fast!

Huge dictionary (200,000 shortcuts): Tumhare paas poore phrases ke shortcuts hain jaise "hello how are you doing today". Super fast! Lekin ab tumhare phone ko 200,000 shortcuts memory mein store karne pad rahe hain. Yeh bahut space leta hai.

The tradeoff: Chhoti dictionary = BAHUT SAARE messages bhejne padte hain (slow). Huge dictionary = tumhara phone shortcuts se bhar jaata hai (memory khaata hai). Tum "goldilocks" size chahte ho: itne shortcuts ki fast ho, lekin itne nahi ki phone mein space na rahe.

AI models ke liye, "messages bhejna" aise hai jaise computer har token ko process karta hai. Zyada tokens = zyada kaam (especially kyunki computer ko har token ko baaki har token se compare karna padta hai, jo bahut slow ho jaata hai). Lekin shortcuts store karna (vocabulary) bhi memory leta hai. Toh hum ek size chunte hain jo task ke liye "just right" ho!