Tokenization fundamentals — Hinglish notes

4.2.1 · AI-ML › Tokenization & Language Modeling

Token Kya Hota Hai?

Words kyun nahi use karte?

Rare words problem: "antidisestablishmentarianism" training data mein sirf ek baar aata hai → model ise properly kabhi nahi seekh paata
Out-of-vocabulary (OOV): Naye words ("COVID-19") exist hi nahi karte the jab model train hua tha
Morphology: "run", "running", "ran" ko bilkul alag-alag treat kiya jaata hai → koi shared understanding nahi
Vocabulary explosion: English mein ~170K words hain, lekin proper nouns, typos, emoji ke saath infinite variations hain

Characters kyun nahi use karte?

Sequence length explosion: "Hello" = 5 tokens vs 1 word token → 5× lambi sequences
Computational cost: Transformer ki complexity sequence length mein O(n²) hai
Long-range dependencies: Model ke liye "H" + "e" + "l" + "o" ko meaning mein connect karna mushkil ho jaata hai

Tokenization Process

Step 1: Normalization

Kya hota hai: Text ko split karne se pehle clean aur standardize karo

Original: "  Hello,  WORLD! 🌍 "
Normalized: "hello, world! 🌍"

Common operations:

Lowercase conversion (optional)
Unicode normalization (NFC, NFD, NFKC, NFKD)
Whitespace trimming
Accent removal (optional)

Yeh kyun matter karta hai: "café" vs "cafe" vs "café" (different unicode) ideally same tokens se map hone chahiye.

Step 2: Pre-tokenization

Kya hota hai: Text ko "words" mein split karo jo aage aur split nahi honge

Input: "Don't stop learning!"
Pre-tokenized: ["Don", "'", "t", "stop", "learning", "!"]

Common rules:

Whitespace par split karo
Punctuation alag karo
Contractions ko saath rakho YA unhe split karo

Kyun zaroorat hai: Tokenizer algorithm ko kaam karne ke liye sensible boundaries milti hain.

Step 3: Token Splitting (Core Algorithm)

Yahi woh jagah hai jahan BPE, WordPiece, ya Unigram jaese subword algorithms operate karte hain. Chaliye Byte Pair Encoding (BPE) ko first principles se derive karte hain:

Formula BPE Algorithm Derivation

Goal: Ek aisi vocabulary seekhna jo frequency aur compositionality ko balance kare

Starting point: Character-level vocabulary $V_{0} = {all unique characters in corpus}$

Iteration step: Step $t$ par, sabse zyada frequent consecutive pair dhundo: $pair^{*} = ar g max_{pair} count (pair in current tokenization)$

Merge rule: Pair ko merge karke naya token banao: $V_{t + 1} = V_{t} \cup {pair^{*}}$

Update corpus: Pair ke saare occurrences ko naye token se replace karo Termination: Repeat karo jab tak $∣ V ∣ =$ target vocabulary size (e.g., 50K)

Yeh kyun kaam karta hai:

Frequent patterns (common words) pehle merge ho jaate hain → single tokens ban jaate hain
Rare patterns character combinations ke roop mein rehte hain → koi OV problem nahi
Subword structure (prefixes/suffixes) frequency se naturally emerge hoti hai

Worked example BPE Training Example

Corpus: ["low", "lower", "newest", "newest", "widest"]

Step 0 - Character vocabulary:

Tokens: l o w ␣ e r ␣ n e s t ␣ w i d e s t
Vocab: {l, o, w, e, r, n, s, t, i, d, ␣}
Tokenized corpus:
l o w | l o w | l o w e r | l o w e r | n e w e s t | n e w e s t | w i d e s t

Step 1 - Sabse zyada frequent pair dhundo: Saare adjacent pairs count karo:

"l o" 5 baar aata hai ("low" × 3 + "lower" × 2 mein)
"o w" 5 baar aata hai
"e s" 3 baar aata hai
"s t" 4 baar aata hai "l o" choose karo (ya "o w", dono 5 hain). Ise merge karo:

Vocab: {l, o, w, e, r, n, s, t, i, d, ␣, lo}
lo w | lo w | lo w | lo w e r | lo w e r | n e w e s t | w i d e s t

Step 2 - Agla frequent pair: Ab "lo w" 5 baar aata hai → merge karo:

Vocab: {l, o, w, e, r, n, s, t, i, d, ␣, lo, low}
low | low | low | low e r | n e w e s t | n e w e s t | w i d e s t

Step 3: "e s" 3 baar aata hai → merge karo:

Vocab: {..., es}
low | low | low | low e r | n e w es t | n e w es t | w i d es t

Step 4: "es t" 4 baar aata hai → merge karo:

Vocab: {..., est}
low | low | low | low e r | low e r | n e w est | n e w est | w i d est

Continue karo jab tak vocab size reach ho jaaye...

Har step kyun: Hum bottom-up se build kar rahe hain, pehle sabse common patterns capture karte hue. "low" frequent hai → ek token ban jaata hai. "est" suffix common hai → ek token ban jaata hai. Yeh generalize karta hai: "newest" aur "widest" ab "est" token share karte hain.

Step 4: Numericalization

Kya hota hai: Har token string ko ek unique integer ID se map karo

Vocabulary: {"<pad>": 0, "<unk>": 1, "hello": 100, "world": 250 ...}
Tokens: ["hello", "world"]
Token IDs: [100, 250]

Yeh ek simple dictionary lookup hai. Special tokens:

<pad> (ID 0): Batching ke liye padding
<unk> (ID 1): Unknown/rare tokens (fallback)
<bos> / <eos>: Sequence ki beginning/end
<mask>: Masked language modeling ke liye

Encoding aur Decoding

Worked example Full Encoding Example (GPT-2 Tokenizer)

Input text: "Hello, world!"

Step 1 - Normalization: (GPT-2 minimal normalization karta hai)

"Hello, world!"

Step 2 - Pre-tokenization: Whitespace par split karo, spaces ko Ġ se mark karo

["Hello", ",", "Ġworld", "!"]

(Ġ ek special character hai jo indicate karta hai "is token se pehle space tha")

Step 3 - BPE tokenization: Seekhe hue merges apply karo

"Hello" → ["Hello"]  (common word, single token)
"," → [","]
"Ġworld" → ["Ġworld"]  (space ke saath common word)
"!" → ["!"]

Step 4 - Numericalization:

Tokens: ["Hello", ",", "Ġworld", "!"]
IDs: [15496, 11, 995, 0]

Decoding:

[15496, 11, 995, 0] → ["Hello", ",", "Ġworld", "!"]
→ "Hello,Ġworld!" → "Hello, world!"

(Ġ ko actual space se replace karo)

GPT-2 Ġ kyun use karta hai: Whitespace position ki information preserve karta hai, perfect reversibility allow karta hai.

Acchi Tokenization Ki Key Properties

Common Tokenization Algorithms

Algorithm	Strategy	Used By
BPE (Byte-Pair Encoding)	Sabse zyada frequent pairs ko iteratively merge karo	GPT-2, GPT-3, RoBERTa
WordPiece	BPE jaisa, lekin likelihood maximize karta hai	BERT, DistilBERT
Unigram	Badi vocab se shuru karo, low-probability tokens prune karo	T5, ALBERT
SentencePiece	Text ko raw byte stream treat karta hai, language-agnostic	T5, XLNet, bahut se multilingual models

Alag-alag algorithms kyun?

BPE: Simple, greedy, deterministic
WordPiece: Zyada principled (likelihood-based), BERT-style models ke liye thoda better
Unigram: Probabilistic tokenization allow karta hai (multiple valid splits), Japanese/Chinese mein use hota hai
SentencePiece: Aise languages handle karta hai jahan clear word boundaries nahi hote, koi pre-tokenization ki zaroorat nahi

Common mistake Common Tokenization Pitfalls

Galti 1: "Tokenization sirf spaces par split karna hai" Yeh sahi kyun lagta hai: English mein words spaces se separate hote hain—obvious lagta hai!

Yeh galat kyun hai:

Chinese/Japanese jaisi languages mein spaces nahi hote: "我爱编程" (I love programming)
Compounds: "New York", "COVID-19" → kya "New" aur "York" alag hone chahiye?
Morphology: "running" = "run" + "ing" → atomic treat karne par shared structure kho jaati hai
Rare words: "antidisestablishmentarianism" → model ise ek unit ke roop mein kabhi nahi seekh paata

Fix: Subword tokenization use karo. BPE/WordPiece seekhte hain ki "running" = "run" + "##ing", toh model "jumping" = "jump" + "##ing" jaisi nayi forms samajh sakta hai.

Galti 2: "Badi vocabulary hamesha better hoti hai"

Yeh sahi kyun lagta hai: Zyada tokens = zyada precision, rare words capture ho jaate hain!

Yeh galat kyun hai:

Embedding matrix size: $∣ V ∣ \times d$ parameters. 100K vocab × 768 dim = sirf embeddings ke liye 77M parameters!
Rare tokens: Agar "antidisestablishmentarianism" sirf ek baar aata hai, toh uski embedding kabhi properly train nahi hoti
Softmax output: 100K classes par probability distribution → expensive, train karna mushkil Fix: Optimal vocab size coverage aur trainability ko balance karta hai. 30K-50K zyaadatar languages ke liye sweet spot hai. Subwords rare words ko composition se handle karte hain.

Galti 3: "Tokenization sirf preprocessing hai, model performance affect nahi karta"

Yeh sahi kyun lagta hai: Yeh toh sirf text ko numbers mein convert kar raha hai—model asli learning karta hai!

Yeh galat kyun hai:

Token boundaries = hard limits on what model can learn. Agar "New York" ek token hai, toh model "New" alag se nahi seekh sakta.
OOV handling: Kharaab tokenization → bahut saare <unk> tokens → model us content ke liye blind ho jaata hai
Sequence length: Inefficient tokenization → lambi sequences → quadratic attention cost
Multilingual: Byte-level tokenization truly universal models (mT5, BLOOM) enable karta hai

Real example: GPT-2 jo English text par train hua, Python code mein " (space)" ko inefficiently encode karta hai, code generation hurt karta hai. Baad ke models (Codex) code-aware tokenization use karte hain.

Fix: Tokenization ek core modeling choice hai. Apne domain aur language distribution ke saath tokenizer match karo.

Recall Ek 12-Saal-Ke Bachche Ko Samjhao

Socho tum ek robot ko padhna sikha rahe ho, lekin robot sirf numbers samajhta hai, letters ya words nahi.

Problem: Tum robot ko sentence dikhate ho "I love pizza," lekin woh sirf tedhe-medhe shapes dekhta hai. Usse ek translator chahiye.

Buri solution 1: Sab kuch letters mein todo: I → 9, (space) → 0, l → 12, o → 15, v → 22, e → 5... Ab "love" hai [12, 15, 22, 5]. Poora sentence 13 numbers ki list ban jaata hai! Yeh bahut lamba hai, aur robot ko yaad rakhna hoga ki "l + o + v + e" feelings ke baare mein kuch kehta hai—super mushkil!

Buri solution 2: Har possible word ko ek number do: "I" → 1, "love" → 2, "pizza" → 3. Great! Choti list: [1, 2, 3]. Lekin kal agar tum kaho "I adore pizza"? Robot "adore" nahi jaanta kyunki humne kabhi woh number nahi sikhaya. Aur English mein 170,000 words hain—bahut zyada numbers yaad rakhne ke liye!

Smart solution (tokenization): Words ko common PIECES mein todo. Jaise Lego blocks!

Common words saath rehte hain: "I" → 1, "love" → 2, "pizza" → 3
Rare words pieces mein toote hain: "adore" → "ad" (17) + "ore" (18)
Ab agar robot ne "ad" "advent" se seekha aur "ore" "explore" se, toh woh "adore" samajh sakta hai chahe yeh naya ho!

Pieces kaise chunte hain: Hum bahut saare sentences dekhte hain aur woh letter pairs dhoondh te hain jo HAMESHA saath aate hain. Jaise "th," "ing," "ed." Woh blocks ban jaate hain. Hum combine karte rehte hain jab tak hamare paas roughly 50,000 alag blocks na ho jaayein—koi bhi word build karne ke liye kaafi, lekin yaad rakhne ke liye bahut zyada nahi.

Magic moment: Jab robot "I'm running to eat pizza" padhta hai, woh jaanta hai "running" = "run" + "ing" aur "eating" = "eat" + "ing," toh woh pattern samajhta hai chahe usne pehle sirf "running" dekha ho, "eating" nahi!

Connections

4.2.02-BPE-algorithm - Byte Pair Encoding ka deep dive
4.2.03-WordPiece-and-SentencePiece - Alternative tokenization algorithms
4.3.01-Word-embedings - Tokens → vectors (agla step)
4.1.05-Vocabulary-and-OV - Vocabulary management kyun matter karta hai
5.1.02-Transformer-architecture - Jahaan tokenized sequences process hote hain
4.2.08-Multilingual-tokenization - Non-English text handle karna
3.4.03-Sequence-length-and-padding - Variable-length tokenized sequences se deal karna

#flashcards/ai-ml

NLP mein token kya hota hai? :: Woh basic unit of text jo ek language model process karta hai. Ek complete word, subword, character, ya byte ho sakta hai. Model ki vocabulary saare possible tokens ka set hoti hai.

Word-level ki jagah subword tokenization kyun use karte hain?

(1) Rare/unseen words ko composition se handle karta hai (2) Vocabulary size reduce karta hai (3) Morphology capture karta hai (prefixes/suffixes) (4) Out-of-vocabulary problem nahi hoti (5) Character-level se zyada efficient

Tokenization ke 4 steps kya hain? :: (1) Normalization (text clean karo) (2) Pre-tokenization (words mein split karo) (3) Token splitting (subword algorithm jaise BPE apply karo) (4) Numericalization (integer IDs se map karo)

BPE ka core idea kya hai?

Character vocabulary se shuru karo. Iteratively sabse zyada frequent consecutive pair dhundo aur use naye token mein merge karo. Target vocabulary size tak repeat karo. Frequent patterns single tokens ban jaate hain; rare words character combinations rehte hain.

Modern LMs ke liye typical vocabulary size kya hoti hai?

30K-100K tokens. Coverage (kam <unk> tokens) aur efficiency (chhhota embedding matrix, faster softmax) ke beech balance.

GPT-2 Ġ character kyun use karta hai?

Woh tokens mark karta hai jinse pehle space tha. Token IDs ko wapas text mein decode karte waqt perfect reversibility ke liye whitespace information preserve karta hai.

Token fertility kya hoti hai?

Average number of tokens per word. Typical BPE fertility 1.3-1.5 hoti hai. Kam = zyada efficient tokenization (same text ke liye choti sequences).

Vocabulary bahut badi ho toh kya hota hai?

(1) Bada embedding matrix (zyada parameters) (2) Rare tokens achhe se train nahi hote (sparse updates) (3) Badi vocabulary par expensive softmax (4) Slower training aur inference

BPE aur WordPiece mein kya fark hai?

BPE sabse zyada frequent pairs merge karta hai (greedy). WordPiece woh pairs merge karta hai jo training data ki likelihood maximize karein (zyada principled). Dono similar results dete hain; WordPiece BERT mein use hota hai, BPE GPT models mein.

Model performance ke liye tokenization kyun important hai?

(1) Token boundaries limit karte hain ki kya seekha ja sakta hai (2) Kharaab tokenization → lambi sequences → zyada compute cost (3) Bahut saare <unk> → model information kho deta hai (4) Domain mismatch hurt karta hai (jaise code vs natural language) (5) Multilingual capability affect hoti hai