Masked language modeling (BERT)

4.2.7 · AI-ML › Tokenization & Language Modeling

Overview

Masked Language Modeling (MLM) ek pre-training objective hai jisme hum input sequence mein kuch tokens ko randomly mask karte hain aur model ko train karte hain ki woh original masked tokens predict kare unke bidirectional context ke basis par. Yahi BERT (Bidirectional Encoder Representations from Transformers) ke peeche ki core innovation hai.

YEH kyun important hai? Traditional language models (jaise GPT) unidirectional hote hain — woh sirf baayein taraf ke words dekhte hain. Lekin language ko samajhne ke liye DONO directions dekhni padti hain. "The bank was steep" vs "The bank was closed" — yeh samajhne ke liye ki hum riverbank ki baat kar rahe hain ya financial institution ki, DONO sides ke words chahiye.

Next-word prediction (jo causal/autoregressive hai) ke unlike, MLM ek denoising autoencoder approach hai — hum input ko corrupt karte hain aur use reconstruct karna seekhte hain.

---

The Masking Strategy

15% kyun? Balance ke liye:

• Zyada kam → training signal kaafi nahi
• Zyada zyada → bahut zyada context hat jaata hai, predict karna mushkil ho jaata hai

100% [MASK] kyun nahi? [MASK] token fine-tuning (downstream tasks) ke dauran kabhi appear nahi hota, isliye hame ek train-test mismatch hogi. 10% random tokens aur 10% unchanged use karke, model robust rehna seekhta hai.

MLM loss yeh hai: ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

jahan ${\mathbf{x}}_{\backslash M}$ corrupted sequence (masks ke saath) ko represent karta hai.

First principles se DERIVATION:

1. Likelihood se shuru karo: Hum chahte hain ki corrupted input ke given original tokens ki probability maximize ho. $P({\mathbf{x}}_M\mid {\mathbf{x}}_{\backslash M})={\prod }_{i\in M}P(x_i\mid {\mathbf{x}}_{\backslash M})$

   Product kyun? Kyunki context ke given predictions ki conditional independence assume ki gayi hai (har masked token independently predict hota hai).

2. Numerical stability ke liye log lo: $\log P({\mathbf{x}}_M\mid {\mathbf{x}}_{\backslash M})={\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

   Log kyun? Probabilities ke products → underflow. Sums stable hote hain.

3. Minimization ke liye negate karo: ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$ Log-likelihood maximize karna = negative log-likelihood minimize karna. Negative isliye kyunki gradient descent loss minimize karta hai.

4. Implementation: Probability $P(x_i\mid {\mathbf{x}}_{\backslash M})$ yahan se aati hai: $P(x_i\mid {\mathbf{x}}_{\backslash M})=\frac{\exp ({\mathbf{w}}_{x_i}^{\top }{\mathbf{h}}_i)}{{\sum }_{v\in \mathcal{V}}\exp ({\mathbf{w}}_v^{\top }{\mathbf{h}}_i)}$

   jahan:

   • ${\mathbf{h}}_i$ = position $i$ par Transformer se contextualized embedding
   • ${\mathbf{w}}_v$ = vocabulary token $v$ ke liye output embedding
   • Yeh poore vocabulary par ek softmax hai

---

Architecture Details

Encoder-only kyun? Kyunki hum sequences autoregressively generate nahi kar rahe. Hume understanding ke liye bidirectional context chahiye, left-to-right generation nahi.

Input Representation

BERT ka input teen embeddings combine karta hai (element-wise add hoti hain):

${\mathbf{E}}_{\text{input}}(i)={\mathbf{E}}_{\text{token}}(x_i)+{\mathbf{E}}_{\text{position}}(i)+{\mathbf{E}}_{\text{segment}}(s_i)$

Teen types kyun?

• Token embedding: Word/subword khud
• Position embedding: Sequence mein kaun si position (learned, original Transformer jaisi sinusoidal nahi)
• Segment embedding: Kaun sa sentence (A ya B) — Next Sentence Prediction (NSP) ke liye zaroori hai, jo doosra pre-training task hai

WordPiece tokenization ke baad: [CLS] the cat sat on the mat [SEP]

Masking (15% → yahan hum "cat" mask karte hain):

[CLS] the [MASK] sat on the mat [SEP]

Token embeddings: Vocab mein look up karo Position embeddings: [0, 1, 2, 3, 4, 5, 6, 7] Segment embeddings: [A, A, A] (sab sentence A)

Model hidden states output karta hai $\mathbf{H}=({\mathbf{h}}_0,{\mathbf{h}}_1,\dots ,{\mathbf{h}}_7)$

Position 2 par (jahan [MASK] hai), hum compute karte hain: $P(x_2=\text{"cat"}\mid \text{context})=\text{softmax}(\mathbf{W}{\mathbf{h}}_2+\mathbf{b})[\text{cat\_id}]$

jahan $\mathbf{W}$ vocabulary size par back project karta hai.

---

Training Process

Input sentence: "The quick brown fox jumps"

Step 1 - Tokenize:

[CLS] the quick brown fox jumps [SEP]

Step 2 - Random selection (15%): Maano hum positions {3, 5} par tokens select karte hain (brown, jumps)

Step 3 - Masking protocol apply karo:

• Position 3: 80% chance → [MASK]
• Position 5: 10% chance → random token, maano "happy"

Result:

[CLS] the quick [MASK] fox happy [SEP]

Step 4 - Forward pass: Transformer encoder ${\mathbf{h}}_3$ aur ${\mathbf{h}}_5$ produce karta hai

Step 5 - Originals predict karo:

• Position 3 par: Vocab par $P(\cdot \mid {\mathbf{h}}_3)$ compute karo. True label: "brown"
• Position 5 par: Vocab par $P(\cdot \mid {\mathbf{h}}_5)$ compute karo. True label: "jumps"

Step 6 - Loss compute karo: $\mathcal{L}=-\log P(\text{"brown"}\mid {\mathbf{h}}_3)-\log P(\text{"jumps"}\mid {\mathbf{h}}_5)$

Step 7 - Backprop aur weights update karo

Yeh kyun kaam karta hai: Model position-token mappings memorize karke cheat NAHI KAR SAKTA kyunki:

1. Sirf 15% masked hai (har position nahi)
2. Random corruption (10% random, 10% unchanged)
3. Bidirectional attention force karta hai ki SAARA context use ho

---

Differences from Other Language Models

Aspect	GPT (Causal LM)	BERT (Masked LM)	T5 (Seq2Seq)
Architecture	Decoder-only	Encoder-only	Encoder-Decoder
Attention	Causal (left-to-right)	Bidirectional	Enc: bidirectional, Dec: causal
Objective	Next token predict karo	Masked tokens predict karo	Span corruption
Use case	Generation	Understanding/Classification	Dono
Context	Unidirectional	Full bidirectional	Bidirectional encoding

GPT bidirectional attention kyun nahi use kar sakta? Kyunki yeh autoregressive generation ke liye design kiya gaya hai. Agar yeh "future dekh sakta", toh generation ke dauran cheat karta.

BERT bidirectional kyun use kar sakta hai? Kyunki hum generate nahi kar rahe — hum understanding ke liye encode kar rahe hain. Fine-tuning ke dauran (classification, QA), hum generate nahi karte, sirf rich representations chahiye hoti hain.

---

Next Sentence Prediction (NSP)

BERT MLM ke saath ek doosra pre-training objective bhi use karta hai:

Loss yeh hai: ${\mathcal{L}}_{\text{NSP}}=-\log P(\text{label}\mid [\text{CLS}])$

jahan [CLS] token ki final hidden state ${\mathbf{h}}_{[\text{CLS}]}$ ek binary classifier se pass ki jaati hai.

NSP kyun? BERT ko sentence-level relationships seekhne mein help karne ke liye, Question Answering aur Natural Language Inference jaise tasks ke liye.

TOTAL BERT LOSS: ${\mathcal{L}}_{\text{BERT}}={\mathcal{L}}_{\text{MLM}}+{\mathcal{L}}_{\text{NSP}}$

BERT ka input:

[CLS] I went to the store [SEP] I bought some milk [SEP]

NotNext pair:

• Sentence A: "I went to the store."
• Sentence B: "The moon orbits Earth."
• Label: 0

Input:

[CLS] I went to the store [SEP] The moon orbits Earth [SEP]

Model seekhta hai ki ${\mathbf{h}}_{[\text{CLS}]}$ ko ek sentence-pair representation ki tarah use kare.

---

Fine-tuning BERT

Massive text corpora par MLM+NSP ke saath pre-training ke baad, BERT ko downstream tasks par fine-tune kiya jaata hai:

Classification tasks ke liye: $y=\text{softmax}(\mathbf{W}{\mathbf{h}}_{[\text{CLS}]}+\mathbf{b})$

Token-level tasks ke liye (NER, POS tagging): $y_i=\text{softmax}(\mathbf{W}{\mathbf{h}}_i+\mathbf{b})$

Question Answering ke liye (span extraction): Start/end pointers seekho: $P(\text{start}=i)=\text{softmax}({\mathbf{S}}^{\top }{\mathbf{h}}_i)$ $P(\text{end}=j)=\text{softmax}({\mathbf{E}}^{\top }{\mathbf{h}}_j)$

Yeh kyun kaam karta hai: Pre-trained bidirectional representations pehle se hi rich contextual information encode karte hain. Hum sirf upar ek patla task-specific layer add karte hain.

Key insight: Same pre-trained model → bahut saare alag-alag tasks. "Knowledge" encoder weights mein hoti hai.

---

Common Mistakes

Kyun sahi lagta hai: Humne [MASK] ke saath train kiya, toh [MASK] ke saath hi test karein.

Fix: Fine-tuning ke dauran kabhi [MASK] use mat karo. Downstream tasks mein real text hota hai, koi masks nahi. Isliye hi BERT sirf 80% ko [MASK] se mask karta hai aur baaki 20% ke liye random/unchanged use karta hai — train-test mismatch kam karne ke liye.

Steel-man: Distribution shift ki chinta valid hai! Lekin pre-training mein 10% random + 10% unchanged robustness create karta hai. Model seekhta hai: "kabhi kabhi [MASK] hota hai, kabhi nahi — mujhe context use karna hi padega."

Kyun sahi lagta hai: BERT tokens predict karta hai, toh text generate karna chahiye.

Fix: BERT ek encoder hai, generator nahi. Yeh bidirectional context ke given tokens parallel mein predict karta hai. Isme left-to-right autoregressive generation ka koi notion nahi hai. Generation ke liye, GPT-style models use karo.

Steel-man: Technically tum iteratively mask aur predict karke generate kar sakte ho, lekin yeh inefficient aur incoherent hoga kyunki BERT ko sequential generation ke liye train nahi kiya gaya. Sahi tool use karo.

Kyun sahi lagta hai: Agar hum sab kuch mask karein, toh sab kuch par train karte hain.

Fix: 10% random + 10% unchanged crucial hai. Agar humne sirf [MASK] dekha hota, toh model ek shortcut seekh leta: "agar [MASK] dikhta hai, context ke basis par predict karo; agar real token dikhta hai, ignore karo." Fine-tuning ke dauran (koi [MASK] nahi), yeh catastrophically fail ho jaata.

---

Mathematical Properties

## 🖼️ Concept Map ```mermaid flowchart TD MLM[Masked Language Modeling] BERT[BERT Bidirectional Encoder] Bidir[Bidirectional Context] Uni[Unidirectional GPT] Denoise[Denoising Autoencoder] Mask15[Mask 15% of Tokens] Split[80% MASK / 10% Random / 10% Keep] Predict[Predict Original Tokens] Loss[MLM Loss] Mismatch[Train-Test Mismatch] MLM -->|core of| BERT BERT -->|uses| Bidir Bidir -->|contrasts with| Uni MLM -->|is a| Denoise Denoise -->|corrupts via| Mask15 Mask15 -->|split into| Split Split -->|avoids| Mismatch Mask15 -->|then| Predict Predict -->|optimized by| Loss Loss -->|derived from| Predict ```