BERT and encoder models — Hinglish notes

4.3.2 · AI-ML › Pretraining & Fine-Tuning LLMs

Encoder model kya hota hai?

CONTRAST kya hai?

Family	Attention	Trained for	Example
Encoder-only	bidirectional	understanding	BERT
Decoder-only	causal (left→right)	next-token generation	GPT
Encoder-decoder	dono	seq-to-seq (translate)	T5

Pretraining objective 1: Masked Language Modelling (MLM)

HOW 15% handle hote hain (80/10/10 rule):

80% → ek special [MASK] token se replace kiya jaata hai
10% → ek random token se replace kiya jaata hai
10% → unchanged chhoda jaata hai

MLM loss ko first principles se derive karna. Maano $M$ masked positions ka set hai. Ek masked position $i$ ke liye, model contextual vector $h_{i}$ output karta hai, use vocab logits $z_{i} = W h_{i} + b$ mein project karta hai, aur vocabulary $V$ par ek probability banata hai:

$p (w ∣ context) = softmax (z_{i})_{w} = \frac{e ^{z_{i, w}}}{\sum _{v \in V} e ^{z_{i, v}}}$

Hum true token $y_{i}$ ki probability maximize karna chahte hain. Probability maximize karna = uska negative log minimize karna. Saare masked positions par sum karte hue:

Pretraining objective 2: Next Sentence Prediction (NSP)

WHY: bahut saare downstream tasks (QA, entailment) ko do sentences ke beech relationships ki zarurat hoti hai, jo single-sentence MLM nahi sikhata.

$L_{NSP} = - [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})], \overset{p}{^} = σ (w^{⊤} h_{[CLS]})$

Total pretraining loss hai $L = L_{MLM} + L_{NSP}$ .

Special tokens aur input format

Har BERT input aisa dikhta hai: $[CLS] A_{1} \dots A_{m} [SEP] B_{1} \dots B_{k} [SEP]$

==[CLS]== — ek token jiska final vector $h_{[CLS]}$ classification ke liye sentence-level summary ke roop mein use hota hai.
[SEP] — sentence A ko sentence B se alag karta hai.
Tin embeddings sum ki jaati hain har token ke liye: token + position + segment (A vs B).

Fine-tuning: WHY ek model → bahut saare tasks

Sentence classification (sentiment): $h_{[CLS]}$ → linear → softmax feed karo.
Token classification (NER, POS): har $h_{i}$ → linear → per-token label feed karo.
Span extraction (SQuAD QA): tokens par start/end positions predict karo.

Worked examples

Common mistakes

Active recall

Recall Khud test karo (hidden — pehle answer karo!)

BERT next-token prediction kyun use nahi kar sakta? → bidirectional attention isko answer dekhne deta.
80/10/10 rule kis liye hai? → sirf [MASK] wale train/test mismatch ko rokne ke liye.
RoBERTa ne kaunsa objective drop kiya? → NSP.
Sentence summary kaunsa token deta hai? → [CLS].

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho ek sentence hai jismein kuch words stickers se chhupe hain. Ek bahut smart bachcha har sticker ke aas-paas sab kuch padhta hai — pehle ke words AND baad ke words — aur chhupa hua word guess karta hai. Yeh pakka karne ke liye ki bachcha actually padhe na ki yaad kare ki stickers kahan hain, kabhi kabhi hum sticker ki jagah galat word daalte hain, aur kabhi asli word chhod dete hain. Bahut zyada practice ke baad, woh bachcha sentences itne achhe se samajhta hai ki aap use chhote naye kaam — jaise "kya yeh review khush hai ya udaas?" — sirf thode examples se jaldi seekha sakte ho. Woh smart reader BERT hai.

Flashcards

BERT kis tarah ka Transformer hai (encoder/decoder)?

Encoder-only, bidirectional attention.

BERT ko next-token prediction se kyun train nahi kar sakte?

Bidirectional attention har token ko us future token ko attend karne deta hai jise use predict karna hai — yeh trivial cheating hai.

MLM mein kitne fraction tokens select hote hain?

15%.

80/10/10 rule describe karo.

Selected tokens mein se: 80% → [MASK], 10% → random token, 10% → unchanged.

10% unchanged / 10% random kyun rakhte hain hamesha [MASK] ki jagah?

[MASK] inference par kabhi nahi aata; har token ke baare mein uncertainty force karna har token ke liye achhi representations deta hai (train/test mismatch avoid karta hai).

MLM loss likho.

- \frac{1}{∣ M ∣} \sum_{i \in M} lo g p (y_{i} ∣ context)

, sirf masked positions par cross-entropy.

NSP kya hai aur kya yeh zaruri hai?

Binary "kya B, A ke baad ka next sentence hai"; RoBERTa ne dikhaya ki yeh aksar unnecessary/harmful hota hai.

BERT mein teen summed input embeddings kya hain?

Token + position + segment embeddings.

Position embeddings kyun zaruri hain?

Self-attention permutation-invariant hai; positions word order inject karte hain.

[CLS] token kis liye use hota hai?

Classification ke liye aggregated sentence-level representation (NSP aur fine-tuning mein use hota hai).

Sentiment ke liye BERT ko fine-tune kaise karte hain?

h_{[C L S]}

par ek linear+softmax head add karo aur labelled data par train karo.

Token classification (NER) kaise hota hai?

Har token ke

h_{i}

par ek shared linear layer apply karo.

Connections

Transformer Architecture — BERT apne encoder blocks stack karta hai.
Self-Attention — bidirectional (unmasked) version BERT ko power deta hai.
GPT and decoder models — generative, causal-masked counterpart.
T5 and encoder-decoder models — text-to-text unifier.
Cross-Entropy Loss — MLM aur NSP objectives.
Fine-Tuning vs Pretraining — do-stage paradigm.
WordPiece Tokenization — BERT text ko sub-word tokens mein kaise split karta hai.
RoBERTa — BERT better train hua, NSP drop kiya.