Intuition The big picture (WHY these exist)Language is bidirectional : to understand a word you need context from both sides. In "I went to the bank to withdraw money", the word after ("withdraw money") is what tells you bank is financial, not a riverbank. Old left-to-right models could not peek at future tokens. BERT's whole reason for existing is to build word representations that look at the entire sentence at once — left AND right — so each token's meaning is fully contextualized. It is an encoder-only model: it reads and understands , it does not generate text word by word.
Definition Encoder-only TransformerA stack of Transformer encoder blocks (self-attention + feed-forward) where attention is bidirectional (no causal mask). It maps a sequence of tokens x 1 , … , x n x_1,\dots,x_n x 1 , … , x n h 1 , … , h n h_1,\dots,h_n h 1 , … , h n h i ∈ R d h_i \in \mathbb{R}^{d} h i ∈ R d understand , then fine-tuned for classification/tagging tasks — not for open-ended generation.
WHAT is the contrast?
Family
Attention
Trained for
Example
Encoder-only
bidirectional
understanding
BERT
Decoder-only
causal (left→right)
next-token generation
GPT
Encoder-decoder
both
seq-to-seq (translate)
T5
Intuition WHY bidirectional needs a special trickIf attention sees the whole sentence and you train it to predict the next word, the model can just look at the answer — it "cheats" by attending to the token it must predict. So a decoder uses a causal mask to hide the future. BERT wants bidirectionality, so it can't use next-word prediction. It needs a different, cheat-proof objective. That objective is Masked Language Modelling .
Randomly 15% of input tokens are chosen. The model must predict the original token at those positions from the full bidirectional context.
HOW the 15% are handled (the 80/10/10 rule):
80% → replaced with a special [MASK] token
10% → replaced with a random token
10% → left unchanged
Intuition WHY not just always use[MASK]?
[MASK] never appears at fine-tuning/inference time. If the model only ever saw [MASK] at the "predict me" positions, it would learn "only produce good representations for [MASK] tokens" — a train/test mismatch. The 10% random + 10% unchanged force the model to build a good representation for every token , because it can never be sure whether a given token is corrupt. That uncertainty is the whole point.
Deriving the MLM loss from first principles. Let M M M i i i h i h_i h i z i = W h i + b z_i = W h_i + b z i = W h i + b V V V
p ( w ∣ context ) = softmax ( z i ) w = e z i , w ∑ v ∈ V e z i , v p(w \mid \text{context}) = \operatorname{softmax}(z_i)_w = \frac{e^{z_{i,w}}}{\sum_{v\in V} e^{z_{i,v}}} p ( w ∣ context ) = softmax ( z i ) w = ∑ v ∈ V e z i , v e z i , w
We want to maximize the probability of the true token y i y_i y i
Given two sentences A and B, a binary classification: is B the actual next sentence after A (label IsNext), or a random sentence from the corpus (NotNext)? 50/50 split.
WHY: many downstream tasks (QA, entailment) need relationships between two sentences , which single-sentence MLM doesn't teach.
L NSP = − [ y log p ^ + ( 1 − y ) log ( 1 − p ^ ) ] , p ^ = σ ( w ⊤ h [CLS] ) \mathcal{L}_{\text{NSP}} = -\big[y\log \hat p + (1-y)\log(1-\hat p)\big],\qquad \hat p = \sigma(w^\top h_{\texttt{[CLS]}}) L NSP = − [ y log p ^ + ( 1 − y ) log ( 1 − p ^ ) ] , p ^ = σ ( w ⊤ h [CLS] )
The total pretraining loss is L = L MLM + L NSP \mathcal{L} = \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}} L = L MLM + L NSP
Common mistake Steel-man: "NSP is essential and always used."It feels right because BERT's paper included it and got great results. The fix: later work (RoBERTa) showed NSP is often useless or slightly harmful ; training MLM alone on more data does better. So NSP is optional — MLM is the load-bearing objective.
Every BERT input looks like:
[CLS] A 1 … A m [SEP] B 1 … B k [SEP] \texttt{[CLS]}\; A_1 \dots A_m \;\texttt{[SEP]}\; B_1 \dots B_k \;\texttt{[SEP]} [CLS] A 1 … A m [SEP] B 1 … B k [SEP]
==[CLS]== — a token whose final vector h [CLS] h_{\texttt{[CLS]}} h [CLS] sentence-level summary for classification.
[SEP] — separates sentence A from sentence B.Three embeddings summed per token: token + position + segment (A vs B).
Intuition WHY position embeddings?Self-attention is permutation-invariant — it treats a bag of tokens the same regardless of order. "Dog bites man" ≠ "Man bites dog", so we inject order by adding a learned position vector to each token.
Pretraining is expensive but done once . It gives universal language representations. Fine-tuning adds a tiny task head (usually one linear layer) on top and trains briefly on a small labelled dataset. Because h i h_i h i
Sentence classification (sentiment): feed h [CLS] h_{\texttt{[CLS]}} h [CLS] Token classification (NER, POS): feed each h i h_i h i Span extraction (SQuAD QA): predict start/end positions over tokens.
Worked example Example 1 — MLM loss for one masked tokenSentence: "The cat sat on the [MASK] ". Vocab has 3 relevant words with logits z = [ mat : 2.0 , car : 1.0 , sky : 0.0 ] z = [\text{mat}:2.0,\ \text{car}:1.0,\ \text{sky}:0.0] z = [ mat : 2.0 , car : 1.0 , sky : 0.0 ] Step 1 — softmax. Why? We need probabilities.
e 2 = 7.389 , e 1 = 2.718 , e 0 = 1 e^{2}=7.389,\ e^{1}=2.718,\ e^{0}=1 e 2 = 7.389 , e 1 = 2.718 , e 0 = 1 = 11.107 =11.107 = 11.107 p ( mat ) = 7.389 / 11.107 = 0.665 p(\text{mat}) = 7.389/11.107 = 0.665 p ( mat ) = 7.389/11.107 = 0.665 Step 2 — loss. Why? MLM = negative log of true-token prob.
L = − log ( 0.665 ) = 0.408 \mathcal{L} = -\log(0.665) = 0.408 L = − log ( 0.665 ) = 0.408 Interpretation: low-ish loss, model already fairly confident.
Worked example Example 2 — Why only masked positions countSentence has 10 tokens, 2 are masked. We compute cross-entropy only for those 2 and average.
Why this step? The 8 visible tokens are given — grading the model for "predicting" tokens it can literally see would teach nothing and leak information.
Worked example Example 3 — Fine-tuning sentimentInput [CLS] this movie is great [SEP]. Take h [CLS] ∈ R 768 h_{\texttt{[CLS]}}\in\mathbb R^{768} h [CLS] ∈ R 768 W ∈ R 2 × 768 W\in\mathbb R^{2\times 768} W ∈ R 2 × 768 [ P ( neg ) , P ( pos ) ] [P(\text{neg}), P(\text{pos})] [ P ( neg ) , P ( pos )] Why [CLS]? During pretraining it was used for NSP, so it learned to aggregate whole-sentence information — perfect for a sentence-level decision.
Common mistake "BERT can generate text like GPT."Feels right because it's a big Transformer trained on text. Fix: BERT is encoder-only and predicts masked slots in fixed-length input with bidirectional context — there is no autoregressive left-to-right generation loop. Use GPT/T5 for generation.
Common mistake "You mask 15% ofwords in the loss and predict them left-to-right."
Fix: you mask 15% of tokens (sub-word WordPieces), predict them all at once from full context, and only 80% actually become [MASK].
[CLS]'s embedding is meaningful before fine-tuning."
Fix: its usefulness comes from NSP/fine-tuning training pressure. A raw untrained pooled vector is not automatically a good sentence embedding (this is why sentence-BERT exists).
Recall Test yourself (hidden — answer first!)
Why can't BERT use next-token prediction? → bidirectional attention would let it see the answer.
What is the 80/10/10 rule for? → prevent [MASK]-only train/test mismatch.
Which objective did RoBERTa drop? → NSP.
Which token gives the sentence summary? → [CLS].
Recall Feynman: explain to a 12-year-oldImagine a sentence with some words hidden by stickers. A really smart kid reads everything around each sticker — words before AND after — and guesses the hidden word. To make sure the kid actually reads instead of memorizing where stickers go, sometimes we swap in a wrong word instead of a sticker, and sometimes leave the real word. After lots of practice guessing, the kid understands sentences so well that you can quickly teach them small new jobs — like "is this review happy or sad?" — with just a few examples. That smart reader is BERT.
"BERT Bidirectionally Masks Before Beginning" — B idirectional, M asked LM (15%, 80/10/10), then B egin fine-tuning with [CLS]
What kind of Transformer is BERT (encoder/decoder)? Encoder-only, bidirectional attention.
Why can't BERT be trained with next-token prediction? Bidirectional attention would let each token attend to the future token it must predict — trivial cheating.
What fraction of tokens are selected in MLM? 15%.
Describe the 80/10/10 rule. Of selected tokens: 80% → [MASK], 10% → random token, 10% → unchanged.
Why keep 10% unchanged / 10% random instead of always [MASK]? [MASK] never appears at inference; forcing uncertainty about every token gives good representations for all tokens (avoids train/test mismatch).
Write the MLM loss. − 1 ∣ M ∣ ∑ i ∈ M log p ( y i ∣ context ) -\frac{1}{|M|}\sum_{i\in M}\log p(y_i\mid\text{context}) − ∣ M ∣ 1 ∑ i ∈ M log p ( y i ∣ context ) , cross-entropy over masked positions only.
What is NSP and is it necessary? Binary "is B the next sentence after A"; RoBERTa showed it's often unnecessary/harmful.
What are the three summed input embeddings in BERT? Token + position + segment embeddings.
Why are position embeddings needed? Self-attention is permutation-invariant; positions inject word order.
What is the [CLS] token used for? Aggregated sentence-level representation for classification (used in NSP and fine-tuning).
How do you fine-tune BERT for sentiment? Add a linear+softmax head on
h [ C L S ] h_{[CLS]} h [ C L S ] and train on labelled data.
How is token classification (NER) done? Apply a shared linear layer to each token's
h i h_i h i .
Transformer Architecture — BERT stacks its encoder blocks.Self-Attention — bidirectional (unmasked) version powers BERT.GPT and decoder models — the generative, causal-masked counterpart.T5 and encoder-decoder models — text-to-text unifier.Cross-Entropy Loss — MLM and NSP objectives.Fine-Tuning vs Pretraining — the two-stage paradigm.WordPiece Tokenization — how BERT splits text into sub-word tokens.RoBERTa — BERT trained better, drops NSP.
Language is bidirectional
Masked Language Modelling
Intuition Hinglish mein samjhoDekho, BERT ka core idea simple hai: kisi word ka matlab samajhne ke liye tumhe uske dono taraf ka context chahiye — left bhi aur right bhi. "Main bank gaya paise nikalne" me bank ka matlab aage ke words se clear hota hai. GPT jaise decoder models sirf left-to-right dekhte hain, lekin BERT ek encoder-only model hai jisme attention bidirectional hoti hai — har token poore sentence ko dekh sakta hai.
Problem yeh hai: agar model poora sentence dekh sakta hai aur usse next word predict karne ko bolo, to woh cheat karega (answer to saamne hai!). Isliye BERT Masked Language Modelling use karta hai — 15% tokens ko chhupa deta hai aur unhe predict karwata hai. Ek trick hai 80/10/10 : 80% ko [MASK] se replace, 10% random word, 10% same rehne do. Yeh isliye ki [MASK] token real use me kabhi aata nahi, to model har token ka accha representation banana seekhe, sirf [MASK] ka nahi.
Ek doosra objective NSP (Next Sentence Prediction) tha, par baad me RoBERTa ne dikhaya ki yeh zaroori nahi — asli kaam MLM karta hai. Loss dono me cross-entropy hi hai, bas MLM me sirf masked positions par grade karte hain.
Kaam ka matlab: ek baar mehnga pretraining karo, phir chhota sa linear head laga ke kisi bhi task (sentiment, NER, QA) par thodi si training se fine-tune kar lo. [CLS] token poore sentence ka summary deta hai — usse classification kar lo. Yaad rakho: BERT samajhne ke liye hai, text generate karne ke liye nahi.
