Noesis
4.3.5Pretraining & Fine-Tuning LLMs

Self-supervised pretraining objectives

1,874 words9 min readdifficulty · medium
🇮🇳 Yeh note Hinglish mein padho →

WHY do we need self-supervised objectives?

WHAT problem are we solving? We want a model that "understands" language, but labeled data (sentiment, translation, etc.) is scarce and expensive. Raw text (web, books) is nearly infinite.

WHY self-supervision works: to predict a missing word correctly you must implicitly learn grammar, facts, coreference, arithmetic, world knowledge. So a trivial-sounding task (guess the next word) forces the model to compress a huge amount of structure into its weights.

The main objectives (the 80/20 core)

1. Causal / Autoregressive Language Modeling (CLM) — GPT-style

WHAT: Predict the next token given all previous ones (left-to-right).

HOW we derive the loss (from first principles): The probability of a whole sequence x1,…,xTx_1,\dots,x_T can always be factorized by the chain rule of probability: pθ(x1,…,xT)=∏t=1Tpθ(xt∣x1,…,xt−1)p_\theta(x_1,\dots,x_T)=\prod_{t=1}^{T} p_\theta(x_t\mid x_{1},\dots,x_{t-1})

Why this step? No assumption made — the chain rule is exact for any joint distribution.

To train, we maximize log-likelihood. Turning "maximize log-prob" into a "minimize loss" (multiply by −1-1) and averaging over positions gives:

Why cross-entropy? Because −log⁡pθ(xt∣⋅)-\log p_\theta(x_t\mid\cdot) is the surprise of the correct token; minimizing average surprise = maximizing likelihood. The perplexity PPL=eLCLM\text{PPL}=e^{\mathcal L_{\text{CLM}}} is just the exponentiated loss.

Key trick — causal masking: a token may only attend to positions ≤t\le t, otherwise it could "peek" at the answer and cheat.

2. Masked Language Modeling (MLM) — BERT-style

WHAT: Randomly hide ~15% of tokens with a [MASK] symbol; predict the originals using both left and right context (bidirectional).

HOW: Let MM be the set of masked positions. Corrupt x→x~x\to\tilde x by replacing xix_i (i∈Mi\in M) with [MASK]. Predict only the masked ones:

Why only sum over MM? Non-masked tokens are visible, so predicting them is trivial and gives no learning signal.

Why is BERT's real recipe 80/10/10? Of the chosen 15%: 80% → [MASK], 10% → a random token, 10% → left unchanged.

  • Why? At fine-tuning time there is no [MASK] token, so a model trained only on [MASK] would see a train/test mismatch. The random/unchanged cases force it to build a useful representation of every token, not just react to the mask symbol.

3. Span Corruption / Denoising (T5, BART)

WHAT: Mask contiguous spans and make the decoder generate the missing spans as a short output sequence. Combines bidirectional encoding with generation.

WHY spans not single tokens? Predicting one masked token is often solvable from local n-gram statistics; masking a whole span forces genuine multi-token reasoning.

Figure — Self-supervised pretraining objectives

Worked examples

Forecast-then-Verify

Common mistakes (Steel-manned)

Mnemonic

Feynman

Recall Explain to a 12-year-old

Imagine reading a story where some words are hidden with stickers. To fill them in, you have to understand the story — who's doing what, what makes sense. A computer plays this fill-in game millions of times on the whole internet. Nobody has to write down answers, because the answers are just the words that were under the stickers! By getting really good at guessing hidden words, the computer accidentally learns grammar, facts, and even a bit of reasoning.

Flashcards

What makes an objective "self-supervised"?
The labels are derived automatically from the input itself (e.g. hidden words), so no human annotation is needed.
State the causal LM loss.
L=−1T∑tlog⁡pθ(xt∣x<t)\mathcal L=-\frac1T\sum_t \log p_\theta(x_t\mid x_{<t}) — average negative log-prob of each next token.
Which probability rule justifies the CLM factorization?
The chain rule of probability: p(x1..xT)=∏tp(xt∣x<t)p(x_1..x_T)=\prod_t p(x_t\mid x_{<t}), exact for any joint distribution.
Relationship between cross-entropy loss and perplexity?
PPL=eLCE\text{PPL}=e^{\mathcal L_{CE}}.
Why does CLM use a causal attention mask?
So a token can't attend to future tokens and 'peek' at its own answer.
In MLM, which positions contribute to the loss?
Only the masked positions MM.
Explain BERT's 80/10/10 masking split and its purpose.
Of the 15% chosen tokens: 80% become [MASK], 10% random token, 10% unchanged — to avoid train/test mismatch since [MASK] never appears at fine-tuning.
Why is CLM better suited for generation than MLM?
Its factorization matches left-to-right sampling; MLM is bidirectional and can't be sampled autoregressively.
Why not mask 100% of tokens?
No context remains to condition on, making prediction unlearnable.
What is span corruption (T5/BART)?
Masking contiguous spans and generating the missing spans, forcing multi-token reasoning while keeping bidirectional encoding.

Connections

  • Transformer Architecture — causal vs bidirectional attention masks implement these objectives.
  • Cross-entropy Loss — the shared underlying loss function.
  • Perplexity — evaluation metric = exp(loss).
  • Fine-Tuning LLMs — pretrained weights are the starting point.
  • Tokenization (BPE) — defines the vocabulary over which softmax is computed.
  • GPT vs BERT — CLM vs MLM design tradeoffs.

Concept Map

enables

defined by

forces model to learn

derived from

main objective

main objective

predicts next token

predicts masked token

prevents peeking via

corrupts input by

exponentiated to

Raw text is free supervision

Self-supervised pretraining

Labels derived from input

Chain rule factorization

Causal LM - GPT

Masked LM - BERT

Mask ~15% tokens

Causal masking

Cross-entropy loss

Perplexity

Grammar facts world knowledge

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, self-supervised pretraining ka core idea bahut simple hai: internet pe itna saara text pada hai, par labels (jaise sentiment, translation) banane bahut mehenge hai. To hum ek chalaki karte hai — text ka hi kuch hissa chhupa do, aur model se guess karwao. Jo shabd chhupaya tha, wahi answer ban gaya! Isliye "self"-supervised — apne aap se hi supervision mil gaya, kisi insaan ne label nahi likha.

Do main flavours hai. Causal LM (GPT) me model sirf peeche ke shabd dekh kar agla shabd predict karta hai, left-to-right. Iska loss chain rule se nikalta hai — poori sentence ki probability ko chhote-chhote "next word" probabilities me tod do, aur cross-entropy minimize karo. Yeh generation ke liye perfect hai kyunki hum bhi token-by-token hi text banate hai. Masked LM (BERT) me kuch tokens [MASK] se dhak dete hai aur dono taraf (left + right) ka context use karke fill karte hai. Yeh generation ke liye seedha kaam nahi aata, par understanding tasks me strong hota hai.

Ek important baat: BERT me 15% tokens choose karke unme se 80% ko [MASK], 10% ko random word, aur 10% ko waisa hi rakhte hai. Kyun? Kyunki fine-tuning ke time [MASK] token hota hi nahi, to model ko sikhna padta hai har token ko samajhna, sirf mask symbol pe react karna nahi. Aur loss sirf masked positions pe lagta hai — jo dikh raha hai usko predict karna to trivial hai, usse kuch nahi seekhega.

Yaad rakhne ki cheez: cross-entropy loss aur perplexity same cheez hai, bas PPL=elossPPL=e^{loss}. Aur 100% mask kabhi mat karo — agar sab kuch chhupa doge to context bachega hi nahi aur model kuch guess nahi kar payega. Balance chahiye: itna chhupao ki task mushkil ho par solvable rahe.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 1,957 wordsD2Visual walkthroughThe derivation in pictures · 1,707 wordsD3Worked examplesEvery scenario covered · 2,272 wordsD4ExercisesGraded, full solutions · 2,278 wordsD5Question bankConcept traps, quick reveals · 1,300 words

Test yourself — Pretraining & Fine-Tuning LLMs

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Log-loss - binary cross-entropyAI-ML · 2.2.10