Self-supervised pretraining objectives: Question bank

Prerequisite ideas leaned on here: Cross-entropy Loss, Perplexity, Tokenization (BPE), Transformer Architecture, GPT vs BERT, Fine-Tuning LLMs.

True or false — justify

Each claim below is a trap. Decide true/false and give the reasoning that makes it so.

"Self-supervised pretraining needs no data labels at all."

True — but only manual labels. A label still exists for every example: it's the hidden/removed piece of the input, generated automatically from the text itself.

"Because CLM only predicts the next token, it never learns grammar or facts."

False — to reliably guess the next token you must implicitly model grammar, coreference, and world facts; the trivial-looking task forces that knowledge into the weights.

"MLM and CLM optimize completely different loss functions."

False — both minimize cross-entropy (average negative log-prob of the correct token). They differ in which positions are scored and in directionality, not in the loss family.

"Minimizing cross-entropy and minimizing perplexity are two separate goals you must trade off."

False — they are the same objective since

PPL = e^{L_{CE}}

; the exponential is monotonic, so lowering one always lowers the other.

"A BERT model can generate fluent text left-to-right just like GPT if you sample its outputs."

False — MLM conditions on both sides, so it has no valid

p (x_{t} ∣ x_{< t})

factorization to sample from token-by-token; it isn't a proper autoregressive sampler.

"In MLM every output position produces a distribution, so every position trains the model."

False — the loss sums only over masked positions

M

. Scoring visible tokens is trivial (the answer is right there) and would dilute the learning signal.

"Causal masking is an optimization trick to make training faster."

False — it is a correctness requirement: without it a token could attend to its own future answer and cheat, so the loss would measure copying, not prediction.

"Span corruption is just MLM with more masks."

False — masking contiguous spans and generating them as an output sequence forces multi-token reasoning; scattered single-token masks are often solvable from local n-gram statistics.

Spot the error

Each line contains one flawed statement. Name the flaw and correct it.

"CLM loss is

L = - \frac{1}{T} \sum_{t} lo g p_{θ} (x_{t} ∣ x_{> t})

Wrong conditioning direction: it must be

x_{< t}

(past tokens), not

x_{> t}

. CLM is left-to-right, so a token conditions only on what came before it.

"BERT masks 15% of tokens and replaces all of them with the [MASK] symbol."

Only 80% of the chosen 15% become [MASK]; 10% become a random token and 10% stay unchanged, to avoid a train/test mismatch since [MASK] never appears at fine-tuning.

"To make MLM harder and learn more, we should raise the mask rate toward 100%."

With everything masked no context remains to condition on, making prediction unlearnable. ~15% balances enough visible context against task difficulty.

"Perplexity of a model that assigns probability 1 to every correct token is very high."

Backwards: perfect prediction gives loss

0

, so

PPL = e^{0} = 1

— the lowest possible. High perplexity means the model is confused, low means confident and correct.

"The chain-rule factorization

p (x_{1} .. x_{T}) = \prod_{t} p (x_{t} ∣ x_{< t})

is only an approximation for text."

It is exact for any joint distribution — no independence assumption is made. The approximation lives in

p_{θ}

, not in the factorization.

"MLM's cross-entropy is normalized by the sequence length

T

It is normalized by

∣ M ∣

, the number of masked positions, since only those contribute to the loss — not by the full length

T

"Because CLM sees no future, its loss can only be computed one position at a time."

False economy of thought — with a causal attention mask the whole sequence is processed in one forward pass and every position is scored simultaneously; the mask enforces left-to-right visibility without serial computation.

Why questions

"Why does removing information (masking) create a learning signal at all?"

The model can only recover the hidden piece by using the surrounding structure; the gap between its guess and the true removed token is exactly the gradient that teaches it that structure.

"Why sum the CLM loss over all positions but the MLM loss over only masked ones?"

In CLM the causal mask already hides each token's future, so scoring every position is legitimate. In MLM the model sees the whole (uncorrupted-elsewhere) sentence, so unmasked targets are visible and give no signal.

"Why is [MASK] a problem BERT has to actively counter, but GPT never needs?"

[MASK] is an artificial token absent at fine-tuning, so a BERT trained only on it suffers train/test mismatch — hence 80/10/10. GPT corrupts nothing structurally; it simply predicts real next tokens, so no fabricated symbol exists.

"Why does taking

- lo g p

of the correct token, rather than

p

itself, give the right loss?"

- lo g p

is the surprise: it blows up toward

\infty

as the correct token's probability drops toward

0

, and hits

0

when probability is

1

— so minimizing average surprise directly maximizes likelihood.

"Why can span corruption blend the strengths of BERT and GPT?"

The encoder reads the whole context bidirectionally (BERT-like understanding) while the decoder generates the missing spans left-to-right (GPT-like production), giving one model both comprehension and generation.

Edge cases

"What is the CLM loss on a single-token sequence?"

There is no preceding context for the first token, so with the standard

t \geq 2

convention there are no predictable positions and the per-sequence contribution is effectively empty; loss is defined only where a left-context exists.

"If MLM randomly selects a position but the 10% 'unchanged' branch fires, what does the model see?"

It sees the original token in place, yet must still predict it as a target — forcing a useful representation of ordinary (unmasked) tokens, not just a reflex to the [MASK] symbol.

"What perplexity does a uniform model over a vocabulary of size

V

achieve?"

Exactly

V

— it is "as unsure as picking uniformly among all

V

words," which is the worst-case baseline any trained model must beat.

"What happens to CLM if the sequence length

T

grows very large?"

Each new token conditions on a longer history

x_{< t}

; the loss stays an average over positions, but the model must carry information across the full context, which is exactly what the Transformer Architecture's attention is built to do.

"Under BPE tokenization, does 'one masked token' equal 'one masked word'?"

No — Tokenization (BPE) can split a word into several subword tokens, so masking one token may hide only a fragment of a word, and masking a span may cut through word boundaries.

"If you mask 0% of tokens in MLM, what does the model learn?"

Nothing useful — with no removed information the target is fully visible, the loss is trivially near zero, and there is no gradient signal to improve representations.

Recall One-line summary of the traps

The recurring theme ::: self-supervision manufactures labels by removing information, and every objective differs only in what is removed, which context is allowed, and which positions are scored — not in the underlying cross-entropy loss.