Question bank — BERT and encoder models

False — only the masked positions (the set $M$) are graded; visible tokens contribute nothing, since scoring a token the model can literally see teaches nothing and leaks the answer.

False — BERT is bidirectional with no causal mask; the whole point is that each token sees left AND right context. See GPT and decoder models for the causal variant.

False — RoBERTa showed NSP is often useless or slightly harmful; training MLM alone on more data does better, so NSP is optional and MLM is load-bearing.

False — its usefulness comes from NSP and fine-tuning training pressure; a raw untrained pooled vector $h_{\text{[CLS]}}$ is not automatically a good sentence summary (this is why sentence-BERT exists).

False — 15% of tokens are chosen, but of those only 80% become [MASK]; 10% become a random token and 10% are left unchanged.

False — it masks 15% of sub-word tokens (WordPiece pieces) and predicts them all at once from full context — no autoregressive loop.

True — softmax turns raw logits $z_i$ (linear scores $W{h}_i+b$) into non-negative numbers summing to 1, so we can ask "how probable is the true token $y_i$?" and then minimize its negative log (Cross-Entropy Loss).

False — self-attention is permutation-invariant (a bag of tokens), so "dog bites man" and "man bites dog" would be identical without an injected position vector.

False — pretraining is done once; fine-tuning adds a tiny head (often one linear layer) and trains briefly, because $h_i$ already encodes rich meaning (see Fine-Tuning vs Pretraining).

The average must be over $|M|$ (masked positions) not $n$; unmasked tokens are excluded entirely from the sum and the denominator.

[MASK] never appears at inference time; a [MASK]-only scheme creates a train/test mismatch, so 10% random + 10% unchanged force good representations for every token.

BERT is encoder-only with no left-to-right generation loop; it fills fixed-length masked slots. Use GPT and decoder models or T5 and encoder-decoder models for generation.

NSP predicts a binary IsNext / NotNext relation between two whole sentences from $h_{\text{[CLS]}}$; it is a sentence-pair classification, unrelated to word-level prediction.

Three embeddings are summed per token — token + position + segment (A vs B) — before entering the encoder stack.

With bidirectional attention the model can attend to the very token it must predict — it cheats. That's exactly why BERT needs the cheat-proof MLM objective instead.

[SEP] separates sentence A from sentence B inside one input; it has nothing to do with which loss (MLM vs NSP) is computed.

Because the model can never be sure whether a given token is corrupt, it is forced to build a good contextual representation $h_i$ for every token — that uncertainty is the training signal, not a side effect.

During pretraining [CLS] was fed to NSP, so its vector $h_{\text{[CLS]}}$ learned to aggregate whole-sentence information; fine-tuning then reuses that summarizing habit for a single sentence-level decision.

MLM only conditions on within-sentence context; tasks like QA or entailment need to compare two sentences, which is why NSP (or its replacements) was added to expose cross-sentence signal.

$-\log$ is a monotonically decreasing function, so making $p(y_i)$ larger makes $-\log p(y_i)$ smaller; minimizing the loss and maximizing the true-token probability are the identical optimization.

MLM forces it to model general bidirectional context across a huge corpus; that broad linguistic knowledge lives in $h_i$, so downstream heads only learn a small task-specific mapping on top.

The set $M$ is empty, so ${\mathcal{L}}_{\text{MLM}}$ is undefined/zero for that example — nothing is graded; in practice at least one token is always masked so this is avoided.

Yes — it is still in $M$, so the model must predict it correctly from context even though the input looks identical; only the input corruption differs, not the loss membership.

There is only segment A, so every token gets the same segment-A vector — it adds a constant and effectively drops out, which is fine; segment embeddings only matter for two-sentence inputs.

The label $y$ is still NotNext because it wasn't the actual following sentence; NSP grades true adjacency, not topical similarity, which is part of why the signal is weak enough for RoBERTa to drop it.

Zero — masking only happens during MLM pretraining; at fine-tuning/inference the real tokens are all present, which is exactly the mismatch the 80/10/10 rule protects against.

There is no learned position embedding for those extra slots, so order information beyond the max length is undefined — the model can't place those tokens correctly.