Masked language modeling (BERT): Question bank

This page is a misconception gym for Masked language modeling (BERT). Every item below hides a trap that people fall into when they first meet BERT. Read the question, say your answer out loud, then reveal.

Before we start, some words and symbols that MUST be crystal clear (everything else builds on them):

Now the symbols in the loss formula, so nothing below is a surprise:

Definition Reading the MLM loss symbols

$x = (x_{1}, x_{2}, \dots, x_{n})$ — the sequence of $n$ tokens; $x_{i}$ is the token (the actual word) sitting at position $i$ .
$M$ — the set of masked positions, e.g. $M = {3, 5}$ means positions 3 and 5 were hidden. " $i \in M$ " reads "for each hidden position $i$ ."
$x_{\ M}$ — the corrupted input: the whole sequence after masking, i.e. with the tokens at positions in $M$ replaced by [MASK] (or corrupted). The little " $\ M$ " is shorthand for "with the true words at $M$ removed from view."
$P (x_{i} ∣ x_{\ M})$ — the model's probability that the hidden word at position $i$ was really $x_{i}$ , given the corrupted input it can see.
$V$ — the vocabulary: the full list of every token the model knows.
$h_{i}$ — the contextual hidden vector the Transformer produces at position $i$ (its rich, context-aware summary of that slot).
$w_{v}$ — the output vector for vocabulary word $v$ ; $w_{v}^{⊤} h_{i}$ scores how well word $v$ matches slot $i$ .

Formula The MLM loss (and WHY it looks like this)

$L_{MLM} = - \sum_{i \in M} lo g P (x_{i} ∣ x_{\ M})$ Read it as "add up, over every hidden position, the log-probability the model gives the true word — then flip the sign."

WHY the $lo g$ ? Predictions multiply (a product of tiny probabilities), which underflows to zero; $lo g$ turns that product into a stable sum, and since $lo g$ is monotonic it doesn't move the maximum.
WHY the minus sign? Training uses gradient descent, which minimizes. Maximizing log-likelihood is the same as minimizing its negative, so we negate.

Each probability is a ==softmax over the whole vocabulary $V$ ==: $P (x_{i} ∣ x_{\ M}) = \frac{e x p ( w _{x_{i}}^{⊤} h _{i} )}{\sum _{v \in V} e x p ( w _{v}^{⊤} h _{i} )}$ The denominator sums the scores of every word $v$ so the numbers add to 1 — that's what makes it a probability.

Keep those pictures in mind — most traps here are just one of these ideas being misremembered.

True or false — justify

BERT is a language model that can generate text left-to-right like GPT.

False. BERT is trained to fill in blanks using both sides of context, not to continue a sentence. It has no built-in notion of "the next word given only the left," so it is poor at free generation.

Masking 100% of selected tokens with [MASK] would give the strongest training signal.

False. The [MASK] symbol never appears at fine-tuning time, so 100% masking creates a train–test mismatch; the 80/10/10 split forces the model to stay useful even when it sees real (unmasked) tokens.

Increasing the mask rate from 15% to 50% will always help because the model gets more prediction targets.

False. Too many masks strips away the context needed to predict any of them, so the puzzle becomes near-impossible and the signal degrades — 15% balances "enough targets" against "enough surviving context."

The MLM loss is summed only over the masked positions, not over every token in the sequence.

True. Loss is

- \sum_{i \in M} lo g P (x_{i} ∣ x_{\ M})

where

M

is the masked set; unmasked positions contribute no prediction loss because their true token is already visible.

Because attention is bidirectional, a masked word can attend to itself and read off its own answer.

False. At a masked position the input embedding is the [MASK] token, not the original word, so there is nothing to copy — the model must reconstruct the answer from neighbours.

BERT's position embeddings are the fixed sine/cosine curves from the original Transformer paper.

False. BERT uses learned position embeddings (a lookup table indexed by position), not the sinusoidal formula.

The 10% "keep unchanged" tokens are still counted in the loss even though they look normal to the model.

True. The model is asked to predict the original at those positions too; since it cannot tell them apart from ordinary tokens, it must learn to represent every token robustly.

Removing NSP (Next Sentence Prediction) from pre-training destroys BERT's ability to do the MLM task.

False. MLM and NSP are independent objectives added together; MLM works fine alone (later models like RoBERTa dropped NSP entirely and improved).

The softmax in the output layer runs over the masked positions.

False. The softmax runs over the entire vocabulary $V$ for each masked position — it answers "which word out of all words," not "which position."

Spot the error

"GPT can't use bidirectional attention simply because it's smaller than BERT."

The error is the reason. Model size is irrelevant; GPT is autoregressive, so letting it see the future would let it copy the very word it's supposed to predict — that's cheating, not a capacity limit.

"We take the log of the probability product mainly to make the numbers bigger."

Wrong motive. We take the log to turn a product of tiny probabilities into a sum, avoiding numerical underflow — and because log is monotonic, maximizing log-likelihood is the same as maximizing likelihood.

"The MLM loss is negative because probabilities are negative."

Probabilities are never negative. The minus sign is bolted on so that gradient descent (which minimizes) achieves the same thing as maximizing the log-likelihood.

"The [CLS] token is where masked-word predictions are read out."

No. Masked-word predictions come from each masked position's hidden state

h_{i}

. The [CLS] hidden state feeds the NSP (Next Sentence Prediction) sentence-pair classifier, a different head.

"BERT is a decoder-only Transformer."

BERT is encoder-only. There is no decoder because it never generates a sequence autoregressively — it produces contextual representations.

"Conditional independence of masked predictions means the masked tokens are genuinely independent in language."

The independence is a modeling assumption used to factor the loss as a product, not a claim about real language. In truth masked words can be correlated; the assumption is a simplification that makes the loss tractable.

"Segment embeddings tell BERT the word's meaning."

Segment embeddings only encode which sentence (A or B) a token belongs to — needed for sentence-pair tasks. Meaning comes from token embeddings plus contextual attention.

Why questions

Why does MLM need both directions of context, while next-word prediction only needs the left?

Filling a blank in the middle of a sentence uses evidence on both sides ("The ___ was steep" vs "The ___ was closed"); predicting the next word by definition only has the past available, so looking right would be seeing the answer.

Why replace 10% of masked positions with a random wrong token instead of always [MASK]?

To teach the model that any token it reads might be corrupted, so it must build a robust representation of every position rather than trusting inputs blindly — this narrows the gap to fine-tuning where inputs are all "real."

Why is the MLM objective called a "denoising autoencoder"?

We corrupt (add noise to) the input by masking, then train the model to reconstruct the clean original — corrupt-then-restore is exactly the denoising-autoencoder pattern.

Why is [MASK] a special reserved token rather than just an existing word like "blank"?

A reserved symbol is guaranteed never to be a real answer, so the model can unambiguously recognise "prediction needed here"; reusing a real word would confuse "hidden slot" with "the actual word blank."

Why can BERT afford bidirectional attention but GPT cannot?

BERT is used for understanding (classification, QA) where we never generate token-by-token, so seeing everything is safe; GPT generates, and seeing future tokens during training would let it cheat at exactly the task it's deployed for.

Why sum the per-token log-probabilities rather than multiply the probabilities directly?

Multiplying many probabilities below 1 collapses toward zero (floating-point underflow); logs convert the product into a numerically stable sum while preserving where the maximum lies.

Edge cases

If a training sequence has 0 masked tokens (rare with 15% and short input), what is the MLM loss?

The sum over an empty masked set

M

is 0, so that sequence contributes no MLM gradient — it's effectively skipped for MLM (though NSP may still apply).

What happens at a masked position that also happens to be at the very end of the sentence, next to [SEP]?

Nothing special — bidirectional attention still gives it the entire left context plus the [SEP] marker; there is no "right neighbour required" rule, so end positions are handled identically.

What happens when two adjacent positions are both masked, e.g. [CLS] the [MASK] [MASK] fox [SEP]?

Each hidden slot must be predicted independently (the loss assumes conditional independence given the corrupted input), so neither can lean on the other's true word — they share only the surrounding unmasked context, which makes adjacent masks genuinely harder.

A rare word is split by WordPiece into subword pieces (e.g. play ##ing) and only one piece gets masked — what does the model predict there?

It predicts the original subword piece, not the whole word; masking operates at the token (subword) level, so the visible sibling piece becomes strong context that helps reconstruct the hidden piece.

If the same word appears twice and only one instance is masked, does BERT get an easy free answer from the other copy?

Not automatically — it can attend to the unmasked copy and often should, which is a legitimate use of bidirectional context, not cheating; the answer still isn't visible at the masked slot itself.

For the "10% keep unchanged" case, how does the model know a prediction is even being asked of it?

It doesn't know from the input — the token looks ordinary. Supervision comes from the loss, which secretly targets those positions, forcing correct representations everywhere rather than only at obvious [MASK] slots.

What does the softmax denominator

\sum_{v \in V} exp (w_{v}^{⊤} h_{i})

do when the vocabulary

V

is enormous?

It sums over every vocabulary token to normalise the scores into probabilities; a huge vocab makes this the computational bottleneck, which is why later work explores approximations — but the definition still requires all of

V

Two sentences fed for NSP where B genuinely follows A but is topically unrelated (a valid but jarring continuation) — what label?

Label = IsNext (1). NSP asks only "did B follow A in the source document," not "is B topically coherent" — adjacency, not meaning, defines the label.

If you fine-tune BERT and never show it a [MASK] token again, is the pre-trained MLM head wasted?

The MLM output head is usually discarded at fine-tuning, but the encoder it trained is exactly what you keep — the point of pre-training was to shape those bidirectional representations, not the throwaway prediction layer.

Recall Quick self-test

The single most repeated trap here is thinking bidirectional attention lets a masked word "see itself." Say why it can't. Because the input at a masked slot is the [MASK] symbol, not the original word — there is no self to see. ::: Correct if you mentioned the input embedding being [MASK], not the true token.