Exercises — Masked language modeling (BERT)

• $\mathbf{x}=(x_1,\dots ,x_n)$ ::: the sequence of tokens (word pieces) fed to the model, position $1$ to $n$.
• $M$ ::: the set of masked positions — the indices we hid and asked the model to guess.
• ${\mathbf{x}}_{\backslash M}$ ::: the corrupted input — the sequence with the chosen positions replaced by [MASK] / random / unchanged.
• $P(x_i\mid {\mathbf{x}}_{\backslash M})$ ::: the probability the model assigns to the true token at position $i$, given everything it can see.
• ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$ ::: the training loss — smaller means better guesses.
• $\log$ here is the natural log (base $e$), the default in ML. $-\log (p)$ is big when $p$ is small (bad guess) and $0$ when $p=1$ (perfect guess).

Of the selected tokens: ==80% → [MASK], 10% → random token, 10% → unchanged==. These are fractions of the 15% selected, not of the whole sequence. So of the whole sequence: $0.15\times 0.80=12\%$ become [MASK], $0.15\times 0.10=1.5\%$ random, $0.15\times 0.10=1.5\%$ unchanged.

• GPT ::: causal (left-to-right only) — cannot see future tokens.
• BERT ::: bidirectional — every position attends to all others.
• T5 encoder ::: bidirectional. Only GPT is unidirectional. GPT must stay causal because it generates one token at a time; seeing the future would be cheating.

${\mathcal{L}}_{\text{BERT}}={\mathcal{L}}_{\text{MLM}}+{\mathcal{L}}_{\text{NSP}}$ ${\mathcal{L}}_{\text{MLM}}$ = Masked Language Modeling loss (predict hidden tokens). ${\mathcal{L}}_{\text{NSP}}$ = Next Sentence Prediction loss (does sentence B follow A?).

Selected: $0.15\times 40=6$ tokens. Literal [MASK]: $0.80\times 6=4.8\approx 5$ tokens. Random: $0.10\times 6=0.6\approx 1$. Unchanged: $0.6\approx 1$. (In practice the split is stochastic per token, so these are expectations.)

${\mathcal{L}}_{\text{MLM}}=-\log (0.2)-\log (0.5)$ $-\log (0.2)=\log 5\approx 1.6094$, and $-\log (0.5)=\log 2\approx 0.6931$. ${\mathcal{L}}_{\text{MLM}}\approx 1.6094+0.6931=2.3026$ What this means: the position where the model was only $20\%$ confident contributes more loss ($1.61$) than the $50\%$ one ($0.69$) — bad guesses are penalised harder, which is the whole point of $-\log$. See the loss curve below.

The contextual hidden state at position $2$ is ${\mathbf{h}}_2$ (the Transformer's output vector there). We project it to vocabulary scores and softmax: $P(x_2=\text{"dog"}\mid {\mathbf{x}}_{\backslash M})=\text{softmax}(\mathbf{W}{\mathbf{h}}_2+\mathbf{b})[\text{dog\_id}]$ Loss contribution: $-\log P(x_2=\text{"dog"}\mid {\mathbf{x}}_{\backslash M})$. Only masked positions contribute — positions $0,1,3,4,5,6$ add nothing to the loss even though the model produces hidden states for them.

Why not 100% [MASK]: the [MASK] token never appears during fine-tuning (downstream tasks feed clean text). If the model only ever learned to fill [MASK] slots, it would face a train–test mismatch and its representations of ordinary tokens would be under-trained. Random 10%: forces the model to not blindly trust the token it sees — even a present-looking token might be wrong, so the model must cross-check against context. This builds error-correcting representations. Unchanged 10%: forces the model to produce a good representation of the correct token as well, so that a token being "kept" is meaningful rather than a signal of "this one is safe to ignore." Because the model can't tell which of the three cases it's in, it must build a strong contextual representation for every selected position.

Student B. The barrier is the autoregressive generation objective, not the age of the design. GPT predicts the next token; if it could attend to future tokens, position $i$ would literally see token $x_{i+1}$ — the very thing it must predict — so training would be trivial and generation impossible. BERT can be bidirectional precisely because it is not generating: it corrupts input and reconstructs it, so "seeing both sides" of a masked slot is fair — the answer at that slot was hidden, not leaked.

Uniform means $P(x_i\mid \cdot )=\frac{1}{30000}$ for every token, including the true one. $-\log \left(\frac{1}{30000}\right)=\log (30000)\approx 10.309$ This is the loss of a model that guesses randomly — the maximum-ignorance baseline (in natural log units, "nats"). Any trained model should sit well below $10.31$. It equals the entropy of a uniform choice among $30000$ options: the number of nats needed to pin down one token with no help from context.

Sentence 1: $-\log 0.5-\log 0.25-\log 0.125=\log 2+\log 4+\log 8=0.6931+1.3863+2.0794=4.1589$. (Note $0.5,0.25,0.125=2^{-1},2^{-2},2^{-3}$, so this is $(1+2+3)\log 2=6\log 2$.) Sentence 2: $-\log 0.5-\log 0.5=2\log 2=1.3863$. Total: $6\log 2+2\log 2=8\log 2\approx 5.5452$.

Input stream (segment A tokens then segment B tokens, separated and terminated by [SEP]):

[CLS] I went to the store [SEP] I bought some milk [SEP]

Segment embeddings: A for [CLS] I went to the store [SEP], B for I bought some milk [SEP].

• MLM term (over whatever tokens are masked in $M$): ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$.
• NSP term: the final hidden state of [CLS], call it ${\mathbf{h}}_{[\text{CLS}]}$, feeds a binary classifier. True label here is IsNext ($=1$): ${\mathcal{L}}_{\text{NSP}}=-\log P(\text{label}=\text{IsNext}\mid {\mathbf{h}}_{[\text{CLS}]}).$ Total: ${\mathcal{L}}_{\text{BERT}}={\mathcal{L}}_{\text{MLM}}+{\mathcal{L}}_{\text{NSP}}$.

${\mathcal{L}}_{\text{NSP}}=-\log (0.9)\approx 0.10536$. ${\mathcal{L}}_{\text{MLM}}=-\log (0.4)-\log (0.6)-\log (0.8)\approx 0.91629+0.51083+0.22314=1.65026$. ${\mathcal{L}}_{\text{BERT}}\approx 1.65026+0.10536=1.75562$

${\mathcal{L}}_X=-\log (0.9)\approx 0.10536$. ${\mathcal{L}}_Y=-\log (0.99)\approx 0.01005$. Ratio ${\mathcal{L}}_X/{\mathcal{L}}_Y\approx 0.10536/0.01005\approx 10.48$. Why it matters: $-\log$ is steep near $p=1$, so tiny gains in already-high probability slash the loss by an order of magnitude. The loss curve (below) is nearly vertical as $p\to 0$ and flattens toward $0$ as $p\to 1$ — this is why late-stage training keeps improving loss even when accuracy looks saturated.

With $M=\varnothing$ (empty set), the sum has no terms: ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in \varnothing }\log P(\cdot )=0.$ A zero loss has zero gradient from the MLM head, so this example contributes no MLM update. (The NSP term, if present, can still contribute.) This is the correct, safe behaviour — an empty sum is $0$, not undefined. Practically, implementations guard against empty $M$ or ensure at least one token is masked.

$P$ is a probability, so $P\in (0,1]$ (it can be $1$ but never exactly $0$ in floating point / after softmax).

• As $P\to 1$: $-\log P\to -\log 1=0$. Lower bound $0$, achieved at perfect confidence in the truth.
• As $P\to 0^{+}$: $-\log P\to +\infty$. No upper bound — the more confidently wrong the model is, the more brutal the penalty. Why it teaches well: the loss can never reward a mistake (it's $\ge 0$) and it punishes confident errors without limit, so the model is strongly discouraged from putting near-all probability on a wrong token. The gentle floor near $p=1$ means correct-and-confident predictions stop generating pressure — training energy flows to the still-wrong positions.