Worked examples — Masked language modeling (BERT)

This page drills Masked language modeling (BERT) through concrete numbers. We attack every case the masking objective and its softmax loss can throw at you: normal masking, the two "no-[MASK]" corruption cases, degenerate zero-mask sequences, the limiting best/worst-case loss, a word problem, and an exam twist that mixes MLM with NSP.

Before we start, one symbol we will lean on constantly, built from zero:

---

The scenario matrix

Every worked example below is tagged with the cell it fills. Together they cover the whole grid.

Cell	What makes it tricky	Covered by
A. Standard 80% [MASK]	ordinary masked-token prediction	Ex 1
B. 10% random-token swap	input is wrong word, target is original	Ex 2
C. 10% keep-unchanged	input already correct, still must predict	Ex 3
D. Zero / degenerate mask	$M=\varnothing$: what is the loss?	Ex 4
E. Limiting values	perfect ($p=1$) vs uniform-guess ($p=1/|\mathcal{V}|$)	Ex 5
F. Multi-token sum	several masked slots added together	Ex 6
G. Real-world word problem	count masked tokens in a real corpus	Ex 7
H. Exam twist	combine MLM loss with NSP loss	Ex 8

---

Ex 1 — Cell A · The ordinary 80% [MASK]

Forecast: cat has the top score, so guess $P(\text{cat})$ around $0.5$ and a modest loss below $1$.

1. Exponentiate every score. Why? Softmax needs positive weights. $e^1=2.7183,e^2=7.3891,e^0=1.0000,e^1=\mathrm{2.7183.}$
2. Add them to get the normaliser. Why? The denominator makes the list sum to $1$. $Z=2.7183+7.3891+1.0000+2.7183=\mathrm{13.8257.}$
3. Divide cat's weight by $Z$. Why? That is the definition of $P(\text{cat})$. $P(\text{cat})=\frac{7.3891}{13.8257}=\mathrm{0.5345.}$
4. Take negative log. Why? That is the per-slot MLM loss. $-\log (0.5345)=\mathrm{0.6264.}$

Verify: All four probabilities are $0.1966,0.5345,0.0723,0.1966$ and sum to $1.000$ ✓. Loss $0.63$ is well under $\log 4=1.386$ (the clueless value), so the model is genuinely confident — consistent with cat having the highest score.

---

Ex 2 — Cell B · The 10% random-token swap

Forecast: the input literally shows mat, so the model is tempted by mat (highest score) but must still say cat — expect a larger loss than Ex 1.

1. Note the target is the clean token. Why? Cell B's whole point is denoising: input is noisy, label is original cat. This is the train–test-robustness trick.
2. Exponentiate: $e^{0.5}=1.6487,e^{1.5}=4.4817,e^{2.0}=7.3891,e^{0.5}=1.6487$.
3. Normaliser: $Z=1.6487+4.4817+7.3891+1.6487=15.1682$. Why? Same softmax rule.
4. Probability of the true word cat: $P(\text{cat})=\frac{4.4817}{15.1682}=0.2955$.
5. Loss: $-\log (0.2955)=1.2192$.

Verify: Probabilities $0.1087,0.2955,0.4872,0.1087$ sum to $1.000$ ✓. Loss $1.22>0.63$ from Ex 1 — correct, because the corrupted input actively misleads the model, so the same objective is harder here. This is exactly why the 10% swap forces robustness.

---

Ex 3 — Cell C · The 10% keep-unchanged

Forecast: input already shows the right word and score for cat is high — expect a small loss.

1. Understand the branch. Why? Cell C stops the model from assuming "a position is masked ⇒ it's wrong". The word is unchanged but still a labelled prediction target.
2. Exponentiate: $e^3=20.0855$; three copies of $e^0=1$.
3. Normaliser: $Z=20.0855+1+1+1=23.0855$.
4. Probability: $P(\text{cat})=\frac{20.0855}{23.0855}=0.8701$.
5. Loss: $-\log (0.8701)=0.1391$.

Verify: Sum $=0.8701+3(0.0433)=1.000$ ✓. Loss $0.14$ — the smallest of the three corruption cells, matching intuition: when the correct token is already visible and highly scored, the model is barely surprised.

---

Ex 4 — Cell D · Degenerate: zero masked tokens

Forecast: no masked slots means nothing to predict — guess loss $=0$.

1. Write the loss as a sum over $M$. Why? The objective ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$ only ranges over masked positions.
2. Empty sum rule. Why? A sum over the empty set is $0$ by definition. ${\mathcal{L}}_{\text{MLM}}={\sum }_{i\in \varnothing }(\cdots )=0.$
3. Gradient consequence. Why? ${\nabla }_{\theta }0=0$, so this sequence contributes no weight update.

Verify: With $|M|=\lfloor 0.15\times 4\rfloor =0$, there is indeed no term. In practice BERT guarantees at least one masked token per sequence precisely to avoid wasting a training example on a zero-loss forward pass. $0$ ✓.

---

Ex 5 — Cell E · Limiting values (best and worst case)

Forecast: best case loss $=0$; worst case is a positive number around $10$ish (since $\log 30000\approx 10$).

1. Best case. Why? $p=1$ means zero surprise. $-\log 1=0$.
2. Worst case — uniform. Why? If every word is equally likely, $p=1/|\mathcal{V}|$, and $-\log \left(\frac{1}{30000}\right)=\log 30000=\mathrm{10.3090.}$
3. Interpret the range. Why? Any real MLM loss lives in $[0,\log |\mathcal{V}|]$. Early training sits near $10.3$; a converged BERT drops toward $1$–$2$.

Verify: $-\log 1=0$ ✓ and $\log 30000=10.309$ ✓. This is the theoretical ceiling — if you ever log an MLM loss above $\log (\text{vocab})$, you have a bug (e.g. labels misaligned).

---

Ex 6 — Cell F · Summing several masked slots

Forecast: two positive contributions added; each is under $2$, so total under $4$.

1. Per-slot losses. Why? Each masked slot contributes $-\log p$ independently (the conditional-independence assumption in the parent derivation). $-\log (0.40)=0.9163,-\log (0.20)=\mathrm{1.6094.}$
2. Sum them. Why? $\mathcal{L}=-{\sum }_{i\in M}\log P(x_i\mid \cdots )$ literally adds over $M=\{3,5\}$. $\mathcal{L}=0.9163+1.6094=\mathrm{2.5257.}$

Verify: Equivalent to $-\log (0.40\times 0.20)=-\log (0.08)=2.5257$ ✓ — confirming the "product-inside becomes sum-outside" log identity from the parent's step 2.

---

Ex 7 — Cell G · Real-world word problem

Forecast: $15\%$ of 2B is $300$M targets; $80\%$ of those are [MASK] $\approx 240$M.

1. Total masked targets. Why? The protocol selects $15\%$ of all tokens as prediction targets. $0.15\times 2,000,000,000=300,000,000.$
2. Of those, actual [MASK] symbols. Why? Only the $80\%$ branch writes the literal [MASK] token. $0.80\times 300,000,000=240,000,000.$
3. Sanity on the other branches. Why? Completeness — the remaining $60$M split as $10\%$ random ($30$M) + $10\%$ unchanged ($30$M).

Verify: $240\text{M}+30\text{M}+30\text{M}=300\text{M}$ = total targets ✓, and $300\text{M}$ is exactly $15\%$ of 2B ✓. Units: all counts are dimensionless token counts, consistent throughout.

---

Ex 8 — Cell H · Exam twist: total BERT loss (MLM + NSP)

Forecast: NSP loss is small (label almost certain), so total just above $2.5$.

1. NSP loss. Why? NSP is a binary classification: ${\mathcal{L}}_{\text{NSP}}=-\log P(\text{correct label})$. ${\mathcal{L}}_{\text{NSP}}=-\log (0.90)=\mathrm{0.1054.}$
2. Add the two objectives. Why? Parent note: ${\mathcal{L}}_{\text{BERT}}={\mathcal{L}}_{\text{MLM}}+{\mathcal{L}}_{\text{NSP}}$. ${\mathcal{L}}_{\text{BERT}}=2.5257+0.1054=\mathrm{2.6311.}$

Verify: NSP contributes only $0.11$ of the $2.63$ total — MLM dominates, which is why later models (RoBERTa) dropped NSP with little loss of quality. $2.6311$ ✓.

---

Recall checks

Recall Why does Cell B give a bigger loss than Cell A?

Because the corrupted input shows a wrong word, actively misleading the model, yet the target is still the original — a harder prediction. ::: The 10% random-swap forces robustness.

Recall What is the maximum possible per-slot MLM loss?

$\log |\mathcal{V}|$ (uniform guessing). ::: For a 30k vocab that is $\approx 10.31$.

Recall Loss of a sequence with an empty mask set?

Exactly $0$ — an empty sum. ::: BERT forces $\ge 1$ mask to avoid wasting the example.