BERT and encoder models: Worked examples — AI-ML deep dive

This deep dive gives you a full worked-example gym for BERT. We do not re-teach the theory — we exercise it. Every knob that BERT can turn (how confident the model is, how many tokens are masked, the 80/10/10 split, a two-sentence NSP head, a fine-tuning classifier head, and the degenerate/limiting corners) gets a concrete number that you can check by hand.

Before we start, one promise: every symbol used below is re-earned here. If you have not read the parent note, that's fine — I define each ingredient the moment it appears.

The scenario matrix

Think of BERT's math as a machine with a few dials. Each row below is a class of situation the machine can be in. Our examples must hit every cell.

#	Case class	What makes it special	Hit by
C1	Confident correct prediction	true token has the biggest logit → small loss	Ex 1
C2	Confident wrong prediction	true token has a small logit → large loss	Ex 2
C3	Uniform / uncertain (degenerate)	all logits equal → loss $= lo g (vocab size)$	Ex 3
C4	Multiple masked tokens averaged	loss is a mean over $∣ M ∣$ positions	Ex 4
C5	Zero-mask / limiting case	nothing masked → loss undefined/zero contribution	Ex 4 (part b)
C6	80/10/10 counting (real-world word problem)	how many tokens become `[MASK]` vs random vs kept	Ex 5
C7	NSP binary head (both signs of label)	`IsNext` vs `NotNext`, sigmoid + binary cross-entropy	Ex 6
C8	Fine-tuning classifier head	`[CLS]` vector → linear → softmax over classes	Ex 7
C9	Exam-style twist	"predict left-to-right?" trap + combined total loss	Ex 8

Two tools appear repeatedly, so let me name them once.

(All logs below are natural logs, base $e$ , to match the theory in the parent note.)

Ex 1 — C1: Confident and correct

Worked example The cat sat on the [MASK]

The model outputs 3 vocab logits at the masked slot: $z = [mat : 2.0, car : 1.0, sky : 0.0]$ . True word $=$ mat.

Forecast: the true word has the largest score, so guess: probability well above one-third, and a small loss (well under $1$ ). Write your guess down.

Step 1 — exponentiate each logit. Why this step? softmax needs positive values; $e^{z}$ delivers them. $e^{2.0} = 7.389, e^{1.0} = 2.718, e^{0.0} = 1.000$

Step 2 — sum them. Why this step? the denominator normalises the probabilities to add to $1$ . $7.389 + 2.718 + 1.000 = 11.107$

Step 3 — probability of the true word. Why this step? the loss only cares about $p_{y}$ , the probability of the correct token. $p (mat) = \frac{7.389}{11.107} = 0.665$

Step 4 — cross-entropy. Why this step? MLM loss at a masked slot is $- lo g p_{y}$ . $L = - lo g (0.665) = 0.408$

Verify: $p (mat) + p (car) + p (sky) = 0.665 + 0.245 + 0.090 = 1.000$ ✓ (probabilities sum to one). Loss $0.408 < 1$ and $p > 1/3$ , matching our forecast of "confident, low loss." ✓

Ex 2 — C2: Confident and WRONG

Worked example Same slot, but the model favours the wrong word

Logits $z = [mat : 0.0, car : 3.0, sky : 0.0]$ . True word is still mat, but the model loves car.

Forecast: the true word now has the smallest score, so $p (mat)$ should be tiny and the loss large (bigger than $2$ ). Guess before reading.

Step 1 — exponentiate. Why? same softmax recipe. $e^{0} = 1.000, e^{3.0} = 20.086, e^{0} = 1.000$

Step 2 — sum. Why? normaliser. $1.000 + 20.086 + 1.000 = 22.086$

Step 3 — probability of the true word. Why? loss uses $p_{mat}$ . $p (mat) = \frac{1.000}{22.086} = 0.0453$

Step 4 — loss. Why? $- lo g p_{y}$ . $L = - lo g (0.0453) = 3.095$

Verify: $p (car) = 20.086/22.086 = 0.9095$ — the model is 91% sure of the wrong answer, and the loss $3.095$ is much larger than Ex 1's $0.408$ . Confident-wrong is punished ~ $7.6 \times$ harder. Look again at the steep left wall in figure s01. ✓

Ex 3 — C3: Uniform logits (the degenerate "no information" case)

Ex 4 — C4 & C5: Averaging over masked positions, and the zero-mask limit

Worked example Two masks, then zero masks

Part (a). A sentence has masked positions $M = {3, 7}$ . Their individual losses are $L_{3} = 0.408$ (from Ex 1) and $L_{7} = 3.095$ (from Ex 2).

Recall the parent's [!formula]: $L_{MLM} = - \frac{1}{∣ M ∣} i \in M \sum lo g p (y_{i} ∣ context)$ , i.e. the mean of the per-mask losses.

Forecast: the average of $0.408$ and $3.095$ — guess roughly $1.75$ .

Step 1 — count masked positions. Why? $∣ M ∣$ is the divisor. $∣ M ∣ = 2$ . Step 2 — average. Why? the loss is a mean, not a sum, so long and short sentences are comparable. $L_{MLM} = \frac{0.408 + 3.095}{2} = 1.7515$

Verify: it sits between the two inputs $0.408$ and $3.095$ ✓, and the 8 unmasked tokens contribute nothing (we grade only where we hid an answer). ✓

Part (b) — the zero-mask limit ( $∣ M ∣ = 0$ ). If no token is masked, the sum over $M$ is empty and the divisor is $0$ — the formula is undefined ( $0/0$ ). In practice BERT guarantees at least one masked token per sequence, so this never fires. Takeaway: MLM loss needs $∣ M ∣ \geq 1$ ; the " $0$ -mask" case is a degenerate input you must exclude, not compute.

Ex 5 — C6: The 80/10/10 counting word problem

Worked example How many of the chosen tokens actually become [MASK]?

A tokenized input has 200 tokens (WordPieces — see WordPiece Tokenization). BERT masks $15%$ , then splits those by the 80/10/10 rule: $80%$ become [MASK], $10%$ become a random token, $10%$ are left unchanged.

Forecast: $15%$ of 200 is 30 chosen tokens; of those, guess how many are [MASK], random, kept.

Step 1 — count the chosen tokens. Why? the 80/10/10 split applies to the chosen set, not all 200. $0.15 \times 200 = 30 chosen tokens$ Step 2 — apply 80/10/10. Why? to see how many the model literally sees as [MASK]. $[MASK] = 0.80 \times 30 = 24, random = 0.10 \times 30 = 3, unchanged = 0.10 \times 30 = 3$ Step 3 — how many positions does the loss grade? Why? the loss covers all 30 chosen positions, including the random and unchanged ones — the model must still predict the original token there. $graded positions = 30$

Verify: $24 + 3 + 3 = 30$ ✓ equals the chosen count. Note the subtle point: only 24 tokens are physically replaced by [MASK], yet the model is graded on all 30 — this uncertainty ("is this token real or corrupt?") is exactly why the 80/10/10 rule exists (parent note). ✓

Ex 6 — C7: The NSP head, both label signs

Worked example Is sentence B really next? — sigmoid binary loss

BERT reads [CLS] A [SEP] B [SEP]. The [CLS] vector $h_{[CLS]}$ is dotted with a learned weight $w$ to give a single score $s = w^{⊤} h_{[CLS]} = 1.2$ . NSP squashes it with the sigmoid $σ (s) = \frac{1}{1 + e ^{- s}}$ to get $\overset{p}{^} = P (IsNext)$ .

Binary cross-entropy: $L_{NSP} = - [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})]$ , where $y = 1$ means IsNext, $y = 0$ means NotNext.

Forecast: $s = 1.2 > 0$ so $\overset{p}{^} > 0.5$ ; the loss is small if the true label is IsNext and large if it's NotNext.

Step 1 — sigmoid. Why? turn the score into a probability. $\overset{p}{^} = \frac{1}{1 + e ^{- 1.2}} = \frac{1}{1 + 0.3012} = 0.7685$ Step 2a — case $y = 1$ (IsNext, correct). Why? cover the positive label. $L = - lo g (0.7685) = 0.2634$ Step 2b — case $y = 0$ (NotNext, wrong). Why? cover the negative label — the other sign. $L = - lo g (1 - 0.7685) = - lo g (0.2315) = 1.4634$

Verify: the two probabilities sum: $0.7685 + 0.2315 = 1.000$ ✓. The correct-label loss ( $0.263$ ) is far smaller than the wrong-label loss ( $1.463$ ), as forecast. (Reminder from the parent note: RoBERTa found NSP often unhelpful and dropped it — but the math still shows how a binary head is scored.) ✓

Ex 7 — C8: Fine-tuning classifier head

Worked example Sentiment: is this review positive or negative?

After pretraining, we add a tiny head $W \in R^{2 \times d}$ on top of $h_{[CLS]}$ . Suppose it produces class logits $z = [neg : 0.5, pos : 2.5]$ . The true label is pos.

Forecast: pos has the higher logit → $P (pos) > 0.5$ and a small loss.

Step 1 — softmax over the 2 classes. Why? multi-class head uses softmax (2 classes here). $e^{0.5} = 1.6487, e^{2.5} = 12.1825, sum = 13.8312$ $P (pos) = \frac{12.1825}{13.8312} = 0.8808, P (neg) = \frac{1.6487}{13.8312} = 0.1192$ Step 2 — loss. Why? cross-entropy on the true class pos. $L = - lo g (0.8808) = 0.1269$

Verify: $0.8808 + 0.1192 = 1.000$ ✓. Prediction $argmax =$ pos matches the true label, loss $\approx 0.13$ is small — a well-fine-tuned classifier. This reuses [CLS] because it was pretrained to aggregate whole-sentence info (see Fine-Tuning vs Pretraining). ✓

Ex 8 — C9: The exam-style twist

Worked example "Compute the total pretraining loss and spot the trap."

A student claims: "BERT masks 15% of tokens and predicts them left-to-right, then sums the losses." You are given one masked-token MLM loss $L_{MLM} = 1.7515$ (Ex 4a) and one NSP loss $L_{NSP} = 0.2634$ (Ex 6, IsNext). Find the total, and fix the student's two errors.

Forecast: total $= L_{MLM} + L_{NSP}$ ; guess $\approx 2.0$ .

Step 1 — combine. Why? parent note: total pretraining loss $L = L_{MLM} + L_{NSP}$ . $L = 1.7515 + 0.2634 = 2.0149$ Step 2 — fix trap #1. Why? BERT is bidirectional (Self-Attention with no causal mask). It predicts masked tokens all at once from both sides, not left-to-right. Left-to-right is GPT's job. Step 3 — fix trap #2. Why? only the 80% of chosen tokens become [MASK]; 10% random + 10% unchanged. And loss is an average over masked positions, not a raw sum.

Verify: $1.7515 + 0.2634 = 2.0149$ ✓, and each masked-slot loss is non-negative so the total is $\geq L_{NSP}$ alone ✓. The two conceptual fixes match the parent's "Common mistakes." ✓

Coverage check

Recall Did we hit every matrix cell?

C1 correct-confident ::: Ex 1 (loss 0.408) C2 confident-wrong ::: Ex 2 (loss 3.095) C3 uniform/degenerate ::: Ex 3 (loss = log V = 1.386) C4 averaging masks ::: Ex 4a (1.7515) C5 zero-mask limit ::: Ex 4b (undefined, excluded) C6 80/10/10 counting ::: Ex 5 (24 / 3 / 3) C7 NSP both signs ::: Ex 6 (0.263 vs 1.463) C8 fine-tune head ::: Ex 7 (0.127) C9 exam twist ::: Ex 8 (total 2.0149)

Return to the parent: BERT and encoder models. Related machinery: Cross-Entropy Loss, Transformer Architecture, T5 and encoder-decoder models.