BERT and encoder models: Exercises — AI-ML deep dive

Before we start, one tiny piece of notation used throughout. When we write $lo g$ we always mean the natural logarithm (base $e$ , where $e \approx 2.718$ ). It is the inverse of " $e$ to the power of": $lo g (e^{x}) = x$ . We use it because the loss BERT minimizes — Cross-Entropy Loss — is built from natural logs (that is what makes the calculus clean when training). And $softmax$ is the function that turns a list of raw scores (called logits) into probabilities that are all positive and sum to $1$ :

$softmax (z)_{w} = \frac{e ^{z_{w}}}{\sum _{v \in V} e ^{z_{v}}}$

Here the index $v$ runs over $V$ , the entire vocabulary — every token the model could possibly output — so the denominator is the sum of $e^{z_{v}}$ across all candidate tokens, and $w$ is the one particular token whose probability we want. Read the whole thing as: "take $e$ to the power of each score, then divide each by the total so they add to one."

Formula The two forms of cross-entropy used on this page

Cross-entropy measures how surprised the model is by the correct answer: the lower the probability it gave the true label, the higher the loss. It comes in two flavours.

Multi-class (used by MLM), when the answer is one of many vocabulary tokens $y_{i}$ : $L_{CE} = - lo g p (y_{i} ∣ context), p (y_{i} ∣ context) = softmax (z_{i})_{y_{i}}$ Averaged over all masked positions this gives $L_{MLM} = - \frac{1}{∣ M ∣} \sum_{i \in M} lo g p (y_{i} ∣ context)$ .

Binary (used by NSP), when the answer is yes/no with label $y \in {0, 1}$ and predicted probability $\overset{p}{^}$ : $L_{NSP} = - [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})], \overset{p}{^} = σ (w^{⊤} h_{[CLS]})$ Both are the same idea (negative log of the probability assigned to the truth); binary is just the two-class special case. See Cross-Entropy Loss.

Reading the first figure. The left panel shows three raw logits (bar heights = scores). The right panel shows what softmax does to them: exponentiate each, then divide by the total so the amber bars sum to exactly $1.0$ . Notice how the largest logit (dog) grabs most of the probability — that is softmax "sharpening" the scores into a decision.

Reading the second figure. Each grid is an attention map: the row is the token doing the looking (query), the column is the token being looked at (key), and a ✓ means "allowed to attend". On the left (BERT) every cell is ✓ — bidirectional, every token sees all others. On the right (GPT) only the lower triangle is ✓ — causal, a token may only look at itself and the past. This one difference is why BERT can be bidirectional and why it cannot use next-token prediction (it would see the answer).

Everything below rests on these tools.

Level 1 — Recognition

Exercise 1.1

For each model, name whether its attention is bidirectional or causal (left→right), and whether its main job is understanding or generation: (a) BERT, (b) GPT, (c) T5.

Recall Solution

(a) BERT — bidirectional attention, job = understanding. It reads left and right context at once. See Self-Attention.
(b) GPT — causal attention, job = generation. It only sees the past so it can predict the next token honestly. See GPT and decoder models.
(c) T5 — both (encoder is bidirectional, decoder is causal), job = seq-to-seq. See T5 and encoder-decoder models.

Exercise 1.2

Fill the blanks: in the 80/10/10 rule, of the 15% chosen tokens, ___% become [MASK], ___% become a random token, and ___% are left unchanged.

Recall Solution

[MASK]: 80%
random token: 10%
unchanged: 10%

Check: $80 + 10 + 10 = 100$ percent of the chosen tokens (not of the whole sentence).

Exercise 1.3

Which special token's final vector is used as the whole-sentence summary for classification?

Recall Solution

The [CLS] token. Its final vector $h_{[CLS]}$ aggregates information about the entire input, so a single linear layer on top can make a sentence-level decision.

Level 2 — Application

Exercise 2.1

A masked position has logits over three candidate words: $z = [dog : 3.0, cat : 1.0, log : 0.0]$ . The true word is dog. (a) Compute $p (dog)$ via softmax. (b) Compute the MLM (Masked Language Modelling) loss $- lo g p (dog)$ .

Recall Solution

Step 1 — softmax. WHY? MLM grades a probability, and softmax is how logits become probabilities. Here the vocabulary $V$ is just these three words, so $v$ runs over {dog, cat, log}. $e^{3} = 20.086, e^{1} = 2.718, e^{0} = 1.000, sum = 23.804$ $p (dog) = \frac{20.086}{23.804} = 0.844$ Step 2 — loss. WHY? MLM loss is the negative natural log of the true token's probability (the multi-class cross-entropy from the definitions above). $L = - lo g (0.844) = 0.169$ Interpretation: small loss — the model is already confident and correct.

Exercise 2.2

Same logits as 2.1, but now the true word is log (the least likely). Compute the loss and explain in one line what a large loss means.

Recall Solution

$p (log) = \frac{1.000}{23.804} = 0.042$ $L = - lo g (0.042) = 3.170$ A large loss means the model assigned low probability to the correct answer — it was wrong and confident, so gradients will push it hard to fix this.

Exercise 2.3

A sentence has 20 tokens. BERT masks 15% of them (rounding to the nearest whole token). During loss computation, how many positions contribute to $L_{MLM}$ , and how many contribute $0$ ?

Recall Solution

$0.15 \times 20 = 3$ masked positions, so the set of masked positions $M$ has $∣ M ∣ = 3$ . Only those 3 contribute to the loss (we grade only over $M$ , where we hid answers). The other 17 contribute exactly $0$ — grading a token the model can see would teach nothing.

Exercise 2.4

For NSP (Next Sentence Prediction), the raw score is $s = w^{⊤} h_{[CLS]} = 1.2$ . The probability of IsNext is $\overset{p}{^} = σ (s)$ where $σ (x) = \frac{1}{1 + e ^{- x}}$ . The true label is IsNext ( $y = 1$ ). Compute $\overset{p}{^}$ and the NSP loss $- [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})]$ .

Recall Solution

Step 1 — sigmoid. WHY? NSP is binary (IsNext vs NotNext), so we squash the single score into one probability with $σ$ . $\overset{p}{^} = \frac{1}{1 + e ^{- 1.2}} = \frac{1}{1 + 0.3012} = 0.7685$ Step 2 — loss. This is the binary cross-entropy from the definitions; with $y = 1$ the second term vanishes: $L_{NSP} = - lo g (0.7685) = 0.2633$

Level 3 — Analysis

Exercise 3.1

The 80/10/10 rule keeps 20% of chosen positions as non-[MASK] (random or unchanged). Argue precisely why removing this and always using [MASK] would hurt the fine-tuned model, even though it might lower pretraining loss.

Recall Solution

During fine-tuning and inference the [MASK] token never appears. If the model only ever builds good representations at [MASK] positions, it learns "produce useful features only for [MASK]" — a train/test distribution mismatch. The 10% random + 10% unchanged mean the model can never be sure a given token is corrupt, so it is forced to build a good contextual representation for every token. Pretraining loss might drop (easier to memorize "predict the [MASK]"), but the representations transfer worse to real, mask-free downstream text. See Fine-Tuning vs Pretraining.

Exercise 3.2

RoBERTa dropped NSP entirely and got better results. Explain the mechanism: what did NSP actually teach, and why was that redundant/harmful?

Recall Solution

NSP mixes two signals: topic prediction (a random sentence B is usually about a different topic) and coherence (true B follows A). The topic signal is easy and shallow — the model can solve NSP by topic alone without learning real inter-sentence reasoning. Feeding two unrelated segments also shortens the effective context each MLM prediction sees. RoBERTa replaced this with longer contiguous spans and more MLM data, giving the model more, harder language-modelling signal. Net effect: dropping the weak NSP task frees capacity for the load-bearing MLM objective.

Exercise 3.3

Why does BERT sum three embeddings (token + position + segment) rather than just the token embedding?

Recall Solution

Self-attention is permutation-invariant: it treats the input as a bag of tokens, so "dog bites man" and "man bites dog" would be identical. Position embeddings inject word order. Segment embeddings tell the model which sentence (A vs B) a token belongs to — essential for two-sentence tasks like NSP or entailment. Without these two, the same token in different roles would get the same starting vector.

Level 4 — Synthesis

Exercise 4.1

You must design fine-tuning heads for three tasks. For each, state which hidden vector(s) you feed into the head, the weight matrix and bias shapes, and the output shape of the head. Tasks: (a) sentiment (2 classes), (b) NER token-tagging (9 tags), (c) SQuAD span extraction. Assume hidden size $d = 768$ and sequence length $n$ .

Recall Solution

A linear head computes $z = W h + b$ : the weight matrix $W$ maps the hidden vector to logits, and the bias $b$ is a learned vector added to every logit (one bias entry per output class). Its length always equals the number of outputs.

(a) Sentiment. Feed only $h_{[CLS]} \in R^{768}$ (the sentence summary) into $W \in R^{2 \times 768}$ with bias $b \in R^{2}$ → 2 logits → softmax. Output shape: $2$ (one decision per sentence).
(b) NER. Feed every token vector $h_{i} \in R^{768}$ into a shared $W \in R^{9 \times 768}$ with bias $b \in R^{9}$ → $9$ logits per token → output shape $n \times 9$ (one decision per token).
(c) SQuAD span. Use one head $W \in R^{2 \times 768}$ , $b \in R^{2}$ applied to every token vector $h_{i}$ . This produces, per token, two scores: a start score and an end score. Collecting them across all $n$ tokens gives a tensor of shape $n \times 2$ , which we split into a length- $n$ start-score vector and a length- $n$ end-score vector. We then apply a softmax over the $n$ positions to the start vector (giving $P (start = i)$ for each token $i$ ) and a separate softmax over the $n$ positions to the end vector (giving $P (end = j)$ ). The predicted answer span is the pair $(i, j)$ with $i \leq j$ maximizing $P (start = i) \cdot P (end = j)$ . So: head output $n \times 2$ → two independent length- $n$ softmax distributions.

Exercise 4.2

Given the sentiment head producing logits $z = [neg : 0.5, pos : 2.0]$ for input [CLS] this movie is great [SEP], compute $P (pos)$ .

Recall Solution

Softmax over 2 classes (so $v$ runs over {neg, pos}): $e^{0.5} = 1.6487, e^{2.0} = 7.3891, sum = 9.0378$ $P (pos) = \frac{7.3891}{9.0378} = 0.8177$ The model is ~82% confident the review is positive — sensible, since "great" dominates.

Exercise 4.3

The word "unhappiness" is one word but WordPiece Tokenization may split it into un ##happi ##ness (3 tokens). If MLM masks 15% of tokens and a sentence has 40 tokens, and you (wrongly) thought 15% of words, how could the two counts differ? Compute the token-based masked count.

Recall Solution

Token-based count: $0.15 \times 40 = 6$ masked tokens. Because one word can be several sub-word tokens, "15% of words" and "15% of tokens" are different quantities — BERT masks by token, so a single long word may have one of its pieces masked while the rest stay visible. The correct answer is 6 masked tokens.

Level 5 — Mastery

Exercise 5.1

Two masked positions in a batch have true-token probabilities $p_{1} = 0.665$ and $p_{2} = 0.042$ . Compute the batch MLM loss $L_{MLM} = - \frac{1}{∣ M ∣} \sum_{i \in M} lo g p_{i}$ , where $M$ is the set of masked positions ( $∣ M ∣ = 2$ here). Then explain which position dominates the gradient and why.

Recall Solution

$L_{MLM} = - \frac{1}{2} (lo g 0.665 + lo g 0.042) = - \frac{1}{2} (- 0.4079 - 3.1701) = \frac{1}{2} (3.5780) = 1.789$ Position 2 dominates. Its probability is tiny, so its $- lo g p$ is large; the cross-entropy gradient is roughly proportional to $(1 - p)$ at the true class, which is $\approx 0.958$ for position 2 vs $\approx 0.335$ for position 1. The model learns most from the tokens it currently gets most wrong. The figure below plots this gradient magnitude against $p$ .

Reading the figure: the amber curve is the loss $- lo g p$ (soars as $p \to 0$ ); the cyan curve is the gradient size $(1 - p)$ . The two marked dots are our two positions — the low- $p$ one sits high on both curves, showing it both hurts more and teaches more.

Exercise 5.2

A student claims: "Since BERT is a huge Transformer trained on text, I can prompt it to continue a story." Refute this at the level of the loss function and attention mask — do not just say "it's encoder-only".

Recall Solution

Attention-mask argument. Generation is autoregressive: produce token $t$ , feed it back, produce token $t + 1$ , and so on. For this to be honest, position $t$ must not see tokens after it, which requires a causal mask (the lower-triangular pattern in figure s02, right). BERT uses the bidirectional mask (figure s02, left) where every position already sees every other position. If you tried to generate with it, position $t$ would be allowed to peek at the very future tokens it is supposed to produce — the setup is structurally incompatible with left-to-right generation.

Loss-function argument. BERT was trained with the MLM cross-entropy $- lo g p (y_{i} ∣ context)$ evaluated only at pre-chosen masked slots inside a fixed-length input. Nowhere in that objective is there a term of the form "predict token $t + 1$ from tokens $1.. t$ ". So even the learned behaviour is "fill a hole given both sides", never "extend the sequence". There is no next-position probability to sample from and no mechanism to grow beyond the input length.

Conclusion. Both the mask (sees the future) and the loss (no next-token term, fixed length) forbid a generation loop. Use GPT and decoder models (causal mask + next-token loss) or T5 and encoder-decoder models (a causal decoder attached to BERT-like encoding) for generation.

Exercise 5.3

Show that if softmax gives the true token probability $p = 1$ , the MLM loss is exactly $0$ , and explain why the loss can approach but never reach $\infty$ as $p \to 0$ .

Recall Solution

$L = - lo g p$ . At $p = 1$ : $- lo g (1) = 0$ — a perfectly confident, correct prediction has zero loss. As $p \to 0^{+}$ , $- lo g p \to + \infty$ : an infinitely confident wrong prediction is punished without bound. But softmax outputs are strictly $> 0$ (numerator $e^{z_{w}} > 0$ ), so $p$ can get arbitrarily small yet never equal $0$ — the loss is always finite in practice, just large.

Recall Final self-check (each line is Question ::: Answer — cover the right side, then reveal)

MLM loss for $p = 0.844$ ? ::: $0.169$
Loss when the true token has $p = 0.042$ ? ::: $3.170$
Batch loss for $p = 0.665, 0.042$ ? ::: $1.789$
Which objective did RoBERTa drop, and why was it safe? ::: NSP — its signal was mostly shallow topic-matching, redundant with more MLM.
Which vector feeds a sentiment head? ::: $h_{[CLS]}$