Next sentence prediction: Exercises — AI-ML deep dive

This page is a self-test ladder for Next sentence prediction. Work each problem before opening its solution. The levels climb from "can you recognise the pieces" (L1) up to "can you invent a variant" (L5). Every symbol used here is built inside the parent note; if a term feels unfamiliar, re-read the parent first.

Level 1 — Recognition

Exercise 1.1 — Label the input format

NSP feeds the model one concatenated string. Fill in the two special tokens and the segment IDs for this pair:

A: the sky is blue
B: it never rains here

Write the full token sequence in the form [?] ... [?] ... [?] and the segment-ID row.

Recall Solution 1.1

The format is ==[CLS] A [SEP] B [SEP]==. Segment ID $0$ covers [CLS] through the first [SEP]; segment ID $1$ covers B's tokens through the final [SEP].

Tokens:   [CLS] the sky is blue [SEP] it never rains here [SEP]
Segment:    0    0   0   0   0    0   1   1    1    1    1

Segment A here has $6$ tokens ([CLS] + 4 words + first [SEP]), segment B has $5$ tokens (4 words + final [SEP]). Total $= 11$ tokens.

Exercise 1.2 — Which label?

For each pair, state IsNext (1) or NotNext (0), given they were built by BERT's standard 50/50 procedure:

(a) A and B are consecutive sentences from the same document.
(b) B is drawn uniformly at random from the whole corpus.

Recall Solution 1.2

(a) ==IsNext, label $= 1$ ==. This is a positive example.
(b) ==NotNext, label $= 0$ ==. This is a negative example (random sampling).

Recall the split is 50% positive, 50% negative.

Exercise 1.3 — What does `[CLS]` do?

In one sentence: why does NSP read out its prediction from the [CLS] token and not, say, the last token of B?

Recall Solution 1.3

The ==[CLS] token's final hidden state is trained to aggregate the whole pair== — it attends to every token in both A and B through self-attention, so its vector is the natural "summary" to feed a classifier. Any token could in principle be used, but [CLS] is the designated slot BERT reserves and trains for this job.

Level 2 — Application

Exercise 2.1 — Count positive/negative pairs

You want a balanced NSP dataset of $40, 000$ training pairs. How many positive and how many negative pairs, following BERT's default ratio?

Recall Solution 2.1

Default ratio is $50%/50%$ . $positives = 0.5 \times 40000 = 20000, negatives = 0.5 \times 40000 = 20000.$

Exercise 2.2 — Softmax from logits

The NSP head outputs logits for $[NotNext, IsNext]$ : $z = [z_{0}, z_{1}] = [0.40, 1.20] .$ Compute $P (IsNext)$ .

Here $softmax$ turns two raw scores into probabilities that sum to $1$ ; we use it (not just "pick the bigger logit") because training needs a probability to plug into cross-entropy.

Recall Solution 2.2

$P (IsNext) = \frac{e ^{z_{1}}}{e ^{z_{0}} + e ^{z_{1}}} = \frac{e ^{1.20}}{e ^{0.40} + e ^{1.20}} .$ Numerically $e^{1.20} \approx 3.3201$ , $e^{0.40} \approx 1.4918$ , so $P (IsNext) = \frac{3.3201}{1.4918 + 3.3201} = \frac{3.3201}{4.8119} \approx 0.690.$ And $P (NotNext) \approx 0.310$ . The model leans IsNext.

Exercise 2.3 — Cross-entropy loss for one example

Using the probabilities from 2.2, the true label is IsNext ( $y = 1$ ). Compute the NSP loss for this single example, where $L = - [y lo g P (IsNext) + (1 - y) lo g P (NotNext)] .$

Recall Solution 2.3

With $y = 1$ only the first term survives: $L = - lo g P (IsNext) = - lo g (0.690) \approx 0.371.$ (Natural log.) Because the model was already fairly confident and correct, the loss is small.

Level 3 — Analysis

Exercise 3.1 — Reproduce the parent's forward pass

Given $h_{CLS} = [0.2, - 0.5, 0.8, 0.1]$ and $W_{NSP} = [0.3 - 0.1 0.1 0.5 - 0.2 0.3 0.4 - 0.2], b_{NSP} = [0.1 - 0.1],$ compute both logits, then $P (NotNext)$ . Confirm the model's predicted class.

Look at figure s01: each logit is a dot product of the CLS vector with one weight row, then a bias shift.

Recall Solution 3.1

Row 0 (NotNext): $z_{0} = (0.3) (0.2) + (0.1) (- 0.5) + (- 0.2) (0.8) + (0.4) (0.1) + 0.1 = 0.06 - 0.05 - 0.16 + 0.04 + 0.1 = - 0.01.$ Row 1 (IsNext): $z_{1} = (- 0.1) (0.2) + (0.5) (- 0.5) + (0.3) (0.8) + (- 0.2) (0.1) - 0.1 = - 0.02 - 0.25 + 0.24 - 0.02 - 0.1 = - 0.15.$ Softmax: $P (NotNext) = \frac{e ^{- 0.01}}{e ^{- 0.01} + e ^{- 0.15}} = \frac{0.99005}{0.99005 + 0.86071} \approx 0.535.$ Since $z_{0} > z_{1}$ , the ==predicted class is NotNext== at $\approx 53.5%$ . Matches the parent note.

Exercise 3.2 — Why NSP accuracy hits 97%+ so fast

NSP training accuracy climbs past $97%$ within a fraction of pre-training. Explain, using the construction of negatives, why this does not prove the model learned deep coherence.

Recall Solution 3.2

Negatives are drawn uniformly from the entire corpus, so B usually comes from a different document on a different topic. The model can separate IsNext from NotNext by a cheap heuristic — detecting topic/word overlap — without reasoning about narrative flow. High accuracy therefore reflects the task being easy, not the model being deep. This is exactly why SOP (same-document, only order swapped) was proposed: it removes the topic shortcut.

Exercise 3.3 — Interpreting the ablation

BERT with NSP scores $84.6%$ MNLI; without NSP, $\approx 83.4%$ . RoBERTa dropped NSP and lost nothing. Compute the reported NSP gain, and reconcile the two findings in one line each.

Recall Solution 3.3

Gain $= 84.6 - 83.4 === 1.2 ==$ points.

BERT view: with BERT's short-pair data setup, NSP added a small but real signal ( $1.2$ pts).
RoBERTa view: by packing full sentences up to $512$ tokens, MLM already saw cross-sentence context, so NSP became redundant — hence no drop when removed. Both are consistent: NSP helped BERT's specific setup, not language modeling universally.

Level 4 — Synthesis

Exercise 4.1 — Design a harder negative sampler

Rewrite the negative-example rule so the model cannot solve NSP by topic detection. State your rule and name the published objective it corresponds to.

Recall Solution 4.1

Rule: take two genuinely consecutive sentences A, B from the same document and, for negatives, swap their order (present B then A) with label "swapped". Both segments share topic, entities, and style, so the topic-overlap shortcut is gone — the model must judge directional coherence. This is Sentence Order Prediction (SOP), from ALBERT. See 4.2.09-Sentence-order-prediction.

Exercise 4.2 — Combined pre-training objective

BERT optimises MLM and NSP together. Write the total loss and explain, in terms of what each signal teaches, why summing them (rather than training separately) is chosen.

Recall Solution 4.2

$L_{total} = L_{MLM} + L_{NSP} .$

$L_{MLM}$ (see 4.2.06-Masked-language-modeling) teaches token-level semantics — filling masked words.
$L_{NSP}$ teaches sentence-level discourse — pair coherence. Summing lets one forward pass produce both gradients, so the shared Transformer body learns complementary signals simultaneously; the same [CLS]-and-tokens representation is pushed to be good at both scales at once, which is cheaper and more stable than alternating separate runs.

Exercise 4.3 — Map NSP to a downstream task

Explain how the exact NSP head transfers to a real task with almost no change, and name one such task from the connections.

Recall Solution 4.3

NSP is already a [CLS]-vector → 2-class linear head over a sentence pair. A pair-classification downstream task reuses this structure verbatim: feed [CLS] premise [SEP] hypothesis [SEP], read [CLS], apply a (re-initialised) linear head. This is the essence of 5.1.04-Transfer-learning-NLP. Concrete task: 6.3.02-Natural-language-inference (premise–hypothesis pairs), or 6.2.01-Question-answering-systems (passage–question pairs).

Level 5 — Mastery

Exercise 5.1 — Expected loss of a random classifier

A freshly initialised NSP head outputs $P (IsNext) = P (NotNext) = 0.5$ for every example. On a balanced dataset, what is the expected per-example cross-entropy loss? Give the exact form and the number.

Recall Solution 5.1

Whatever the true label, the loss is $- lo g (0.5)$ . So the expectation is $E [L] = - lo g (0.5) = lo g 2 \approx 0.6931.$ This is the classic " $ln 2$ " starting loss for a balanced binary task — a sanity-check baseline: if your NSP loss starts far from $0.693$ , something is mis-initialised.

Exercise 5.2 — Confidence needed to beat baseline by a target margin

You want each correct, confident example to have loss $\leq 0.20$ . What minimum $P (true label)$ achieves this?

Recall Solution 5.2

Require $- lo g p \leq 0.20 \Rightarrow p \geq e^{- 0.20}$ . $p \geq e^{- 0.20} \approx 0.8187.$ So the model must assign $\geq\approx 81.9%$ probability to the correct class.

Exercise 5.3 — Full end-to-end synthesis

A negative pair is built. After the Transformer, $h_{CLS} = [1.0, 0.0, - 1.0, 0.5]$ . Using the same $W_{NSP}, b_{NSP}$ as Exercise 3.1: (a) compute logits, (b) $P (NotNext)$ , (c) the loss given true label NotNext ( $y = 0$ ), (d) state whether the prediction is correct.

Recall Solution 5.3

(a) Logits: $z_{0} = (0.3) (1.0) + (0.1) (0.0) + (- 0.2) (- 1.0) + (0.4) (0.5) + 0.1 = 0.3 + 0 + 0.2 + 0.2 + 0.1 = 0.8.$ $z_{1} = (- 0.1) (1.0) + (0.5) (0.0) + (0.3) (- 1.0) + (- 0.2) (0.5) - 0.1 = - 0.1 + 0 - 0.3 - 0.1 - 0.1 = - 0.6.$ (b) Softmax for NotNext: $P (NotNext) = \frac{e ^{0.8}}{e ^{0.8} + e ^{- 0.6}} = \frac{2.22554}{2.22554 + 0.54881} \approx 0.8022.$ (c) True label NotNext ( $y = 0$ ): loss $= - lo g P (NotNext) = - lo g (0.8022) \approx 0.2204.$ (d) Since $z_{0} > z_{1}$ , predicted class is ==NotNext, which is correct==, with $\approx 80.2%$ confidence.

Recall Quick self-quiz (reveal answers)

Format [CLS] A [SEP] B [SEP] — final [SEP] segment ID? ::: 1 (belongs to B) BERT positive/negative NSP split? ::: 50% / 50% NSP gain on MNLI in BERT ablation? ::: about 1.2 points (84.6 vs 83.4) Baseline cross-entropy for balanced binary? ::: ln 2 ≈ 0.693 Objective that replaced NSP in ALBERT? ::: Sentence Order Prediction (SOP) Why RoBERTa lost nothing dropping NSP? ::: long packed sequences let MLM capture cross-sentence context

Related: 4.2.06-Masked-language-modeling · 4.2.09-Sentence-order-prediction · 4.3.02-BERT-architecture · 4.3.08-RoBERTa · 4.2.03-Segment-embeddings · 5.1.04-Transfer-learning-NLP · 6.2.01-Question-answering-systems · 6.3.02-Natural-language-inference