Next sentence prediction: Worked examples — AI-ML deep dive

This page hammers the Next Sentence Prediction objective from every angle. To keep it self-contained, we restate every symbol and assumption from zero below, then squeeze the machinery through every case class it can hit.

Definition The complete NSP setup (all symbols defined here)

Input: a token sequence [CLS] A [SEP] B [SEP], where A and B are two text segments, [CLS] is a special front token and [SEP] marks boundaries.
$d_{model}$ — the width of the model's hidden vectors (e.g. $768$ ; we use $4$ in examples so arithmetic is doable by hand).
$h_{CLS} \in R^{d_{model}}$ — the final hidden state of the [CLS] token. Through attention it has "seen" every token of A and B, so it summarises the whole pair in one vector.
NSP head: a single linear layer with weight matrix $W_{NSP} \in R^{2 \times d_{model}}$ (two rows, one per class) and bias $b_{NSP} \in R^{2}$ . It produces two logits $z = W_{NSP} h_{CLS} + b_{NSP}$ .
Label convention (fix this once, use everywhere): the true label is $y \in {0, 1}$ where $y = 1$ means IsNext (B truly follows A) and $y = 0$ means NotNext (B is a random impostor).

Two tools we lean on constantly, defined from zero:

Formula Cross-entropy loss (stated before we ever use it)

Here the summation index $c$ ranges over the two class labels, $c \in {0, 1}$ (0 = NotNext, 1 = IsNext). The symbol $1 [\cdot]$ is the indicator: it equals $1$ when the statement inside the brackets is true and $0$ when it is false. So $1 [y = c]$ is $1$ only for the class that is actually correct, and $0$ for the other. With that spelled out, for a single example with true label $y$ the cross-entropy loss is $L = - \sum_{c \in {0, 1}} 1 [y = c] lo g P (y = c) = - lo g P (true class)$ Because only the true class survives the sum (every other term is multiplied by $0$ ), the loss is simply minus the log-probability the model gave to the correct answer. Its binary special case is $L = - [y lo g P (IsNext) + (1 - y) lo g P (NotNext)] .$ Both forms are identical; we quote whichever is handier. Every " $- lo g P$ " you see below is exactly this formula.

Recall Why the [CLS] token carries the answer

Question: which single vector do we feed to the NSP classifier, and why that one? ::: The final hidden state of the [CLS] token — it sits at the front and, through attention, has "looked at" every token in both A and B, so it aggregates the whole pair into one vector. Why this step? Feeding one pair-summary vector to a tiny linear head is what lets a single 2-way classifier decide IsNext/NotNext without re-reading every token itself.

The scenario matrix

Every situation NSP can produce falls into one of these cells. The examples below are labelled with the cell they cover.

Cell	What varies	Concrete case	Example
C1	Positive pair, clear coherence	Real next sentence, anaphora resolves	Ex 1
C2	Negative pair, easy (topic shift)	Random cross-document B	Ex 2
C3	Sign of logits: model correct & confident	High $P$ for true label	Ex 3
C4	Sign of logits: model wrong	High $P$ for wrong label → big loss	Ex 4
C5	Degenerate: tie ( $z_{0} = z_{1}$ )	Equal logits, $P = 0.5$	Ex 5
C6	Limiting behaviour of loss	$P \to 1$ and $P \to 0$	Ex 6
C7	Real-world word problem	Build a batch, count labels	Ex 7
C8	Exam twist: hard negative (same doc)	Why NSP fails, SOP fix	Ex 8
C9	Combined objective	$L_{MLM} + L_{NSP}$ total	Ex 9

Figures s01–s03 give the geometric backbone: the logit-to-probability curve, the loss curve, and the decision picture that places all four logit-sign examples on that loss curve.

The curve above is the whole story of softmax for two classes: the x-axis is the signed vote gap $Δ = z_{1} - z_{0}$ (how much more the model votes IsNext than NotNext), the y-axis is $P (IsNext)$ . When $Δ = 0$ , $P = 0.5$ (cell C5). Positive $Δ$ pushes toward $1$ , negative toward $0$ . Keep this picture in mind — every example is just a dot on this S-curve.

Worked examples

Worked example Example 1 — Cell C1: a clean positive pair

Corpus: "The cat sat on the mat. It purred loudly." A = "The cat sat on the mat.", B = "It purred loudly." What is the label, and what linguistic cue tells us?

Forecast: guess the label and the single word that gives it away before reading on.

Identify the pair source. B is the literal sentence following A in the same document. Why this step? NSP's positive rule is purely mechanical: "did B actually come right after A?" — nothing semantic is required to label it.
Assign the label. Since B is the true successor, label $y = 1$ (IsNext). Why this step? The target $y$ is what the loss compares against.
Spot the cue the model can learn from. "It" is a pronoun (anaphora: a word pointing back to an earlier noun) resolving to "cat". Topic continuity (cat → purred) is strong. Why this step? This is why a positive pair is learnable — coherent pairs share entities and topic.

Verify: The label is $1$ . Sanity check: swap B for any sentence about the cat and it should still feel like a continuation — it does ("It stretched."). ✓

Worked example Example 2 — Cell C2: an easy negative pair

Corpus: Doc1 (cats) and Doc2 ("Quantum computers use qubits. They leverage superposition."). A = "The cat sat on the mat.", B = "They leverage superposition." (randomly sampled, landed in Doc2). Label and why it is easy.

Forecast: will the model need deep reasoning, or a shallow trick?

Assign the label. B was drawn at random, not the true successor, so $y = 0$ (NotNext). Why this step? Negative rule: uniform random B from the corpus.
Measure topic overlap. Content words of A: {cat, sat, mat}. Of B: {leverage, superposition}. Overlap $= 0$ . Why this step? Zero shared vocabulary is the shallow signal the model exploits.
Note the broken anaphora. "They" has no plural antecedent in A. Why this step? Confirms the mismatch is easy to detect without discourse reasoning.

Verify: Label $= 0$ ; overlap count $= 0$ → the parent note's warning holds: random negatives are trivially separable by topic. ✓

Worked example Example 3 — Cell C3: model correct and confident (full forward pass)

Take $d_{model} = 4$ , $h_{CLS} = [0.2, - 0.5, 0.8, 0.1]$ , and the NSP head defined above. True label is NotNext ( $y = 0$ ). Compute logits, probabilities, and loss.

$W_{NSP} = [0.3 - 0.1 0.1 0.5 - 0.2 0.3 0.4 - 0.2], b_{NSP} = [0.1 - 0.1]$

Recall the index convention: row 0 → NotNext ( $z_{0}$ ), row 1 → IsNext ( $z_{1}$ ).

Forecast: guess whether $P (NotNext)$ lands above or below $0.5$ .

Row-0 logit (NotNext). Dot the first row with $h_{CLS}$ , add bias: $z_{0} = 0.3 (0.2) + 0.1 (- 0.5) + (- 0.2) (0.8) + 0.4 (0.1) + 0.1 = - 0.01$ Why this step? A logit is exactly "weighted vote" = (row of weights) · (features) + bias.
Row-1 logit (IsNext). $z_{1} = - 0.1 (0.2) + 0.5 (- 0.5) + 0.3 (0.8) + (- 0.2) (0.1) - 0.1 = - 0.15$ Why this step? Same rule, second class; two logits are all a binary head needs.
Signed vote gap and softmax. Using our convention, $Δ = z_{1} - z_{0} = - 0.15 - (- 0.01) = - 0.14$ . Negative $Δ$ means NotNext wins: $P (NotNext) = \frac{e ^{z_{0}}}{e ^{z_{0}} + e ^{z_{1}}} = \frac{e ^{- 0.01}}{e ^{- 0.01} + e ^{- 0.15}} \approx 0.535$ Why this step? Softmax converts the gap into a probability; the dot sits on the s01 curve at $Δ = - 0.14$ (just left of centre, so $P (IsNext) < 0.5$ ).
Loss (cross-entropy formula). True class is NotNext, so $L = - lo g P (true class) = - lo g P (NotNext) = - lo g (0.535) \approx 0.625$ . Why this step? Direct application of the cross-entropy formula stated above — only the true class's probability matters.

Verify: $P (NotNext) + P (IsNext) = 0.535 + 0.465 = 1.0$ . ✓ Loss $\approx 0.625$ is below $lo g 2 \approx 0.693$ (the "coin-flip" loss), confirming the model is slightly better than random on the correct side. ✓

The loss curve (s02) shows why "correct but unconfident" (Ex 3) sits at moderate loss $\approx 0.63$ : it is on the gentle right side of the curve where $P$ is above $0.5$ but not near $1$ .

Worked example Example 4 — Cell C4: model wrong (large loss)

Same $h_{CLS}$ and weights as Ex 3, but now the true label is IsNext ( $y = 1$ ). Same logits $z_{0} = - 0.01$ , $z_{1} = - 0.15$ , hence same $Δ = - 0.14$ . Compute the loss.

Forecast: will the loss be bigger or smaller than Ex 3?

Reuse the probabilities. $P (IsNext) \approx 0.465$ (unchanged — logits don't depend on the label). Why this step? The forward pass is label-blind; only the loss step consults $y$ .
Compute loss on the true class. $L = - lo g P (IsNext) = - lo g (0.465) \approx 0.766$ . Why this step? Cross-entropy again picks the true class's probability; here $Δ = - 0.14$ points the wrong way (toward NotNext), so the true class got under $0.5$ and the loss rises.
Interpret the gradient direction. Loss $> lo g 2$ means "worse than a coin flip"; backprop will push $z_{1}$ up, $z_{0}$ down (i.e. drive $Δ$ positive). Why this step? Shows the learning signal is strongest exactly when the model is wrong.

Verify: $0.766 > 0.693 = lo g 2 > 0.625$ . So a wrong-side prediction costs more than a coin flip, which costs more than a right-side prediction. Ordering is consistent. ✓

Worked example Example 5 — Cell C5: the degenerate tie

Suppose after training on an ambiguous pair the two logits come out exactly equal: $z_{0} = z_{1} = 0.42$ , so $Δ = 0$ . What is $P (IsNext)$ and the loss (for either true label)?

Forecast: guess the probability before computing.

Softmax of equal logits. $P (IsNext) = \frac{e ^{0.42}}{e ^{0.42} + e ^{0.42}} = \frac{1}{2}$ . Why this step? When votes tie ( $Δ = 0$ ), softmax must split probability evenly — the constant $0.42$ cancels top and bottom.
Note the constant is irrelevant. Shifting both logits by any amount leaves $P$ unchanged. Why this step? Softmax depends only on the difference $Δ = z_{1} - z_{0}$ (exactly the x-axis of s01).
Loss at the tie. $L = - lo g (0.5) = lo g 2 \approx 0.693$ , regardless of which label is true. Why this step? The maximum-uncertainty point is the reference loss for a 2-class problem.

Verify: This is the exact centre of the s01 S-curve ( $Δ = 0 \Rightarrow P = 0.5$ ) and the minimum-informative point of s02. ✓

Worked example Example 6 — Cell C6: limiting behaviour (

$P \to 1$ and $P \to 0$ ) A perfectly trained model outputs $Δ = z_{1} - z_{0} = 10$ on a true IsNext pair, and $Δ = - 10$ on a true NotNext pair. What happens to $P$ and the loss at these extremes?

Forecast: which case gives near-zero loss, which gives huge loss?

Confident-correct limit. $P (IsNext) = \frac{1}{1 + e ^{- 10}} \approx 0.9999546$ . Why this step? As $Δ \to + \infty$ , softmax saturates at $1$ — the flat right tail of s01.
Loss there. $L = - lo g (0.99995) \approx 4.54 \times 1 0^{- 5}$ — practically zero. Why this step? Being right and confident is what training drives loss toward.
Confident-wrong limit. If the true label were NotNext but $Δ = 10$ (model shouts IsNext), then $P (NotNext) \approx 4.5 \times 1 0^{- 5}$ and $L = - lo g (4.5 \times 1 0^{- 5}) \approx 10.0$ . Why this step? Confidently wrong is the most expensive place on the loss curve — it explodes toward $+ \infty$ .

Verify: As $P \to 1$ , $- lo g P \to 0$ ; as $P \to 0$ , $- lo g P \to \infty$ . The two computed numbers ( $\approx 4.5 \times 1 0^{- 5}$ and $\approx 10.0$ ) sit on opposite ends of s02. ✓

Worked example Example 7 — Cell C7: real-world word problem (building a batch)

You have a corpus of $8$ documents, $30$ sentences total. BERT wants a batch of $256$ NSP examples with the standard $50/50$ split. How many positive and negative pairs, and how do you draw each negative B?

Forecast: guess the two counts.

Apply the 50/50 rule. Positives $= 0.5 \times 256 = 128$ ; negatives $= 256 - 128 = 128$ . Why this step? Balanced labels stop the classifier from cheating by always predicting one class.
Draw positives. For each, pick a sentence A that has a successor, take the real next sentence B, label $1$ (IsNext). Why this step? Positive = true adjacency, by definition.
Draw negatives. For each, keep A, sample B uniformly from all $30$ sentences in the corpus, label $0$ (NotNext). Why this step? Uniform sampling is BERT's original recipe (the parent note's "random negatives").
Estimate accidental same-doc negatives. With uniform sampling over $30$ sentences and, say, $\approx 4$ sentences in A's own document, the chance a negative B lands in the same document is $\approx 4/30 \approx 13.3%$ . Why this step? Shows the parent note's caveat: same-doc negatives can occur but are rare and unintentional.

Verify: $128 + 128 = 256$ ✓. Probability $4/30 = 0.1 \overline{3} \approx 13.3%$ ✓ — a minority, matching "rare".

Worked example Example 8 — Cell C8: exam twist (why NSP is too easy, SOP fix)

Same document: S1 = "Quantum computers use qubits.", S2 = "They leverage superposition." Build (a) an NSP negative and (b) an SOP negative from these, and explain which forces deeper reasoning.

Forecast: which negative can the model solve with mere word-overlap?

NSP negative. Pair S1 with a random cross-document sentence, e.g. "The cat sat on the mat." Label NotNext. Overlap with S1 = $0$ content words. Why this step? Illustrates the shallow topic-shift signal — solvable without discourse understanding.
SOP negative. Take the same two consecutive sentences but swap the order: A = S2, B = S1. Label = "wrong order". Overlap is high (both about quantum computing). Why this step? SOP (Sentence Order Prediction) removes topic overlap as a cue, so the model must judge direction of coherence — "They" must follow its antecedent "qubits", not precede it.
Contrast the difficulty. NSP: separable by topic (easy). SOP: same topic, only order differs (hard). Why this step? Explains why RoBERTa could drop NSP with no loss, while SOP gave ALBERT real gains.

Verify: NSP-negative overlap $= 0$ (trivial); SOP-negative overlap $> 0$ with only order flipped (non-trivial). Matches the parent note's "high 97%+ NSP accuracy from shallow heuristics." ✓

Worked example Example 9 — Cell C9: the combined training objective

BERT trains two heads at once. The masked-language-modeling loss $L_{MLM}$ is the average cross-entropy over the masked tokens (predicting each hidden word from the vocabulary); the NSP loss $L_{NSP}$ is the single cross-entropy from our head above. Suppose on one batch $L_{MLM} = 2.10$ and $L_{NSP} = 0.625$ (the value from Ex 3). What total loss does BERT backpropagate, and why simply add them?

Forecast: guess the total, and guess whether NSP or MLM dominates it.

Add the two losses. $L_{total} = L_{MLM} + L_{NSP} = 2.10 + 0.625 = 2.725$ . Why this step? BERT weights both objectives equally (coefficient $1$ each); one shared Transformer body feeds two separate heads.
Why addition (not, say, a product)? The gradient of a sum is the sum of gradients, so each head updates from its own signal while the shared body receives both signals added together. Why this step? This is multi-task pre-training: complementary signals (MLM semantics + NSP discourse) folded into one backbone later reused by BERT for QA and NLI.
Compute the NSP share. $\frac{L _{NSP}}{L _{total}} = \frac{0.625}{2.725} \approx 0.229 = 22.9%$ . MLM dominates because it averages a large-vocabulary softmax, while NSP is a mere 2-way choice. Why this step? Explains why removing NSP barely moved BERT's total loss — it was always a small slice, consistent with the "≈1.2 point" MNLI effect in the parent note.

Verify: $2.10 + 0.625 = 2.725$ ✓; NSP share $0.625/2.725 \approx 22.9%$ ✓ — modest, matching the small measured downstream effect.

Figure s03 is the decision picture: it drops the four "logit-sign" examples (Ex 3 correct-but-unsure, Ex 4 wrong, Ex 5 tie, Ex 6 both limits) as dots onto the very same loss curve. Reading it left to right you literally see the ranking we proved: confident-correct sits near $0$ , the tie sits at $lo g 2 \approx 0.69$ , correct-but-unsure just below it, wrong just above it, and confident-wrong shoots toward the top. That one image is the summary of the whole page.

Recall Quick self-test

Which row/index of the NSP head corresponds to IsNext? ::: Row 1 / index 1 (row 0 / index 0 is NotNext). A model outputs $z_{0} = z_{1}$ . What is the NSP loss no matter the true label? ::: $lo g 2 \approx 0.693$ . If $P (IsNext) \to 0$ but the true label is IsNext, what happens to the loss? ::: It diverges to $+ \infty$ (confidently wrong = most expensive). Why does BERT add MLM and NSP losses instead of choosing one? ::: A sum's gradient is the sum of gradients, so each head trains from its own signal while sharing one Transformer body (multi-task learning). What does SOP stand for and how does it differ from NSP? ::: Sentence Order Prediction — it swaps two same-document consecutive sentences and asks if the order is correct, removing NSP's easy topic-shift cue.