Next sentence prediction: Visual walkthrough — AI-ML deep dive

This page rebuilds Next Sentence Prediction from absolute zero, in pictures. By the end you will see, step by step, how two chunks of text get squeezed into a single number between 0 and 1 that answers one question: did sentence B really come right after sentence A?

We assume nothing. Every symbol is drawn before it is used.

Step 1 — What are we even trying to compute?

WHAT. A probability is just a confidence dial. If it reads $1.0$ , the model is fully sure B follows A. If it reads $0.0$ , it is fully sure B does not follow A. If it reads $0.5$ , it is guessing.

WHY. Later tasks — like question answering and natural language inference — are all about relating two sentences. If we can teach a model to judge "does B follow A?", we give it a head start on "does B answer A?" or "does B contradict A?".

PICTURE. Look at the two boxes. The green wire (real next sentence) and the red wire (random sentence) both feed a machine that spits out one dial reading.

Step 2 — Gluing A and B into one strip of tokens

WHAT. Before any maths, we lay A and B end to end and mark the seams with special tokens (a token is one chunk of text — a word or word-piece).

[CLS] — a blank token bolted to the very front. Its job: after all the processing, its slot will hold the summary of the whole pair. Think of it as an empty scoreboard.
[SEP] — a fence token that says "sentence ends here".

WHY the [CLS] token? We need one place to read the final answer from. Instead of inventing a new slot, BERT reserves position 0 for it from the start, so the machine learns to pour the summary there. (This is the same trick used in BERT's architecture.)

PICTURE. The strip below shows the layout. Notice the segment row underneath: every token in A is stamped $0$ , every token in B is stamped $1$ . Those stamps are the segment embeddings — they let the model tell "which side am I on?".

Step 3 — Where do the training pairs come from?

WHAT. Half the time we build an honest pair; half the time we build a fake one.

50% IsNext: pick sentence A, take the actual sentence right after it as B. Label $y = 1$ .
50% NotNext: pick sentence A, then grab a B sampled uniformly from the whole corpus. Label $y = 0$ .

WHY 50/50? If we fed mostly IsNext pairs, the model could score high just by always guessing "IsNext". A balanced diet forces it to actually look.

WHY random negatives? A random sentence almost always comes from a different document about a different topic. That makes NotNext usually easy — a strong, clean signal to learn from. (The catch, unpacked later: too easy. That is the seed of Sentence Order Prediction.)

PICTURE. The green arrow keeps B adjacent to A; the red arrow yanks B out of a faraway document.

Step 4 — Squeezing the whole pair into one vector

WHAT. We push the token strip through the Transformer (the big stack of attention layers from BERT). Out the other end, every token position gets a fresh vector. We only keep position 0 — the [CLS] slot.

$h_{CLS} = Transformer (input)_{0}$

$Transformer (\cdot)$ — the machine that lets every token attend to (peek at) every other token.
the subscript $0$ — "take the output at position 0", i.e. the [CLS] scoreboard.
$h_{CLS}$ — a list of $d_{model}$ numbers (e.g. 768). A vector is just an ordered list of numbers; picture it as an arrow / a bar chart of heights.

WHY only [CLS]? Because [CLS] attended to every token in both A and B, its final vector already blends the whole pair. We do not need to read every position — the summary lives in one place.

PICTURE. Watch the arrows: [CLS] reaches into all tokens of A and B, and the mixed result becomes the bar chart on the right — that bar chart is $h_{CLS}$ .

Step 5 — From a vector to two raw scores (logits)

WHAT. We now collapse the $d_{model}$ -long vector down to just two numbers — one score for "NotNext", one for "IsNext".

$z = W_{NSP} h_{CLS} + b_{NSP}$

Term by term:

$W_{NSP}$ — a $2 \times d_{model}$ table of learned weights. Row 0 measures "how NotNext-ish is this vector?", row 1 measures "how IsNext-ish?".
multiplying $W_{NSP} h_{CLS}$ — each row takes a weighted sum of the vector's entries: multiply matching numbers, add them up. It answers "how much does this vector point in my direction?".
$b_{NSP}$ — a length-2 bias, a baseline nudge added to each score.
$z = (z_{0}, z_{1})$ — the two logits (raw, un-normalised scores; they can be any real number, positive or negative).

WHY a plain linear layer (and not something fancy)? Because the Transformer already did the hard non-linear mixing. All that is left is to read off two directions from the summary vector — and a straight-line map (a matrix) is exactly the tool for "project a vector onto a couple of directions".

PICTURE. Each row of $W$ is a comb sliding along the bars of $h_{CLS}$ , multiplying and summing into a single score.

Step 6 — Turning two scores into a probability with softmax

WHAT. Logits can be any numbers. We need a probability. Softmax is the tool that turns a pair of scores into two positive numbers that add to 1.

$P (IsNext) = \frac{e ^{z_{1}}}{e ^{z_{0}} + e ^{z_{1}}}, P (NotNext) = \frac{e ^{z_{0}}}{e ^{z_{0}} + e ^{z_{1}}}$

Term by term:

$e^{z}$ — the exponential. WHY exponentiate? Two reasons: it makes every score positive (probabilities can't be negative), and it exaggerates gaps — a slightly higher logit becomes a clearly higher probability.
the denominator $e^{z_{0}} + e^{z_{1}}$ — the total, so dividing forces the two outputs to sum to exactly 1.

PICTURE. Two raw bars (that could be negative) go into the exp-and-normalise machine and come out as two bars that fit inside a single unit-height box.

Step 7 — Scoring the guess: the loss

WHAT. We measure how wrong the model was with binary cross-entropy:

$L_{NSP} = - [y lo g P (IsNext) + (1 - y) lo g P (NotNext)]$

Term by term:

$y$ — the true label (1 or 0). The two terms are a switch: if $y = 1$ only the first term survives; if $y = 0$ only the second.
$lo g P (\cdot)$ — the logarithm. WHY log? Because $lo g 1 = 0$ (perfect confidence, no penalty) and $lo g (small) \to - \infty$ (confident and wrong is punished hard). The leading minus flips it so loss is $\geq 0$ .

WHY this shape? Minimising $L$ literally means "push the probability of the correct label toward 1".

PICTURE. The loss curve dives to 0 as the correct probability approaches 1 and rockets up as it approaches 0.

Step 8 — The edge case that broke NSP

WHAT. Consider the degenerate situation the parent note warns about: what if the model can win without understanding coherence at all?

WHY it happens. Random negatives usually flip the topic. So a lazy detector — "do A and B share vocabulary?" — already separates most IsNext from NotNext. The model reaches 97%+ accuracy using this shallow shortcut.

PICTURE. On the left, easy random negatives are far apart (a wide, trivial gap). On the right, the hard same-document swapped negatives of SOP sit right on top of the positives — no vocabulary shortcut survives, forcing real coherence reasoning. This is exactly why RoBERTa could drop NSP with no loss, and why transfer learning to NLI still needed genuine fine-tuning.

The one-picture summary

Everything above is one pipeline: glue → embed → mix → project → normalise → score.

Recall Feynman retelling — say it back in plain words

We take two sentences and tape them together with a blank scoreboard token in front and fences between them, stamping each side with a 0 or a 1 so the machine knows which is which. Half our examples are honest neighbours, half are random strangers. We shove the whole strip through the big attention machine; because the scoreboard token peeked at every word, its final list of numbers is a summary of the pair. We slide two combs (learned weight rows) over that summary to get two scores — one for "strangers", one for "neighbours". We can't read scores as chances, so we exp-and-normalise them into two numbers that add to one. We compare the "neighbours" chance against the truth: if we were confidently right, no penalty; confidently wrong, huge penalty. That penalty flows backward and tweaks every knob. The twist: random strangers are so obviously off-topic that the machine can cheat by just spotting the topic change — which is why later methods swap in same-document, order-flipped negatives to force it to actually think.

Recall Quick self-test

Why keep only the [CLS] output vector? ::: Because it attended to all tokens of A and B, so its final vector already summarises the whole pair — one place to read the answer. Why exponentiate the logits in softmax? ::: To make them positive and exaggerate gaps, so dividing by the total gives valid probabilities that sum to 1. Our example gave $P (NotNext) \approx 0.535$ ; what is the loss if the true label is NotNext? ::: $- lo g (0.535) \approx 0.626$ . Why did NSP turn out to be an easy task? ::: Random negatives usually differ in topic, so a shallow vocabulary-overlap heuristic already separates them — no deep coherence needed.