Noesis
4.2.8 · D5Tokenization & Language Modeling

Question bank — Next sentence prediction

1,502 words7 min read↑ Back to topic

Before the questions, three words we lean on everywhere below. If any feel shaky, this is where to rebuild them.

Related pages you may want open: 4.2.06-Masked-language-modeling, 4.2.09-Sentence-order-prediction, 4.3.02-BERT-architecture, 4.3.08-RoBERTa, 4.2.03-Segment-embeddings, 6.3.02-Natural-language-inference, 6.2.01-Question-answering-systems, 5.1.04-Transfer-learning-NLP.

True or false — justify

NSP is a binary classification task.
True. The model chooses between exactly two labels (IsNext / NotNext), so it is a 2-way classifier, not a generator or a ranker.
NSP predicts which words come next in sentence B.
False. NSP never generates or predicts tokens; it only judges whether the already-given B is the genuine continuation of A. Word-level prediction is the job of masked LM, not NSP.
Half of NSP training pairs are labelled NotNext.
True. BERT builds a balanced 50/50 split — 50% real consecutive pairs (IsNext) and 50% pairs whose B is a randomly sampled sentence (NotNext) — so the model can't win by always guessing one label.
Because negatives are sampled uniformly over the whole corpus, a negative B can never come from the same document as A.
False. Uniform sampling usually lands in a different document, but nothing forbids the same document — it just happens rarely and by accident, not by design.
The [CLS] hidden state used by NSP already "sees" both sentence A and sentence B.
True. Transformer self-attention lets [CLS] attend to every token in both segments, so its final vector aggregates the whole pair — that's precisely why a single linear layer on top suffices.
NSP alone would train a strong language model without MLM.
False. NSP gives only one bit of signal per example (a coarse pair label); it teaches nothing about word semantics. BERT needs MLM running simultaneously to learn token-level meaning.
Removing NSP always hurts downstream accuracy.
False. RoBERTa dropped NSP entirely and saw no drop, sometimes gains — because its longer packed sequences let MLM capture cross-sentence structure on its own.
High NSP accuracy during pre-training proves the model deeply understands discourse.
False. 97%+ accuracy is evidence of a shallow solution: random negatives differ so much in topic that a topic-shift heuristic already scores near-perfect, so the number tells us little about real coherence reasoning.

Spot the error

"NSP is like GPT: both predict the next thing in the sequence, so they're the same idea."
Error: GPT does autoregressive next-token prediction (generating words left to right), while NSP is a binary judgment on a whole sentence pair. One generates, the other classifies — different mechanics and different training signals.
"We feed sentence A into the model for NSP, and separately feed sentence B, then compare."
Error: A and B go in together as one sequence [CLS] A [SEP] B [SEP]. They are never encoded separately — the whole point is that attention lets B's representation depend on A's.
"The segment embeddings mark which token is [CLS] versus a word."
Error: Segment embeddings mark whether a token belongs to segment A (0) or segment B (1), not CLS-vs-word. They let the model tell the two sentences apart even after mixing them.
"NSP uses cross-entropy over the 30k-word vocabulary."
Error: NSP's output space is size 2 (IsNext / NotNext), so its cross-entropy is over two classes. The vocabulary-sized softmax belongs to MLM, which predicts masked words.
"ALBERT's SOP is easier than NSP, that's why it helps."
Error: SOP is deliberately harder. Both its segments come from the same document, only possibly swapped in order — so topic overlap can't give the answer away, forcing genuine order/coherence modeling. See 4.2.09-Sentence-order-prediction.
"To predict IsNext we take the last token's hidden state before the final [SEP]."
Error: NSP reads the [CLS] token's final hidden state (position 0), not the last content token. [CLS] is the designated pair-summary slot.
"Since NSP only adds one linear layer, it must contribute most of BERT's power."
Error: The thin head contributes little — the heavy lifting is done by the shared Transformer. The BERT ablation showed NSP gives only about a 1.2-point MNLI gain, modest not dominant.

Why questions

Why does BERT pick negative sentences randomly rather than from the same document?
Random negatives are cheap and usually obviously wrong (different topic), giving a clean, easy learning signal. The downside — later fixed by SOP — is that "easy" means the model can cheat with topic shifts instead of coherence.
Why is a single linear classification layer enough on top of [CLS]?
Because the Transformer has already mixed A and B through attention and packed the relevant interaction into [CLS]. The remaining decision is nearly linearly separable, so a matrix suffices.
Why train NSP and MLM at the same time instead of one after the other?
They give complementary signals — MLM teaches word-level semantics, NSP teaches sentence-level (discourse) structure — and sharing one forward pass lets both gradients shape the same representation, which is cheaper and mutually reinforcing.
Why did NSP help BERT but not RoBERTa?
RoBERTa packed much longer inputs (up to 512 tokens spanning many sentences), so MLM alone already had to model cross-sentence dependencies. NSP's extra signal became redundant under that data setup.
Why is NSP relevant to tasks like NLI and QA?
Those tasks reason over pairs of text (premise/hypothesis, question/passage). Pre-training on pairs via NSP mirrors that structure, so fine-tuning (transfer learning) adapts faster — though evidence shows NSP didn't fully solve the deep reasoning NLI still requires.
Why does the loss use the true label to pick which log-probability to penalize?
Cross-entropy only rewards the probability mass placed on the correct answer; pushing that term up necessarily pushes the wrong label's probability down, steering the model toward correct, confident predictions.

Edge cases

What happens if the randomly sampled negative B does land right after A in the corpus?
It would be labelled NotNext but is actually a valid continuation — a rare label-noise case. Because it's infrequent, it barely perturbs training, but it is a genuine (unintended) flaw of uniform sampling.
What if segments A and B share almost all their vocabulary but B is still NotNext?
This is exactly where the topic-shift heuristic fails: high word overlap tempts the model to say IsNext. Such hard cases reveal that NSP-trained models often lack true coherence reasoning — motivation for SOP's same-document negatives.
If the two logits come out exactly equal, what does NSP predict?
Softmax of equal logits gives for each class — a maximally uncertain tie. In practice ties are near-impossible with real weights, but conceptually the model is expressing "no information", and the loss will be regardless of the true label.
What if segment B is empty (zero content tokens)?
The input degenerates to [CLS] A [SEP] [SEP]; there is nothing for [CLS] to compare A against, so any prediction is uninformed. Well-formed NSP data never does this — both segments must carry real sentences.
Can NSP handle three sentences (A, B, C) at once?
Not natively — NSP is defined for exactly two segments and one boundary. Multi-sentence coherence needs a different objective (SOP over ordered chunks) or simply longer packed MLM sequences.
What if every training pair were IsNext (no negatives)?
The task collapses: the model can score 100% by always predicting IsNext, learning nothing about coherence. The 50/50 balance is essential to make the label carry information.
Recall One-line summary to lock in

NSP is a coarse binary pair-coherence signal read off [CLS]; it helps BERT modestly, is easily gamed by topic shifts, and is superseded by SOP or long-context MLM.