Next sentence prediction

Next Sentence Prediction (NSP) is a binary classification pre-training objective where a model learns whether two sentences appear consecutively in a document or are randomly paired. Introduced in BERT (2018), NSP aimed to teach models sentence-level coherence beyond token-level patterns.

Why NSP Exists

The motivation:

• Downstream tasks need pair reasoning: Tasks like QA ("Does this passage answer this question?") and NLI ("Does sentence B contradict A?") require comparing two text segments.
• Document structure matters: Consecutive sentences share context, pronouns resolve across boundaries, and topics flow logically.
• Pre-training should mirror fine-tuning: If the model sees sentence pairs during pre-training, it adapts faster to pair-based downstream tasks.

How NSP Works

Construction from First Principles

Step 1: Data Preparation

From a document corpus, create training pairs:

1. Positive examples (50%): Choose a sentence A, take the actual following sentence B → label = IsNext
2. Negative examples (50%): Choose a sentence A, sample a random sentence B uniformly from the entire corpus → label = NotNext

Why random negatives? In BERT, the negative sentence B is drawn uniformly from all documents in the corpus. Because most randomly sampled sentences come from different documents with different topics, they are usually easy to distinguish, creating a signal strong enough for the model to learn coarse coherence. (Note: since sampling is uniform over the whole corpus, a negative B could occasionally come from the same document, but this is rare and unintentional.) Later work like ALBERT's SOP argued that harder, same-document negatives force deeper coherence modeling.

Step 2: Model Architecture

Using BERT's structure:

1. Input format: [CLS] Sentence_A [SEP] Sentence_B [SEP]

   • [CLS] is a special classification token whose final hidden state aggregates the pair representation
   • [SEP] separates the two segments
   • Segment embeddings distinguish tokens from A vs B

2. Forward pass: $\mathbf{h}_{\text{CLS}} = \text{Transformer}(\text{input tokens})_0$ Extract the final hidden state of [CLS] token (dimension $d_{\text{model}}$, e.g., 768).

3. Classification head: $\mathbf{z} = \mathbf{W}_{\text{NSP}} \mathbf{h}_{\text{CLS}} + \mathbf{b}_{\text{NSP}}$ where $\mathbf{W}_{\text{NSP}} \in \mathbb{R}^{2 \times d_{\text{model}}}$ (2 classes: IsNext/NotNext).

4. Output probabilities: $P(\text{IsNext} \mid A, B) = \text{softmax}(\mathbf{z})_1$

Why this step? The [CLS] token attends to all tokens in both sentences, so its representation captures the combined context. A simple linear layer suffices because the Transformer already learned complex interactions.

Step 3: Loss Function

Binary cross-entropy: $\mathcal{L}_{\text{NSP}} = -\mathbb{E}_{(A,B,y)} \left[ y \log P(\text{IsNext}) + (1-y) \log P(\text{NotNext}) \right]$

where $y \in \{0, 1\}$ is the true label.

Derivation: Standard classification loss. Minimizing this encourages high probability for the correct label. Combined with masked LM loss: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}}$

Training Procedure

Multi-task learning: BERT trains on NSP and MLM simultaneously. Each batch contains both tasks:

• 15% of tokens in A and B are masked for MLM
• The same input is classified for NSP

Why simultaneous? Token-level (MLM) and sentence-level (NSP) objectives provide complementary signals. MLM learns semantics; NSP learns discourse structure.

Corpus:

Doc1: "The cat sat on the mat. It purred loudly."
Doc2: "Quantum computers use qubits. They leverage superposition."

Positive pair (IsNext):

• A: "The cat sat on the mat."
• B: "It purred loudly."
• Label: 1 (IsNext)
• Why this works? "It" refers to "cat", showing anaphora. Topic continuity is clear.

Negative pair (NotNext):

• A: "The cat sat on the mat."
• B: "They leverage superposition." (randomly sampled from the corpus, here landing in Doc2)
• Label: 0 (NotNext)
• Why this works? Abrupt topic shift (cats → quantum computing). "They" doesn't resolve. Model learns this disjointness.

Tokenized input for negative pair:

Tokens:   [CLS] the cat sat on the mat . [SEP] they leverage super ##position . [SEP]
Segment:    0    0   0   0   0  0   0  0   0     1     1      1       1       1   1

There are 9 tokens in segment A ([CLS] through the first [SEP]) all labelled 0, and 6 tokens in segment B (they through the final [SEP]) all labelled 1. The segment embeddings (0 vs 1) help the model distinguish A from B.

Assume simplified $d_{\text{model}} = 4$. After Transformer: $\mathbf{h}_{\text{CLS}} = [0.2, -0.5, 0.8, 0.1]$

NSP classifier weights (random initialization shown): $\mathbf{W}_{\text{NSP}} = \begin{bmatrix} 0.3 & 0.1 & -0.2 & 0.4 \\ -0.1 & 0.5 & 0.3 & -0.2 \end{bmatrix}, \quad \mathbf{b}_{\text{NSP}} = \begin{bmatrix} 0.1 \\ -0.1 \end{bmatrix}$

Logits: $\mathbf{z} = \mathbf{W}_{\text{NSP}} \mathbf{h}_{\text{CLS}} + \mathbf{b}_{\text{NSP}}$

For class NotNext (row 0): $z_0 = 0.3(0.2) + 0.1(-0.5) + (-0.2)(0.8) + 0.4(0.1) + 0.1 = 0.06 - 0.05 - 0.16 + 0.04 + 0.1 = -0.01$

For class IsNext (row 1): $z_1 = -0.1(0.2) + 0.5(-0.5) + 0.3(0.8) + (-0.2)(0.1) - 0.1 = -0.02 - 0.25 + 0.24 - 0.02 - 0.1 = -0.15$

Softmax: $P(\text{NotNext}) = \frac{e^{-0.01}}{e^{-0.01} + e^{-0.15}} = \frac{0.99}{0.99 + 0.86} \approx 0.535$ $P(\text{IsNext}) = 0.465$

Why this step? The model predicts NotNext with 53.5% confidence. For true label NotNext (y=0), loss is $-\log(0.535) \approx 0.63$. Backprop will adjust weights to increase this probability.

Effectiveness and Controversy

Difference: ≈1.2 points gain. Modest but consistent across tasks.

Later research challenged NSP:

• RoBERTa (2019) removed NSP, showing no performance drop or even gains on some benchmarks.
• Why the discrepancy? RoBERTa used longer sequences (full sentences packed up to 512 tokens spanning multiple sentences), which may have implicitly captured inter-sentence relationships through MLM alone. BERT's NSP benefit came partly from its specific data setup.

Sentence Order Prediction (SOP): ALBERT (2020) replaced NSP with SOP (predict if two consecutive sentences are in correct or swapped order). SOP is harder (both segments come from the same document, just possibly reversed) and showed better gains.

Key insight: NSP was beneficial for BERT's specific training setup, but not universally necessary.

Wrong idea: "NSP makes the model understand narrative flow, causal reasoning, and complex discourse structure."

Why it feels right: The task is called 'next sentence prediction'—sounds like it models how ideas develop across text.

The reality: NSP negative examples are too easy. Randomly sampled sentences usually differ in topic, entities, and style. The model often solves NSP by detecting topic shifts (word overlap) rather than reasoning about coherence.

Evidence:

• High NSP accuracy (97%+) during pre-training suggests shallow heuristics suffice.
• Fine-tuning on NLI (which requires deep reasoning) still helps BERT significantly, meaning NSP didn't fully solve inter-sentence reasoning.

The fix: Use harder negatives (SOP, or same-document non-consecutive/swapped sentences) to force genuine coherence modeling. Or rely on longer context windows where MLM naturally captures cross-sentence dependencies.

Wrong idea: "NSP predicts the next sentence like GPT predicts the next token."

Why it feels right: Both involve 'next' and 'prediction'.

The fix:

• NSP: Binary classification (yes/no: is B the next sentence?). Bidirectional context (BERT sees all of A and B).
• Autoregressive next-token: Generates one token at a time, left-to-right, predicting $P(w_t | w_{<t})$. Unidirectional.

NSP is a classification objective for pair understanding. Autoregressive modeling is a generation objective for sequential synthesis.

Implementation Details

Why equal weight? Empirical choice. NSP converges faster than MLM (simpler task), but both contribute useful gradients.

Pseudocode

def create_nsp_example(documents, max_seq_len):
    doc = random.choice(documents)
    sentences = split_sentences(doc)
 
    i = random.randint(0, len(sentences) - 2)
    sent_a = sentences[i]
    if random.random() < 0.5:
        # IsNext
        sent_b = sentences[i + 1]
        label = 1
    else:
        # NotNext: sample a random sentence uniformly from the whole corpus
        other_doc = random.choice(documents)
        sent_b = random.choice(split_sentences(other_doc))
        label = 0
    tokens = ['[CLS]'] + tokenize(sent_a) + ['[SEP]'] + tokenize(sent_b) + ['[SEP]']
    segment_ids = [0] * (len(tokenize(sent_a)) + 2) + [1] * (len(tokenize(sent_b)) + 1)
 
    return tokens[:max_seq_len], segment_ids[:max_seq_len], label

Why this matters: The 50/50 split ensures balanced classes. Truncation to max_seq_len prevents memory overflow.

Next Sentence Prediction trains a computer to do that same check. It reads two sentences and says, "Yeah, these go together" or "Nope, these are from totally different places."

Why? Because lots of computer tasks need to compare two pieces of text—like "Does this answer match this question?" If the computer learned to spot when sentences fit together, it gets better at those comparison tasks. It's like a practice exercise for the real tests!

CLS = "Coherence Level Signal": The [CLS] token collects info about whether the pair makes sense.

Connections

• 4.2.06-Masked-language-modeling: NSP's partner objective in BERT pre-training
• 4.2.09-Sentence-order-prediction: ALBERT's improved replacement for NSP
• 4.3.02-BERT-architecture: Model structure that implements NSP
• 4.3.08-RoBERTa: Showed NSP removal doesn't hurt with better training
• 5.1.04-Transfer-learning-NLP: NSP as a pre-training strategy for transfer
• 4.2.03-Segment-embeddings: How BERT distinguishes sentence A from B
• 6.2.01-Question-answering-systems: Downstream task that benefits from pair reasoning
• 6.3.02-Natural-language-inference: Another pair-based task NSP targets

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)