Next sentence prediction

4.2.8 · AI-ML › Tokenization & Language Modeling

Next Sentence Prediction (NSP) ek binary classification pre-training objective hai jisme ek model seekhta hai ki do sentences kisi document mein consecutively aate hain ya randomly pair kiye gaye hain. BERT (2018) mein introduce kiya gaya, NSP ka maqsad tha models ko token-level patterns se aage sentence-level coherence sikhana.

Why NSP Exists

Motivation:

• Downstream tasks ko pair reasoning chahiye: QA ("Kya ye passage is question ka answer deta hai?") aur NLI ("Kya sentence B, sentence A ko contradict karta hai?") jaise tasks mein do text segments ki comparison zaroori hai.
• Document structure mayne rakhta hai: Consecutive sentences context share karte hain, pronouns boundaries ke paar resolve hote hain, aur topics logically flow karte hain.
• Pre-training ko fine-tuning se milna chahiye: Agar model pre-training ke dauran sentence pairs dekhta hai, toh wo pair-based downstream tasks ke liye jaldi adapt karta hai.

How NSP Works

Construction from First Principles

Step 1: Data Preparation

Ek document corpus se, training pairs banao:

1. Positive examples (50%): Ek sentence A chunno, uske actual baad wala sentence B lo → label = IsNext
2. Negative examples (50%): Ek sentence A chunno, poore corpus mein se uniformly ek random sentence B sample karo → label = NotNext

Random negatives kyun? BERT mein, negative sentence B corpus ke saare documents mein se uniformly draw kiya jaata hai. Kyunki zyaadatar randomly sampled sentences alag documents se alag topics ke saath aate hain, inhe distinguish karna usually aasaan hota hai, jo model ko coarse coherence seekhne ke liye kaafi strong signal banata hai. (Note: kyunki sampling poore corpus mein uniform hai, ek negative B kabhi-kabhi same document se bhi aa sakta hai, lekin ye rare aur unintentional hai.) Baad ka kaam jaise ALBERT ka SOP yeh argue karta hai ki harder, same-document negatives deeper coherence modeling ke liye majboor karte hain.

Step 2: Model Architecture

BERT ki structure use karte hue:

1. Input format: [CLS] Sentence_A [SEP] Sentence_B [SEP]

   • [CLS] ek special classification token hai jiska final hidden state pair representation ko aggregate karta hai
   • [SEP] do segments ko alag karta hai
   • Segment embeddings A vs B ke tokens mein distinguish karte hain

2. Forward pass: ${\mathbf{h}}_{\text{CLS}}=\text{Transformer}(\text{input tokens}{)}_0$ [CLS] token ka final hidden state extract karo (dimension $d_{\text{model}}$, e.g., 768).

3. Classification head: $\mathbf{z}={\mathbf{W}}_{\text{NSP}}{\mathbf{h}}_{\text{CLS}}+{\mathbf{b}}_{\text{NSP}}$ jahan ${\mathbf{W}}_{\text{NSP}}\in {\mathbb{R}}^{2\times d_{\text{model}}}$ (2 classes: IsNext/NotNext).

4. Output probabilities: $P(\text{IsNext}\mid A,B)=\text{softmax}(\mathbf{z}{)}_1$

Ye step kyun? [CLS] token dono sentences ke saare tokens ko attend karta hai, isliye uska representation combined context capture karta hai. Ek simple linear layer kaafi hai kyunki Transformer ne pehle se complex interactions seekh li hain.

Step 3: Loss Function

Binary cross-entropy: ${\mathcal{L}}_{\text{NSP}}=-{\mathbb{E}}_{(A,B,y)}\left[y\log P(\text{IsNext})+(1-y)\log P(\text{NotNext})\right]$

jahan $y\in \{0,1\}$ true label hai.

Derivation: Standard classification loss. Ise minimize karna correct label ke liye high probability encourage karta hai. Masked LM loss ke saath combine karke: ${\mathcal{L}}_{\text{total}}={\mathcal{L}}_{\text{MLM}}+{\mathcal{L}}_{\text{NSP}}$

Training Procedure

Multi-task learning: BERT NSP aur MLM par simultaneously train karta hai. Har batch mein dono tasks hote hain:

• A aur B mein 15% tokens MLM ke liye mask kiye jaate hain
• Wahi input NSP ke liye classify kiya jaata hai

Simultaneously kyun? Token-level (MLM) aur sentence-level (NSP) objectives complementary signals dete hain. MLM semantics sikhata hai; NSP discourse structure sikhata hai.

Corpus:

Doc1: "The cat sat on the mat. It purred loudly."
Doc2: "Quantum computers use qubits. They leverage superposition."

Positive pair (IsNext):

• A: "The cat sat on the mat."
• B: "It purred loudly."
• Label: 1 (IsNext)
• Ye kyun kaam karta hai? "It" "cat" ko refer karta hai, anaphora dikhata hai. Topic continuity clear hai.

Negative pair (NotNext):

• A: "The cat sat on the mat."
• B: "They leverage superposition." (corpus se randomly sampled, yahan Doc2 mein jaake mila)
• Label: 0 (NotNext)
• Ye kyun kaam karta hai? Abrupt topic shift (cats → quantum computing). "They" resolve nahi hota. Model ye disjointness seekhta hai.

Negative pair ke liye tokenized input:

Tokens:   [CLS] the cat sat on the mat . [SEP] they leverage super ##position . [SEP]
Segment:    0    0   0   0   0  0   0  0   0     1     1      1       1       1   1

Segment A mein 9 tokens hain ([CLS] se pehle wale [SEP] tak) jinhe 0 label kiya gaya hai, aur segment B mein 6 tokens hain (they se final [SEP] tak) jinhe 1 label kiya gaya hai. Segment embeddings (0 vs 1) model ko A aur B mein distinguish karne mein help karte hain.

Simplified $d_{\text{model}}=4$ assume karo. Transformer ke baad: ${\mathbf{h}}_{\text{CLS}}=[0.2,-0.5,0.8,0.1]$

NSP classifier weights (random initialization shown): ${\mathbf{W}}_{\text{NSP}}=\left[\begin{array}{cccc}0.3& 0.1& -0.2& 0.4\\ -0.1& 0.5& 0.3& -0.2\end{array}\right],{\mathbf{b}}_{\text{NSP}}=\left[\begin{array}{c}0.1\\ -0.1\end{array}\right]$

Logits: $\mathbf{z}={\mathbf{W}}_{\text{NSP}}{\mathbf{h}}_{\text{CLS}}+{\mathbf{b}}_{\text{NSP}}$

Class NotNext ke liye (row 0): $z_0=0.3(0.2)+0.1(-0.5)+(-0.2)(0.8)+0.4(0.1)+0.1=0.06-0.05-0.16+0.04+0.1=-0.01$

Class IsNext ke liye (row 1): $z_1=-0.1(0.2)+0.5(-0.5)+0.3(0.8)+(-0.2)(0.1)-0.1=-0.02-0.25+0.24-0.02-0.1=-0.15$

Softmax: $P(\text{NotNext})=\frac{e^{-0.01}}{e^{-0.01}+e^{-0.15}}=\frac{0.99}{0.99+0.86}\approx 0.535$ $P(\text{IsNext})=0.465$

Ye step kyun? Model NotNext ko 53.5% confidence ke saath predict karta hai. True label NotNext (y=0) ke liye, loss hai $-\log (0.535)\approx 0.63$. Backprop is probability ko badhane ke liye weights adjust karega.

Effectiveness and Controversy

Difference: ≈1.2 points gain. Modest lekin tasks mein consistent.

Baad ki research ne NSP ko challenge kiya:

• RoBERTa (2019) ne NSP hataya, dikhaya ki kuch benchmarks par performance drop nahi hua ya gains bhi the.
• Discrepancy kyun? RoBERTa ne longer sequences (multiple sentences span karte hue 512 tokens tak pack full sentences) use kiye, jo akele MLM ke through inter-sentence relationships implicitly capture kar sakte the. BERT ka NSP benefit partly uske specific data setup se aaya tha.

Sentence Order Prediction (SOP): ALBERT (2020) ne NSP ko SOP se replace kiya (predict karo ki do consecutive sentences correct order mein hain ya swapped order mein). SOP mushkil hai (dono segments same document se aate hain, bas possibly reversed) aur better gains dikhaye.

Key insight: NSP BERT ke specific training setup ke liye beneficial tha, lekin universally zaroori nahi.

Galat idea: "NSP model ko narrative flow, causal reasoning, aur complex discourse structure samajhna sikhata hai."

Ye sahi kyun lagta hai: Task ka naam hai 'next sentence prediction'—aise lagta hai jaise ye model karta hai ki ideas text mein kaise develop hote hain.

Haqeeqat: NSP negative examples bahut aasaan hote hain. Randomly sampled sentences zyaadatar topic, entities, aur style mein alag hote hain. Model aksar NSP ko topic shifts detect karke (word overlap) solve karta hai, coherence ke baare mein reasoning karne ki bajay.

Evidence:

• Pre-training ke dauran high NSP accuracy (97%+) suggest karta hai ki shallow heuristics kaafi hain.
• NLI (jo deep reasoning require karti hai) par fine-tuning BERT ki kaafi help karta hai, matlab NSP ne inter-sentence reasoning poori tarah solve nahi ki.

Fix: Harder negatives use karo (SOP, ya same-document non-consecutive/swapped sentences) taaki genuine coherence modeling force ho. Ya longer context windows par rely karo jahan MLM naturally cross-sentence dependencies capture karta hai.

Galat idea: "NSP next sentence predict karta hai jaise GPT next token predict karta hai."

Ye sahi kyun lagta hai: Dono mein 'next' aur 'prediction' hain.

Fix:

• NSP: Binary classification (haan/nahi: kya B next sentence hai?). Bidirectional context (BERT A aur B dono dekhta hai).
• Autoregressive next-token: Ek waqt mein ek token generate karta hai, left-to-right, $P(w_t|w_{<t})$ predict karta hai. Unidirectional.

NSP pair understanding ke liye ek classification objective hai. Autoregressive modeling sequential synthesis ke liye ek generation objective hai.

Implementation Details

Equal weight kyun? Empirical choice. NSP MLM se jaldi converge karta hai (simpler task), lekin dono useful gradients contribute karte hain.

Pseudocode

def create_nsp_example(documents, max_seq_len):
    doc = random.choice(documents)
    sentences = split_sentences(doc)
 
    i = random.randint(0, len(sentences) - 2)
    sent_a = sentences[i]
    if random.random() < 0.5:
        # IsNext
        sent_b = sentences[i + 1]
        label = 1
    else:
        # NotNext: sample a random sentence uniformly from the whole corpus
        other_doc = random.choice(documents)
        sent_b = random.choice(split_sentences(other_doc))
        label = 0
    tokens = ['[CLS]'] + tokenize(sent_a) + ['[SEP]'] + tokenize(sent_b) + ['[SEP]']
    segment_ids = [0] * (len(tokenize(sent_a)) + 2) + [1] * (len(tokenize(sent_b)) + 1)
 
    return tokens[:max_seq_len], segment_ids[:max_seq_len], label

Ye kyun important hai: 50/50 split balanced classes ensure karta hai. max_seq_len par truncation memory overflow rokta hai.

Next Sentence Prediction ek computer ko wahi kaam karna sikhata hai. Wo do sentences padhta hai aur kehta hai, "Haan, ye saath mein fit hote hain" ya "Nahi, ye bilkul alag jagahon se hain."

Kyun? Kyunki bahut saare computer tasks mein do pieces of text compare karne padte hain—jaise "Kya ye answer is question se match karta hai?" Agar computer ne seekha ki sentences kab saath mein fit hote hain, toh wo un comparison tasks mein behtar ho jaata hai. Ye real tests ki practice exercise hai!

CLS = "Coherence Level Signal": [CLS] token collect karta hai info ki kya ye pair sense banata hai.

Connections

• 4.2.06-Masked-language-modeling: BERT pre-training mein NSP ka partner objective
• 4.2.09-Sentence-order-prediction: NSP ka ALBERT ka improved replacement
• 4.3.02-BERT-architecture: Model structure jo NSP implement karta hai
• 4.3.08-RoBERTa: Dikhaya ki better training ke saath NSP hatane se koi nuksan nahi
• 5.1.04-Transfer-learning-NLP: Transfer ke liye pre-training strategy ke roop mein NSP
• 4.2.03-Segment-embeddings: BERT sentence A aur B mein kaise distinguish karta hai
• 6.2.01-Question-answering-systems: Downstream task jo pair reasoning se benefit karta hai
• 6.3.02-Natural-language-inference: Ek aur pair-based task jise NSP target karta hai

#flashcards/ai-ml

Concept Map