This page assumes you have seen none of that machinery. We build every symbol the parent note uses, one at a time, each anchored to a picture, so that when you return to Next sentence prediction not a single squiggle is a stranger.
Figure s01 — A single document with its sentences listed on the left. Teal box: sentence A pulled out as the "context". Orange box: the genuine next sentence (label IsNext). Dashed plum box: a random impostor sentence yanked from a different document (label NotNext). Look at the three arrows leaving the document — they show the two ways B can be chosen.
A document is just text a human wrote, one sentence after another. NSP grabs two sentences from that pile — call the first one A and the second one B — and glues them into one input. The whole game is deciding whether B is the true neighbour of A (they were next to each other) or an impostor pulled from somewhere random.
The impostor arrow in the figure is not thrown in at random amounts — BERT builds each training batch with a deliberate coin-flip.
Why does the topic need this? Because tasks the model will later do — answering questions, checking if one sentence contradicts another (NLI, QA) — are all "compare two pieces of text" tasks. NSP is a rehearsal for exactly that shape of problem.
A computer cannot hold the word "purred"; it holds numbers. So the first thing that happens is tokenization — cutting text into small chunks called tokens (whole words, or word-pieces like super + ##position). See Masked language modeling for where this chopping comes from.
Figure s02 — The sentence "They leverage superposition." above, and its token boxes below, connected by a downward arrow labelled "tokenizer chops into pieces". Notice super and ##position are separate boxes: the ## (teal) marks a piece glued to the one before it. The plum boxes [CLS] and [SEP] are control markers, not real words.
Why does the topic need this? Everything downstream — hidden states, the matrix multiply — operates on a fixed list of positions. Tokens give us those positions.
A token is still just a name (like cat or [CLS]). The model needs a vector of numbers for each one. Three separate lookups produce three vectors, and their sum is what actually flows into the network.
Figure s05 — Three stacked rows of small vectors for the tokens [CLS] the cat .... Top row (teal): token embeddings, looked up by the token's name. Middle row (plum): position embeddings, looked up by the slot number 0,1,2,... Bottom row (orange): segment embeddings (0 for A, 1 for B). The big plus signs and the arrow pointing down show that all three are added column-by-column into one input vector per token.
The third of these — the segment embedding — is the one NSP leans on hardest, so it gets its own section next.
We just met segment embeddings as one of the three lookups. Zoom in on them, because they are how the model keeps sentence A and sentence B from blurring together.
Figure s03 — The token sequence [CLS] the cat sat [SEP] it purred . [SEP] in the top row, coloured teal (segment A) then orange (segment B). Directly below each token, a shaded cell shows its segment id: 0 for every token up to and including the first [SEP], then 1 for the rest. The colour split makes the A/B boundary visible at a glance.
Why does the topic need this? NSP is fundamentally about the relationship between two segments. If the model can't tell where A stops and B starts, the question "does B follow A?" is meaningless.
The parent turns the summary vector into a score for each of the two answers with:
z=WNSPhCLS+bNSP
Let's earn every piece.
Figure s04 — Left: the summary vector h_CLS as a column of 4 teal cells. Middle: the 2×4 plum grid W_NSP, row0 for NotNext and row1 for IsNext. An arrow labelled "W h + b" leads to the two logits z (NotNext = -0.01, IsNext = -0.15). A second orange arrow labelled "softmax" turns those into probabilities P(NotNext)=0.535 and P(IsNext)=0.465 that sum to 1. Follow the arrows left-to-right to see one row of W become one score.
Why a single linear layer? Because the heavy lifting — mixing A and B, resolving pronouns — was already done inside the model. By the time we reach hCLS, only a simple weighted vote is needed to read off the answer.
Two raw scores like z=[−0.01,−0.15] are hard to interpret. Softmax squashes them into two positive numbers that add to 1 — a genuine probability split.
The parent's numbers: z=[−0.01,−0.15] give P(NotNext)≈0.535, P(IsNext)≈0.465.
The chart below is a dependency map: read each box as one foundation this page built, and each arrow as "you need the box at the tail before the box at the head makes sense". Follow the arrows top-to-bottom and you retrace the exact order of sections above — raw tokens and special markers first, then the three embeddings that sum into an input vector, then the hidden state, the matrix multiply, softmax, and finally the loss. Everything funnels into the single bottom node, Next Sentence Prediction: that is the point of the whole climb.
Related objectives that reuse the same plumbing: Sentence order prediction, and the architecture that hosts it all, BERT (later trimmed by RoBERTa and adapted via transfer learning).
What is a token, and why can't the model use raw text?
A token is one chopped-up unit of text (a word or word-piece); the model only works on ordered lists of numbered pieces, not raw characters.
What does [CLS] do, and where is it placed?
It is a special summary slot placed at the very front whose final hidden state becomes the whole-pair representation used for the yes/no answer.
What does [SEP] do?
It marks the boundary after A and after B, telling the model where each segment ends.
What three vectors are summed to form a token's input embedding?
The token embedding (which word), the position embedding (which slot), and the segment embedding (A or B).
Why add the embeddings instead of stacking them side by side?
Adding keeps every input vector the same width d_model so downstream shapes never change, while each table still learns its own directions in that shared space.
What is a segment embedding?
A learnable vector added to each token encoding whether it belongs to sentence A (id 0) or B (id 1).
What is the 50/50 sampling rule and why is it used?
Half of training pairs keep the true next sentence (IsNext), half swap in a random one (NotNext); the balance stops a lazy model from always guessing one label and forces it to read A and B.
What is hCLS and what is dmodel?
The hidden-state vector in the CLS slot summarising the pair; dmodel is how many slots (numbers) that vector has, e.g. 768.
What shape is WNSP and why?
A 2×dmodel matrix — 2 rows because there are 2 answers (IsNext/NotNext), dmodel columns to dot against every slot of h.
What are logits z?
The raw un-normalised scores (one per answer), possibly negative, before softmax.
Why apply softmax, and what property does it guarantee?
It uses the exponential to make scores positive and normalises them so the two outputs are positive and sum to 1 — real probabilities.
How does cross-entropy loss punish a confident wrong answer?
The correct-label term is −logP(correct); if that probability is near 0 the log is a large negative number, so the loss becomes huge.
What is Ltotal?
The sum LMLM+LNSP, training word-level and pair-level skills simultaneously.
