BERT and encoder models: Foundations — AI-ML deep dive

This page builds every symbol, letter, and piece of notation the parent note BERT and encoder models throws at you — starting from a smart 12-year-old who has seen none of it. Read top to bottom; each block earns the next.

0. What is a "token"? (before any math)

Look at Figure 1. The sentence gets sliced into little boxes (top row). Each box is a token, and directly beneath it sits its numbered slot (bottom row). This matters because everything below counts tokens, never words.

1. The vocabulary $V$

Before any subscripts or vectors, we need the pool that all tokens come from.

Why the topic needs it: the embedding table, the softmax, and the MLM head all have a size tied to $∣ V ∣$ . So $V$ must come first — every later shape refers back to it.

2. The subscript: $x_{1}, x_{2}, \dots, x_{n}$

Look again at Figure 1: the bottom row of teal boxes is exactly this — the numbered slots $x_{1}, x_{2}, \dots$ . So $x_{i}$ just means "look inside slot number $i$ ". The letter $i$ is a placeholder for "some slot we're pointing at right now".

$n$ ::: total number of tokens (the length of the sequence).
$i$ ::: a stand-in for "whichever position we're currently talking about".

Why the topic needs it: BERT processes a sequence, so we must be able to say "the 3rd token" precisely. Subscripts are how.

3. Vectors, boldface, and $R^{d}$

The little $\in$ symbol means belongs to / "is a member of". Figure 2 shows a 3-number vector as an arrow in space so you can see what a list of numbers looks like.

$d$ ::: how long each meaning-vector is (BERT-base uses $d = 768$ ).
$R^{768}$ ::: a list of 768 decimal numbers.
bold vs plain ::: bold = a list (vector); plain italic = one number.

Why the topic needs it: the parent says $h_{i} \in R^{d}$ and $W \in R^{2 \times 768}$ . You cannot read those until "list of numbers" (and what bold signals) is solid.

4. How a token $x_{i}$ becomes a numeric vector: the embedding matrix $E$

Before self-attention can mix anything, each raw token has to be turned into numbers. A token like play is really just an ID number — its row-number in the vocabulary $V$ (defined in §1). We look that row up in a big table.

So the pipeline is: raw token $x_{i}$ → look up in $E$ (+ position + segment) → a numeric input vector → then the Transformer.

5. From input vector to output $h_{i}$

The key word is contextual: the same token gets a different $h_{i}$ depending on its neighbours. "bank" near "river" and "bank" near "money" produce different $h_{i}$ . This is the whole point of self-attention and the Transformer.

Why the topic needs it: every BERT task reads $h_{i}$ , not the raw token. MLM predicts hidden words from $h_{i}$ , sentiment reads $h_{[CLS]}$ , tagging reads each $h_{i}$ . Without the context-aware $h_{i}$ , "bank" would have one frozen meaning — and the entire bidirectional idea in the parent note would collapse.

6. Matrix–vector product $W h_{i}$ — and which shape of $W$

This is the task head: a small layer bolted on top of $h_{i}$ . The shape of $W$ depends on the job:

Head	Question asked	Rows of $W$	Shape
Sentiment (2 classes)	pos or neg?	2	$W \in R^{2 \times d}$
NSP (IsNext yes/no)	is B the real next sentence?	2	$W \in R^{2 \times d}$ (or 1 row + sigmoid)
MLM (predict token)	which word of the vocab?	$	V

So $W \in R^{2 \times 768}$ is a two-class head (2 rows, 768 columns). For MLM the very same idea is used but the head must output one score per vocabulary word, so it has $∣ V ∣$ rows — one per possible answer. Figure 3 draws the two-class collapse from 768 down to 2; picture the MLM head as the same funnel but widening to $∣ V ∣$ outputs.

$b$ (the "bias") ::: a small fixed vector added on after, to shift the scores.
$z_{i} = W h_{i} + b$ ::: the raw logits (unnormalised scores) for token $i$ ; it has one entry per output the head produces.

Why the topic needs it: turning a rich 768-number meaning into task scores is exactly what every head does — 2 scores for sentiment/NSP, $∣ V ∣$ scores for MLM.

7. Logits, $e^{x}$ , and softmax

To turn scores into probabilities we need two things: (a) make everything positive, (b) make them add to 1.

Reading the pieces:

$V$ ::: the vocabulary (from §1) — the full list of tokens the model can output.
$\sum_{v \in V}$ ::: "add this up, letting $v$ run over every token in the vocabulary". The big $\sum$ (Greek capital sigma) just means sum.
$p_{w} = softmax (z)_{w}$ ::: the resulting probability of word $w$ ; the collection of all $p_{w}$ is the model's probability distribution.

Figure 4 shows three logits becoming three bars that add to 1 — the exponential stretches the gaps, then division normalises.

8. Logarithm and the minus sign: $- lo g p$

Here $p$ is the softmax probability from §7 — specifically $p = p_{y}$ , the probability the model gave to the true answer $y$ . Loss punishes the model when this number is small.

$p = 1$ (perfect) $\Rightarrow - lo g 1 = 0$ (no penalty).
$p = 0.5 \Rightarrow - lo g 0.5 = 0.693$ .
$p \to 0$ (confidently wrong) $\Rightarrow - lo g p \to \infty$ (huge penalty).

This $- lo g p$ is the heart of cross-entropy loss, the engine behind both MLM and NSP.

9. The `[MASK]` token, the masked set $M$ , and the MLM loss formula

The `[MASK]` token

The masked set $M$

The MLM loss

Now every symbol is earned, so we can write the actual loss the parent note uses. We grade only the masked slots and average their $- lo g$ penalties:

10. Remaining symbols: $L$ , $σ$ , $\overset{p}{^}$ , $y$ — and the NSP head

11. Special tokens `[CLS]` and `[SEP]`

These aren't real words — they're signposts BERT invents so it can do sentence-level jobs. The subscript in $h_{[CLS]}$ is not a number; it just names which slot (the [CLS] slot).

Prerequisite map

Read it top to bottom: raw tokens are looked up in $E$ to become number-lists, self-attention mixes them into contextual $h_{i}$ , a matrix turns $h_{i}$ into scores, softmax makes probabilities $p_{w}$ , $- lo g p$ grades them, we average over the masked set $M$ , and that grade drives BERT's training.

Equipment checklist

Cover the right side. If you can answer each, you are ready for the parent note.

What is the vocabulary $V$ , and what does $∣ V ∣$ mean? ::: the full fixed list of tokens BERT knows; $∣ V ∣$ is how many there are (~30k).
What does the subscript $i$ in $x_{i}$ mean? ::: the position/slot of a token in the sequence.
What does boldface (e.g. $h_{i}$ ) signal versus a plain italic letter? ::: bold = a vector (a list of numbers); plain italic = a single number.
What is $R^{768}$ ? ::: the set of all lists of 768 real numbers; each meaning-vector lives here.
What is the embedding matrix $E$ and its shape? ::: a table with one row per vocabulary token and $d$ columns, $E \in R^{∣ V ∣ \times d}$ ; its row for a token is that token's embedding.
What three embeddings are summed to form the input vector? ::: token, position, and segment embeddings.
Difference between the input vector and $h_{i}$ ? ::: the input $e_{i} + p_{i} + s_{i}$ is context-free; $h_{i}$ is context-aware after self-attention.
What does $W h_{i} + b$ produce and why? ::: logits (raw scores) by mixing the 768 numbers into per-output scores.
Which shape does $W$ take for sentiment/NSP vs MLM? ::: $2 \times d$ for a two-class head; $∣ V ∣ \times d$ for MLM (one score per vocabulary word).
Why does softmax use $e^{z}$ ? ::: $e^{z}$ is always positive and amplifies larger scores, giving a clean probability distribution that sums to 1.
What is $\sum_{v \in V}$ in the softmax denominator? ::: the sum of the exponentials of every token's score (the normaliser).
What is $p$ in $- lo g p$ ? ::: the softmax probability $p_{y}$ the model assigned to the true answer $y$ .
Why the minus in $- lo g p$ ? ::: so the loss is positive and smaller means better; perfect $p = 1$ gives loss 0.
What is the [MASK] token and why isn't it used 100% of the time? ::: a placeholder inserted where a word is hidden; using it always would cause a train/test mismatch, so it's 80% MASK / 10% random / 10% unchanged.
Write the MLM loss formula in words. ::: average over the masked set $M$ of $- lo g$ of the true-token probability: $L_{MLM} = - \frac{1}{∣ M ∣} \sum_{i \in M} lo g p_{y_{i}}$ .
What is the set $M$ and what does $∣ M ∣$ mean? ::: $M$ is the set of masked slot-numbers; $∣ M ∣$ is how many were masked (the count of $M$ ).
When do we use $σ$ vs softmax, and what shape is the NSP head? ::: $σ$ (sigmoid) for a single yes/no probability like NSP (a $1 \times d$ row $w^{⊤} h_{[CLS]} + b$ ); softmax for choosing among many classes like MLM.
What is $h_{[CLS]}$ used for? ::: as the whole-sentence summary vector for classification (sentiment, NSP).

Next: return to BERT and encoder models and every symbol will now read as plain English. See also Fine-Tuning vs Pretraining, RoBERTa, GPT and decoder models, and T5 and encoder-decoder models.