Foundations — Masked language modeling (BERT)

This page assumes you know nothing. Before you can read the parent note, every squiggle it uses must first be a picture in your head. We build them one at a time, each from the one before.

0. What is a "token"?

Before any math, the atom of everything here is the token.

Picture a sentence as a row of boxes, one word per box:

Look at the figure. "The cat sat" becomes three boxes: [the] [cat] [sat]. Each box is a token. The model never sees letters — it sees this row of boxes.

Why the topic needs it: MLM works by hiding a box and asking the model to refill it. If we couldn't cut text into boxes, there would be nothing to hide.

1. Subscripts and the sequence $\mathbf{x}=(x_1,x_2,\dots ,x_n)$

Now that we have boxes, we need to name each box so we can point at one.

• $x_1$ = the token in box 1
• $x_2$ = the token in box 2
• $x_i$ = the token in box number $i$ (where $i$ is a stand-in for any number)
• $n$ = how many boxes there are in total (the length of the sentence)

Why the topic needs it: The MLM loss sums over specific boxes (the hidden ones). We can only say "box 3 and box 5" if every box has a number.

2. Sets and the masked positions $M$

We hide some boxes, not all. We need a way to say "the collection of box-numbers we hid."

In the figure, out of 7 boxes we hide boxes 3 and 5, so $M=\{3,5\}$. The hidden boxes turn coral; the visible ones stay lavender.

Why the topic needs it: BERT hides ~15% of tokens. $M$ is exactly which ones. Every prediction and every piece of loss happens only at positions inside $M$.

3. Probability $P(\cdot \mid \cdot )$ — the model's confidence

The model doesn't know the hidden word; it guesses with a confidence. That confidence is a probability.

Why the topic needs it: MLM is literally "predict the word given the clues." That single expression $P(x_i\mid {\mathbf{x}}_{\backslash M})$ is the whole task, written in symbols.

4. Vectors and ${\mathbf{h}}_i$ — turning a word-in-context into numbers

A computer can't multiply words. Every box must become a list of numbers.

The figure shows the pipeline: word → its own number-vector → plus context from neighbours → the final vector ${\mathbf{h}}_i$. Two identical words in different sentences get different ${\mathbf{h}}_i$ — that's the whole point of "contextual."

Why the topic needs it: Loss and softmax operate on numbers, not letters. ${\mathbf{h}}_i$ is the bridge from language to arithmetic.

5. The dot product ${\mathbf{w}}_v^{\top }{\mathbf{h}}_i$ — measuring "match"

How do we score how well a target ${\mathbf{w}}_v$ matches the context vector ${\mathbf{h}}_i$? We use the dot product.

Why the topic needs it: ${\mathbf{w}}_v^{\top }{\mathbf{h}}_i$ is the raw score (a "logit") for word $v$ at box $i$. Higher score → the model likes word $v$ more here.

6. exp and the softmax — turning scores into probabilities

Scores can be any number (even negative). Probabilities must be between 0 and 1 and add up to 1. We need a converter.

Putting it together, the softmax for word $x_i$ at box $i$: $P(x_i\mid {\mathbf{x}}_{\backslash M})=\frac{\exp ({\mathbf{w}}_{x_i}^{\top }{\mathbf{h}}_i)}{\sum _{v\in \mathcal{V}}\exp ({\mathbf{w}}_v^{\top }{\mathbf{h}}_i)}$

Read it as a fraction: top = "how much score this one word got (made positive)"; bottom = "the total score of all words." Dividing forces the numbers to add to 1 across the vocabulary — a valid probability.

The figure shows raw scores (some negative) becoming positive bars via $\exp$, then squeezed down so they sum to exactly 1.

Why the topic needs it: This is how the raw dot-product scores become the confidence $P(\cdot )$ that the loss reads.

7. log and the minus sign — building the loss

The last piece: turning "confidence in the right word" into a number to shrink.

Now the parent's loss reads plainly: ${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$ "For every hidden box $i$, take the model's confidence in the true word, log it, flip the sign, and add them up. Shrinking this number = making the model more confident in the right words."

Prerequisite map

Read top to bottom: boxes give us a numbered sequence; hiding some gives $M$; each box becomes a vector ${\mathbf{h}}_i$; dot products score candidate words; exp + softmax turn scores into probabilities; log + minus assemble the loss — and that is MLM.

Equipment checklist

Self-test: cover the right side and answer each before revealing.