Visual walkthrough — Masked language modeling (BERT)

This page takes the single most important formula in Masked language modeling (BERT) — the training loss

${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

— and builds it symbol by symbol, picture by picture, starting from "we hid a word, now guess it." You do not need to already know what a probability, a logarithm, or a softmax is. We earn each one.

---

Step 1 — Hide a word: the setup

WHAT. We write our sentence as a list of tokens (word-pieces). Call the whole list $\mathbf{x}$:

$\mathbf{x}=(x_1,x_2,\dots ,x_n)$

• $\mathbf{x}$ — the bold letter means "the whole list at once."
• $x_1,x_2,\dots$ — each individual token; the little number below is its position (1st, 2nd, ...).
• $n$ — how many tokens there are in total.

Now we pick some positions to hide. The set of hidden positions is $M$ (M for Masked):

$M\subset \{1,2,\dots ,n\}$

• $M$ — a collection of position numbers, e.g. $M=\{3\}$ means "only position 3 is hidden."
• $\subset$ — this curly symbol reads "is a subset of": every number in $M$ is one of the real positions $1$ through $n$. You cannot mask position 99 in a 7-word sentence.

WHY. Before we can ask "how good is the guess?", we must be precise about which words are being guessed. $M$ is exactly that list.

PICTURE. The sentence sits in a row of boxes; one box has a thumb over it, and its index is dropped into the set $M$.

---

Step 2 — What the model sees vs. what it must produce

WHAT. The model does not get to see the hidden word. It only sees the corrupted sentence — everything except the masked positions. We write that as:

${\mathbf{x}}_{\backslash M}$

• $\mathbf{x}$ — the sentence again.
• $\backslash M$ — the backslash reads "without $M$". So ${\mathbf{x}}_{\backslash M}$ = "the sentence with the masked positions removed / replaced by [MASK]."

The thing the model must produce is the original token that used to sit at a hidden position, e.g. $x_3=\text{"cat"}$.

WHY. This split is the heart of MLM: input = corrupted, target = original. If the model could see the original it would just copy it — no learning. Hiding forces it to reason from the neighbours on both sides.

PICTURE. Left half: the boxes the model reads (blur over the masked one). Right half: the true word it is being quizzed on.

---

Step 3 — A guess is a probability, not a single word

WHAT. The model does not blurt out one word. It hands you a confidence — a number between $0$ and $1$ — for the true answer. We write it:

$P(x_i\mid {\mathbf{x}}_{\backslash M})$

Read this left to right:

• $P(\cdots )$ — "the probability of", a number from $0$ (impossible) to $1$ (certain).
• $x_i$ — the true token at position $i$ (the answer we care about).
• $\mid$ — the vertical bar reads "given". Everything after it is what the model was allowed to look at.
• ${\mathbf{x}}_{\backslash M}$ — the corrupted sentence it looked at.

So the whole thing reads: "how much probability does the model put on the correct word, given only the visible context."

WHY a probability and not a yes/no? Because "how bad you felt" needs a dial, not a switch. Being 0.9 sure and right is better than being 0.51 sure and right; a number captures that fine grading, a yes/no cannot.

PICTURE. A dial from 0 to 1; the needle sits on the probability the model assigned to the true word "cat".

---

Step 4 — Where that probability comes from: softmax

WHAT. How does the Transformer turn its internal thought into a number between 0 and 1 for every possible word? First it produces one raw score per vocabulary word:

$\text{score for word }v={\mathbf{w}}_v^{\top }{\mathbf{h}}_i$

• ${\mathbf{h}}_i$ — the model's contextual thought at position $i$: a list of numbers summarising everything it read on both sides. This is the only place the bidirectional reading enters.
• ${\mathbf{w}}_v$ — a fixed list of numbers that represents candidate word $v$.
• ${\mathbf{w}}_v^{\top }{\mathbf{h}}_i$ — the dot product: multiply matching entries and add them up. It is a match score — big when the thought ${\mathbf{h}}_i$ points the same way as the word ${\mathbf{w}}_v$.

Raw scores can be any size, even negative — not probabilities yet. Softmax fixes that:

$P(x_i\mid {\mathbf{x}}_{\backslash M})=\frac{\exp ({\mathbf{w}}_{x_i}^{\top }{\mathbf{h}}_i)}{\sum _{v\in \mathcal{V}}\exp ({\mathbf{w}}_v^{\top }{\mathbf{h}}_i)}$

Term by term:

• $\exp (\cdots )$ — the exponential; it turns any score into a positive number and blows up big scores much larger than small ones (so the winner stands out).
• top (numerator) — $\exp$ of the score of the correct word $x_i$.
• ${\sum }_{v\in \mathcal{V}}$ — add up $\exp (\text{score})$ over every word $v$ in the vocabulary $\mathcal{V}$.
• The fraction = "correct word's share of the total." Because we divide by the total, all the words' probabilities add to exactly $1$. That is what makes it a genuine probability.

WHY exponential and division? We need (a) every number positive and (b) all of them summing to 1. Exponential guarantees positivity; dividing by the sum guarantees they total 1. It is the simplest tool that does both while keeping the ranking of scores.

PICTURE. Bars of raw scores (some negative) → $\exp$ makes them all positive and stretches them → normalise so the heights sum to 1.

---

Step 5 — Grading one guess: the logarithm

WHAT. We now turn one confidence into one "badness" number. The tool is the logarithm:

$\text{badness at }i=-\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

• $\log$ — the (natural) logarithm. Feed it a number between 0 and 1 and it returns something $\le 0$.
• $\log 1=0$ — perfect confidence → zero cost.
• $\log (\text{tiny})\to -\infty$ — near-zero confidence in the truth → the log dives to a huge negative number.
• the leading $-$ (minus) flips that so badness is positive and grows without bound as confidence in the truth drops.

WHY log and why the minus? Two reasons.

1. Shape: we want a gentle penalty when you are almost right and a savage penalty when you were confidently wrong. $-\log$ does exactly that curve.
2. Multiplication → addition: in the next step we combine many guesses; probabilities multiply (and tiny products underflow to 0 on a computer). $\log$ turns products into sums, which are numerically safe. The minus just makes "smaller = better," which is what gradient descent wants to minimise.

PICTURE. The curve $y=-\log p$ over $p\in (0,1]$: flat and near zero at $p=1$, shooting up toward the left.

---

Step 6 — Add up over all masked words: the full loss

WHAT. A sentence usually has several masked positions. We grade each one and add the badnesses:

${\mathcal{L}}_{\text{MLM}}=-{\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

• ${\mathcal{L}}_{\text{MLM}}$ — the total loss for this sentence (script L for "Loss").
• ${\sum }_{i\in M}$ — "add the following over every masked position $i$ in the set $M$."
• $\log P(x_i\mid {\mathbf{x}}_{\backslash M})$ — the (still negative) log-confidence in the truth at position $i$.
• the single leading $-$ pulls the minus outside the sum, so the total is positive.

WHY a sum (why is adding allowed)? Because in Step 5 the log already converted the product of the per-word probabilities into a sum of logs:

$\log ({\prod }_{i\in M}P(x_i\mid {\mathbf{x}}_{\backslash M}))={\sum }_{i\in M}\log P(x_i\mid {\mathbf{x}}_{\backslash M})$

• $\prod$ — "multiply together." We multiply because we treat each masked guess as an independent quiz question, and the chance of getting all of them right is the product of getting each right.
• The identity "log of a product = sum of logs" is what lets the multiplication collapse into the clean sum above.

PICTURE. Three masked slots, each with its own dial and its own bar of badness; the bars stack into one total tower — the loss.

---

Step 7 — The edge cases (never skip these)

WHAT & WHY, case by case.

Case A — nothing masked, $M=\varnothing$. The empty set has no positions, so ${\sum }_{i\in \varnothing }=0$. Loss is $0$ — there is nothing to grade. Correct: a sentence with no blanks teaches nothing.

Case B — the model is perfect, $P=1$. Then $-\log 1=0$ at every slot, so ${\mathcal{L}}_{\text{MLM}}=0$. Zero loss = perfection. Good.

Case C — the model assigns $P\to 0$ to the truth. Then $-\log P\to +\infty$. One confident, wrong guess can dominate the whole sum. This is why over-confidence is dangerous during training and why the gradient pushes hard to fix it.

Case D — the "10% random / 10% unchanged" masked tokens. These are still positions in $M$, so they still contribute a $-\log P$ term. The model is graded on recovering the original token even when the visible token is a random word or the real word. This is what stops it from lazily copying the input.

PICTURE. Four mini-panels: empty $M$ (flat zero), perfect (flat zero), catastrophic (spike to infinity), and the corrupted-token variants still feeding the sum.

---

The one-picture summary

Everything on this page is one pipeline: hide → read both sides → score every word → softmax to a probability → $-\log$ the truth → sum over masks.

Recall Feynman retelling — say it out loud

I write a sentence and cover a few words with my thumb; the covered spots make a set $M$. I show the model only what's left over — the "given" part, ${\mathbf{x}}_{\backslash M}$. For each covered spot the model builds a thought ${\mathbf{h}}_i$ from the words on both sides, matches that thought against every word in its dictionary by a dot product, and passes those match-scores through softmax so they become a set of confidences that add to one. I look up the confidence it gave to the actual hidden word. If that confidence is near 1, $-\log$ of it is near 0 — no penalty. If it's near 0, $-\log$ blows up — big penalty. I add those penalties across all the covered spots to get one total loss ${\mathcal{L}}_{\text{MLM}}$. Training just wiggles the model's numbers to make that total as small as possible, which forces it to actually understand the words around each blank.

Recall Check yourself

What does the vertical bar in $P(x_i\mid {\mathbf{x}}_{\backslash M})$ mean? ::: "given" — it separates the answer we want from the visible context the model is allowed to use. Why do we take the log before summing? ::: Combined probabilities multiply and underflow; log turns the product into a numerically safe sum ($\log$ of a product = sum of logs). Why the leading minus sign? ::: $\log$ of a probability (≤1) is ≤0; the minus makes the loss positive and "smaller = better," which gradient descent minimises. If no tokens are masked, what is the loss? ::: $0$ — the sum over the empty set $M$ has no terms. What tool guarantees the outputs are positive and sum to 1? ::: softmax — $\exp$ makes them positive, dividing by the total makes them sum to 1.

Prefer to read this in Hinglish? → 4.2.07 Masked language modeling (BERT) (Hinglish)