BERT and encoder models: Visual walkthrough — AI-ML deep dive

We are answering ONE question the whole way down:

"I hid a word. The model has some numbers. How do I turn those numbers into a single score that is small when the model is right and big when it is wrong?"

Step 1 — Hiding a word, and what the model gives back

WHAT. Take the sentence "The cat sat on the ___". We cover the last word with a sticker (this is the [MASK] token). The model reads everything else and, for the hidden slot, hands us one number for every word it knows. That list of known words is the vocabulary, written $V$ . Here we shrink $V$ down to three words so we can draw it: mat, car, sky.

WHY. Before we can build any loss, we must see what the model actually outputs. It does not output a word. It outputs a raw score per word — call these scores logits. A logit is just "how much the model likes this word", and it can be any real number: positive, negative, zero.

PICTURE. Three bars, one per word. The height is the logit. Taller bar = model likes that word more. The true hidden word is mat, marked in green.

Step 2 — Why raw scores are not yet probabilities

WHAT. We want probabilities: numbers that are never negative and that add to $1$ across all words. Our logits are $(2.0, 1.0, 0.0)$ — one is zero, and they sum to $3$ , not $1$ . So they are not probabilities yet.

WHY. Two problems to fix, in order:

Logits can be negative. A probability can never be negative.
Logits don't sum to 1. Probabilities must.

PICTURE. Left panel: the raw logits (one dips to zero — imagine another dipping below the axis). Right panel: what we want — bars that are all above zero and whose total shaded area is exactly one whole unit.

This gap — from "any real numbers" to "positive numbers summing to one" — is exactly the job of the next two steps.

Step 3 — The exponential: making everything positive

WHAT. Apply $e^{(\cdot)}$ to every logit. The symbol $e \approx 2.718$ is a fixed number; $e^{x}$ means " $e$ multiplied by itself $x$ times" (and it still makes sense for fractions and negatives).

e^{2.0} = 7.389 e^{z_{mat}} e^{1.0} = 2.718 e^{z_{car}} e^{0.0} = 1.000 e^{z_{sky}}

WHY this tool and not another? We need a function that turns any real number into a positive one, and that never destroys order (a bigger logit must stay bigger). We could imagine other positive functions, but $e^{x}$ is special:

$e^{x} > 0$ for every $x$ — solves problem 1 from Step 2. Even $e^{- 5} = 0.0067$ is positive.
It is monotonic: if $z_{a} > z_{b}$ then $e^{z_{a}} > e^{z_{b}}$ , so the model's ranking of words is preserved.
It turns addition of logits into multiplication, which is what makes the log in Step 6 collapse so cleanly (we cash this in later).

PICTURE. The curve $y = e^{x}$ . Notice it hugs zero on the left (never touching it) and shoots up on the right. We drop our three logits onto the $x$ -axis and read the positive heights off the curve.

Step 4 — Normalizing: making them sum to one

WHAT. Add up the exponentials, then divide each one by that total.

S = the sum S = 7.389 + 2.718 + 1.000 = 11.107 e^{z_{mat}} + e^{z_{car}} + e^{z_{sky}}

p (mat) = \frac{e ^{z_{mat}}}{S} = \frac{7.389}{11.107} = 0.665

WHY. Dividing every part by their common total forces the parts to add to $1$ — that is the definition of a fraction of a whole. This fixes problem 2 from Step 2. The general recipe, now fully earned, is the softmax:

p (w ∣ context) = softmax (z_{i})_{w} \frac{e ^{z_{i, w}}}{v \in V \sum e ^{z_{i, v}}}

Term by term: the top $e^{z_{i, w}}$ is this word's positive weight; the bottom $\sum_{v \in V} e^{z_{i, v}}$ (the symbol $\sum$ means "add over every word $v$ in the vocabulary $V$ ") is the total of all weights; the fraction is this word's share of the whole. See Self-Attention and Transformer Architecture for where these logits physically come from ( $z_{i} = W h_{i} + b$ ).

PICTURE. The three positive weights, each cut into its slice of one full pie. mat takes the biggest slice, $0.665$ .

Recall Why can't a softmax output be exactly 0 or exactly 1?

Because every numerator $e^{z} > 0$ and the denominator is a sum of positive terms — so every share is strictly between $0$ and $1$ ::: it can get arbitrarily close but never touches the ends.

Step 5 — The two edge cases every reader must see

WHAT. Push softmax to its extremes so nothing surprises you later.

Case A — a total tie. All logits equal, $z = (1, 1, 1)$ . Then $e^{1} = e^{1} = e^{1}$ , so every share is $1/3$ . Maximum uncertainty: the model has no preference.

Case B — a runaway winner. One logit huge, $z = (10, 1, 0)$ . Then $e^{10} = 22026$ dwarfs $e^{1} = 2.72$ and $e^{0} = 1$ , so $p (mat) = 22026/22030 \approx 0.9998$ . Softmax approaches a confident spike but still never hits exactly $1$ (Step 4's rule).

WHY show these? So you know the full range of behaviour: flat when clueless, spiked when sure — and that the two ends ( $0$ and $1$ ) are limits it approaches but never reaches. This matters for the loss in Step 6, which blows up near those ends.

PICTURE. Left: flat thirds (the tie). Right: one spike near $1$ with two nearly-invisible slivers (the runaway).

Step 6 — From probability to a loss: the negative log

WHAT. We now have $p (mat) = 0.665$ : the probability the model gave the correct word. We want a loss — a number that is small when we're right, large when we're wrong. But $0.665$ is big when right — the wrong direction. Fix it in two moves:

Take the log: $lo g (0.665) = - 0.408$ .
Flip the sign: $- lo g (0.665) = 0.408$ .

L = - lo g prob of the TRUE word y_{i} p (y_{i} ∣ context)

Here $y_{i}$ is the symbol for the actual hidden word at position $i$ (its label). We only ever plug the true word's probability into the loss.

WHY the log, and why the minus?

A probability lives in $(0, 1]$ . On that range, $lo g$ is negative or zero: $lo g (1) = 0$ (perfect), and $lo g$ dives to $- \infty$ as $p \to 0$ (a confident wrong answer). So $lo g$ already grows huge in exactly the way we want a penalty to grow — it just has the wrong sign.
The minus flips negatives to positives, giving a clean penalty: $0$ when perfect, $\to + \infty$ when the model was sure and wrong.
Why $lo g$ and not, say, $1 - p$ ? Because $lo g$ turns the product of many tokens' probabilities into a sum of losses (that's the payoff we set up in Step 3), and it punishes overconfident mistakes brutally, which is what good training needs. This is exactly Cross-Entropy Loss.

PICTURE. The curve $y = - lo g (p)$ over $p \in (0, 1]$ : it touches $0$ at $p = 1$ and rockets toward the sky as $p \to 0$ . Our point $p = 0.665$ sits at height $0.408$ .

Step 7 — Many masked words: averaging over the set $M$

WHAT. A real sentence hides several tokens. Let $M$ be the set of masked positions (in "The cat sat on the [MASK]" with one sticker, $∣ M ∣ = 1$ ; the symbol $∣ M ∣$ means "how many positions are in $M$ "). Compute the Step-6 loss at each masked position and take the average.

L_{MLM} = average over the ∣ M ∣ masks - \frac{1}{∣ M ∣} add over every masked slot i i \in M \sum Step 6 term for one masked slot lo g p (y_{i} ∣ context)

WHY average, not just sum? If we only summed, a sentence with 30 masks would produce a bigger loss than one with 3 masks purely because it has more masks, not because the model is worse. Dividing by $∣ M ∣$ makes the loss a fair "per-mask" score, comparable across sentences.

WHY only $i \in M$ ? The un-hidden tokens are visible — the model can literally read them. Grading it for "predicting" a word it can see teaches nothing and leaks the answer (see Fine-Tuning vs Pretraining for why clean objectives matter). So the sum runs only over masked positions.

PICTURE. A 10-token sentence. Two slots wear stickers (orange). Only those two feed arrows into the average box; the eight visible tokens (gray) feed nothing.

Step 8 — Why the 80/10/10 corruption keeps this loss honest

WHAT. Of the chosen 15% tokens: 80% become [MASK], 10% become a random word, 10% stay unchanged. But the loss above is computed at all of these positions against the original word $y_{i}$ .

WHY this doesn't break the loss. The loss formula never changed — it still asks "what probability did you give the true word $y_{i}$ ?" The corruption only changes the input the model reads, not the target it's graded on. Because sometimes the visible token is the real answer (10% unchanged) and sometimes a lie (10% random), the model can never trust the surface token and must build a genuine representation for every position — solving the train/test mismatch, since [MASK] never appears at inference. See RoBERTa for how this recipe was later tuned.

PICTURE. Three example slots: one [MASK], one swapped to a random word ("banana"), one left as the true word — all three graded against the original word underneath.

The one-picture summary

Everything above, as a single pipeline: raw logits → exponentiate (positive) → normalize (sum to one) → pick the true word's probability → negative-log it → average over masks.

Recall Feynman retelling — explain the whole walkthrough to a friend

The model hides a word and gives a raw liking score for every word it knows (Step 1). Those scores can be negative and don't add up nicely, so they aren't probabilities yet (Step 2). We raise $e$ to each score to force them all positive without scrambling the order (Step 3), then divide each by their total so they add to one — that's softmax (Step 4). We sanity-check the extremes: all-tied gives equal shares, one-huge gives a near-spike but never a perfect $1$ (Step 5). We grab the probability of the correct word and take its negative log — zero when perfect, exploding when confidently wrong (Step 6). With many hidden words we average these penalties, and we grade only the hidden slots, never the visible ones (Step 7). Finally, the 80/10/10 corruption changes what the model reads but not what it's graded against, so it can't cheat and must truly understand every position (Step 8). Stack it all up and you have the MLM loss.

Back to the parent: BERT and encoder models · related: T5 and encoder-decoder models, WordPiece Tokenization.