Foundations — Perplexity as a metric

This page assumes nothing. If the parent note Perplexity as a metric used a symbol, we build it here from the ground up, in an order where each idea leans only on the ones before it.

0. What is a "token"? (the atom of everything)

Before we can talk about probabilities of text, we need to know what text is made of.

The picture (Figure 1): imagine a sentence as a train. Each carriage is one token. The model reads the train one carriage at a time, left to right.

Why the topic needs it: perplexity is measured per token. If you don't know what a token is, you can't know what "per token" divides by. Two people who chop the same sentence into different-sized carriages will get different perplexities — this is exactly why you can only compare on the same tokenization.

1. The symbols $w_i$, $N$, and the subscript

Now we name the carriages.

The picture (back to Figure 1): the same train, but now each carriage wears a label $w_1,w_2,\dots ,w_N$.

2. Probability $P(\cdot )$ — the model's "how likely" number

The picture (Figure 2): a horizontal bar. Full bar = probability $1$. Half-full = $0.5$. Empty = $0$.

Why the topic needs it: perplexity's whole job is to grade how much probability the model handed to the tokens that actually appeared. High probability on the real token = good. That grading needs this number.

3. The vertical bar: $P(w_i\mid w_{<i})$ — "given what came before"

Language is not a bag of independent words; the next word depends on the earlier ones. We need notation for "the probability of this, given that."

The picture: the reader stands on carriage $i$ and looks backward down the track at everything already read; that backward view is $w_{<i}$.

Why the topic needs it: a good model uses context. After "the cat sat on the ___", the word "mat" is likely and "helicopter" is not. Only conditional probability captures this, and language models are trained exactly to predict $P(w_i\mid w_{<i})$.

4. Joint probability $P(A,B)$ and the chain rule

To score a whole sentence we must combine per-token probabilities. First we need the notation for "the probability that several things all happen together."

The key link between "and" and "given." How likely are $A$ and $B$ together? First $B$ happens (probability $P(B)$), then $A$ happens given $B$ (probability $P(A\mid B)$). Multiply those two:

Now unroll it for a sequence. Apply the product rule again and again, peeling off one token at a time from the back:

Each step splits off the last token as "given everything before it," leaving a smaller joint probability to split again. Do this for all $N$ tokens and the pattern is:

Why the topic needs it: this product is the raw score $P(w_1,\dots ,w_N)$ that appears inside perplexity. Notice the problem it creates: multiplying many numbers below $1$ makes the result tiny and shrinks with length — a 100-token sentence scores far lower than a 5-token one, even from the same model. Fixing that shrinkage is the entire reason perplexity does its next two tricks (the root and the log).

5. Logarithms $\log$ — turning multiplying into adding

Multiplying many small numbers is numerically nasty and hard to average. The logarithm is the tool that rescues us.

The picture (Figure 3): think of a factory line. Multiplying is stacking blocks (heights multiply, awkward). $\log$ is a magic ruler that turns "stacking" into "laying blocks end-to-end" (heights add).

6. Surprisal $-\log P$ — measuring "how shocked am I?"

Now we assemble logs into the emotional core of perplexity: surprise.

The picture (Figure 4): a curve that is $0$ on the right (at $P=1$, no surprise) and shoots up toward infinity on the left (as $P\to 0$, unbounded shock).

Why the minus sign? Because $P$ is between $0$ and $1$, its log is negative. Multiplying by $-1$ flips it to a positive "amount of surprise." A cost should be positive.

Why the topic needs it: perplexity is literally the average surprise, exponentiated. Read more at Entropy and Information Content and Cross-Entropy Loss.

7. Averaging with $\frac{1}{N}\sum$ and the "root" $x^{-1/N}$

Two final tools glue everything into a length-independent score.

The picture: the geometric mean of $\{0.5,0.25,0.1\}$ is the single value that, multiplied by itself 3 times, gives the same product $0.0125$ — one "typical" per-token probability standing in for all of them.

8. The capstone: assembling the perplexity formula

Now every symbol is earned, so we write the whole thing in one place.

Step A — start from the sentence score (Section 4): $P(w_1,\dots ,w_N)={\prod }_{i=1}^{N}P(w_i\mid w_{<i})$

Step B — turn "bigger = good" into a length-independent cost. Take the $N$-th root and the reciprocal (Section 7):

Step C — the equivalent surprise form. Using $\log$ (Section 5) to turn the product into a sum of surprisals (Section 6) and then averaging (Section 7):

These two boxed forms are the same quantity written two ways; the parent note Perplexity as a metric proves their equality line by line. And from Section 6's edge case: if any $P(w_i\mid w_{<i})=0$, the sum contains $+\infty$ and $\text{PPL}=\infty$.

Prerequisite map

Equipment checklist

Test yourself — cover the right side and answer aloud.