Perplexity as a metric: Worked examples — AI-ML deep dive

Before anything, a one-line reminder of the two tools we lean on the whole way down. Both come straight from the parent, but let's re-anchor them in plain words so nothing is assumed.

The scenario matrix

Every case class perplexity can throw at you, and the example that nails it.

Cell	Scenario class	What's special	Example
A	Uniform over $V$ tokens	Degenerate "no knowledge" model; $PPL = V$ exactly	Ex 1
B	Perfect model	Every $P_{i} = 1$ ; lower limit	Ex 2
C	Mixed per-token probs	The everyday geometric-mean case	Ex 3
D	Zero-probability real token	Limiting value $\to \infty$	Ex 4
E	Compare two models, same data	The practical decision use	Ex 5
F	Base trap ( $ln$ vs $lo g_{2}$ )	Same answer only if bases match	Ex 6
G	Different tokenizers, same text	$N$ changes → PPL not comparable; use Bits-Per-Character (BPC)	Ex 7
H	Real-world word problem	Weather forecaster analogy, from raw counts	Ex 8
I	Exam twist: PPL → recover a hidden prob	Invert the formula	Ex 9

We now walk each cell. Figures accompany the ones where a picture carries the reasoning.

Cell A — the uniform (know-nothing) model

Worked example Ex 1 — vocabulary

$V = 8$ , model is uniform The model always spreads probability equally over $V = 8$ possible tokens, no matter the context. Find the perplexity of any sequence.

Forecast: guess the number before computing. What would "torn between 8 equally-good words" give?

Each per-token probability. $P_{i} = \frac{1}{8}$ for every $i$ . Why this step? "Uniform over 8" is the definition of splitting $1$ into $8$ equal shares — see Softmax and Probability Distributions for how a flat logit vector produces exactly this.
Average negative log. $- lo g_{2} P_{i} = - lo g_{2} \frac{1}{8} = 3$ bits, and this is the same for every token, so the average $H = 3$ bits. Why this step? Perplexity is $b^{H}$ ; we need $H$ first. Base 2 is convenient because $lo g_{2} 8 = 3$ is clean.
Exponentiate. $PPL = 2^{3} = 8$ . Why this step? $b^{H}$ with the same base $b = 2$ we used for the log — the base cancels (we prove that cancellation in Ex 6).

Verify: $PPL = 8 = V$ . Look at the figure: 8 equal bars, each height $1/8$ ; the model is genuinely choosing among all 8. This is the worst useful case — a model that learned nothing still scores $V$ , so any real model must beat $V$ to be worth anything.

Cell B — the perfect model

Worked example Ex 2 — every real token predicted with certainty

A (fantasy) model assigns $P_{i} = 1$ to the token that actually appears, every single step, over a sequence of $N = 100$ tokens.

Forecast: if the model is never surprised, what is the smallest perplexity can be?

Product of probabilities. $\prod_{i = 1}^{100} 1 = 1$ . Why this step? Chain rule for the whole-sequence probability; multiplying $1$ 's stays $1$ .
Apply the root. $PPL = 1^{- 1/100} = 1$ . Why this step? Any root of $1$ is $1$ ; the $- 1/ N$ power can't push it below $1$ .
Interpret the floor. Perplexity can never be less than $1$ , because each $P_{i} \leq 1 \Rightarrow - lo g P_{i} \geq 0 \Rightarrow H \geq 0 \Rightarrow b^{H} \geq 1$ . Why this step? Establishes the lower bound, matching Cell A's upper bound $V$ . Real models live in $[1, V]$ .

Verify: $1$ means "torn between exactly 1 option" = perfectly decided = no puzzle. Consistent with the Language Modeling Objective: minimizing negative log-likelihood drives PPL toward this floor of $1$ .

Cell C — the everyday mixed case

Worked example Ex 3 — three tokens with different confidences

The model gives $P_{1} = 0.5, P_{2} = 0.25, P_{3} = 0.1$ for the three real tokens. Compute PPL.

Forecast: the probs range from confident ( $0.5$ ) to unsure ( $0.1$ ). Will PPL sit nearer $2$ or nearer $10$ ?

Sequence probability (chain rule). $P = 0.5 \times 0.25 \times 0.1 = 0.0125$ . Why this step? $P (w_{1}, w_{2}, w_{3}) = P_{1} P_{2} P_{3}$ ; this is the joint probability the whole line is right.
Geometric mean of the probs. $\overset{ˉ}{P} = (0.0125)^{1/3} \approx 0.2321$ . Why this step? Perplexity is $1/ \overset{ˉ}{P}$ ; the $1/ N$ root is the geometric mean, which is the "typical" per-token probability — it weights the small $0.1$ heavily, which is why PPL is closer to $1/0.1$ than to $1/0.5$ .
Reciprocal. $PPL = 1/ \overset{ˉ}{P} = (0.0125)^{- 1/3} \approx 4.31$ . Why this step? Negative exponent flips "typical probability" into "typical branching factor."

Verify: the arithmetic mean of the probs is $(0.5 + 0.25 + 0.1) /3 = 0.283$ , giving $1/0.283 = 3.53$ ; the geometric mean $0.232$ gives $4.31$ . Geometric mean is always $\leq$ arithmetic, so PPL is pulled up by the unsure token — matches the intuition that one bad guess hurts. $4.31$ sits between the confident $2$ and unsure $10$ , as forecast.

Cell D — the catastrophic zero

Worked example Ex 4 — the model says a real token is impossible

Same as Ex 3 but the fourth token is one the model gave $P_{4} = 0$ .

Forecast: what happens to the whole line's score when one token has probability $0$ ?

Surprisal of that token. $- lo g P_{4} = - lo g 0 = + \infty$ . Why this step? Surprise blows up as probability $\to 0$ ; "impossible-but-it-happened" is infinite surprise. Trace the curve $- lo g p$ in the figure as $p \to 0^{+}$ .
Average. $H = \frac{1}{4} (finite + finite + finite + \infty) = + \infty$ . Why this step? One infinite term dominates any average of finitely many terms.
Exponentiate. $PPL = b^{+ \infty} = + \infty$ . Why this step? $b > 1$ raised to $+ \infty$ diverges.

Verify: the product route agrees: $P = 0.5 \cdot 0.25 \cdot 0.1 \cdot 0 = 0$ , and $0^{- 1/4} = 1/0 = \infty$ . One zero destroys everything — this is exactly why we smooth probabilities and why softmax (which outputs strictly positive numbers) is used: it can never emit a hard $0$ .

Cell E — comparing two models on the same data

Worked example Ex 5 — Model A vs Model B, identical text

On the same held-out sentence of $N = 5$ tokens, Model A gives the true tokens probabilities $[0.4, 0.5, 0.3, 0.4, 0.6]$ and Model B gives $[0.2, 0.3, 0.1, 0.25, 0.2]$ . Which is better?

Forecast: B's numbers are all smaller. Before computing — must B's perplexity be larger?

Model A geometric mean. $\overset{ˉ}{P}_{A} = (0.4 \cdot 0.5 \cdot 0.3 \cdot 0.4 \cdot 0.6)^{1/5} = (0.0144)^{1/5} \approx 0.4275$ . Why this step? Same recipe as Ex 3; the geometric mean is the typical per-token prob.
Model A perplexity. $PPL_{A} = 1/0.4275 \approx 2.339$ . Why this step? Reciprocal of typical prob = effective branching factor.
Model B. $\overset{ˉ}{P}_{B} = (0.2 \cdot 0.3 \cdot 0.1 \cdot 0.25 \cdot 0.2)^{1/5} = (0.0003)^{1/5} \approx 0.1974$ , so $PPL_{B} \approx 5.066$ . Why this step? Repeat for B on the same tokens and same $N$ — the only fair comparison.

Verify: $PPL_{A} \approx 2.34 < PPL_{B} \approx 5.07$ , so A wins. A assigns higher probability to what actually happened, so it is less surprised — the whole point of Cross-Entropy Loss. Sanity: smaller per-token probs ⇒ larger PPL, as forecast.

Cell F — the base trap

Worked example Ex 6 — does using

$ln$ instead of $lo g_{2}$ change the answer? Take Ex 1's uniform model ( $P_{i} = 1/8$ ). Compute PPL two ways: with base-2 logs, and with natural logs. Show they match.

Forecast: will $ln$ give a different number than $lo g_{2}$ ?

Base-2 route. $H_{2} = - lo g_{2} \frac{1}{8} = 3$ bits, $PPL = 2^{H_{2}} = 2^{3} = 8$ . Why this step? Reproduces Ex 1 as the reference answer.
Natural-log route. $H_{e} = - ln \frac{1}{8} = ln 8 \approx 2.0794$ nats, $PPL = e^{H_{e}} = e^{2.0794} \approx 8$ . Why this step? Libraries compute $H$ in nats; you MUST exponentiate with $e$ , not $2$ .
Why they agree (the cancellation). $b^{H_{b}} = b^{- \frac{1}{N} \sum l o g_{b} P_{i}} = b^{l o g_{b} (\prod P_{i})^{- 1/ N}} = (\prod P_{i})^{- 1/ N}$ . The base $b$ appears once as the log and once as the exponent, and $b^{l o g_{b} x} = x$ — the base evaporates, leaving a base-free quantity. Why this step? Proves the answer is a property of the probabilities, not of your logarithm choice.

Verify: $8 = 8$ . The mistake to avoid (from the parent): computing $H$ in nats then raising $2^{H_{e}} = 2^{2.0794} \approx 4.23 \neq = 8$ — that mixes bases and is wrong.

Cell G — different tokenizers, same text

Worked example Ex 7 — why you can't compare PPL across tokenizers

The text "banana" is encoded two ways. Word tokenizer: 1 token, model gives $P = 0.05$ . Character tokenizer: 6 tokens b,a,n,a,n,a, model gives each prob $0.5$ (letters are easy). Compare the two perplexities.

Forecast: same text, same underlying probability $0.05$ of producing "banana" ( $0. 5^{6} = 0.015625$ , close-ish). Will the PPLs be comparable?

Word-level PPL. $N = 1$ , $PPL_{word} = 0.0 5^{- 1/1} = 1/0.05 = 20$ . Why this step? One token means $N = 1$ ; the root does nothing, PPL is just $1/ P$ .
Char-level PPL. $N = 6$ , product $= 0. 5^{6} = 0.015625$ , $PPL_{char} = 0.01562 5^{- 1/6} = 0. 5^{- 1} = 2$ . Why this step? Six easy tokens ⇒ per-token branching factor only $2$ , even though the word was rare.

Verify: $20$ vs $2$ — wildly different for the same string! The mismatch comes entirely from $N$ (1 vs 6) and the per-token distribution changing. See Tokenization. The fix: normalize by characters/bytes instead — use Bits-Per-Character (BPC), which divides by the number of characters (a fixed 6 here) so the comparison is fair.

Cell H — real-world word problem

Worked example Ex 8 — the weather forecaster

Over 4 days the true weather was Sun, Sun, Rain, Sun. A forecaster reports these probabilities for what actually happened: day1 Sun $0.8$ , day2 Sun $0.9$ , day3 Rain $0.3$ , day4 Sun $0.7$ . Treat each day as a "token." What's the forecaster's perplexity?

Forecast: the forecaster was confident except on the rainy day. Nearer $1$ (perfect) or nearer $2$ (coin-flip)?

Product of the day-probabilities. $0.8 \times 0.9 \times 0.3 \times 0.7 = 0.1512$ . Why this step? Chain rule; probability the whole 4-day sequence is called correctly.
Geometric mean. $\overset{ˉ}{P} = 0.151 2^{1/4} \approx 0.6237$ . Why this step? Typical single-day probability the forecaster assigned to the truth.
Perplexity. $PPL = 1/0.6237 \approx 1.603$ . Why this step? Effective number of outcomes the forecaster was torn between per day.

Verify: $\approx 1.60$ sits between $1$ (perfect) and $2$ (pure coin flip) — the forecaster is decent but the surprise rain (only $0.3$ ) pushes it above $1$ . Units check: probabilities are dimensionless, so PPL is a pure count of "effective outcomes," as it should be.

Cell I — the exam twist (invert it)

Worked example Ex 9 — recover a hidden probability from PPL

A model processes $N = 3$ tokens. You know $P_{1} = 0.5$ , $P_{2} = 0.4$ , and the reported $PPL = 4$ . Find the missing $P_{3}$ .

Forecast: PPL $= 4$ is fairly high given $P_{1}, P_{2}$ are decent. Do you expect $P_{3}$ to be large or small?

From PPL back to the product. $PPL = P^{- 1/3} = 4 \Rightarrow P = 4^{- 3} = \frac{1}{64} = 0.015625$ . Why this step? Undo the $- 1/3$ power: raise both sides to $- 3$ . This inverts the definition.
Divide out the known probs. $P_{3} = \frac{P}{P _{1} P _{2}} = \frac{0.015625}{0.5 \times 0.4} = \frac{0.015625}{0.2} = 0.078125$ . Why this step? The product is $P_{1} P_{2} P_{3}$ , so isolate $P_{3}$ by division.
Check the range. $0.078 \in (0, 1]$ ✅ — a valid probability. Why this step? If it exceeded $1$ we'd know the given PPL was impossible for those $P_{1}, P_{2}$ .

Verify: rebuild PPL: $(0.5 \cdot 0.4 \cdot 0.078125)^{- 1/3} = (0.015625)^{- 1/3} = 6 4^{1/3} = 4$ ✅. As forecast, $P_{3} \approx 0.078$ is small — one unsure token dragging the PPL up to $4$ .

The 80/20 of "every scenario"

Recall What each cell taught

Uniform model → PPL $= V$ (ceiling) ::: Ex 1 Perfect model → PPL $= 1$ (floor) ::: Ex 2 Real models live in $[1, V]$ ::: Ex 1 + Ex 2 One zero-prob token → PPL $= \infty$ ::: Ex 4 Geometric mean (not arithmetic) controls PPL ::: Ex 3 Compare only on same data ::: Ex 5 Base cancels iff you exponentiate with the same base ::: Ex 6 Different tokenizers are NOT comparable ::: Ex 7 You can invert PPL to recover a probability ::: Ex 9

Connections

Perplexity as a metric (Hinglish) — the parent, in Hinglish.
Cross-Entropy Loss — every example here is $e$ raised to a cross-entropy.
Entropy and Information Content — the $- lo g p$ surprisal used in Ex 1, 4, 6.
Language Modeling Objective — driving PPL to its floor of $1$ .
Tokenization — the $N$ -changes-everything trap of Ex 7.
Bits-Per-Character (BPC) — the tokenizer-fair fix for Ex 7.
Softmax and Probability Distributions — why real models never emit the $0$ of Ex 4.