Perplexity as a metric: Question bank — AI-ML deep dive

Notation this page uses (defined before we probe it)

Before the traps, we pin down every symbol so nothing is used before it is earned.

Everything below assumes these definitions.

True or false — justify

A perplexity of 1 means a perfect model.

True — perplexity 1 requires every real token to get probability

P_{i} = 1

, so

H = 0

and the model is never surprised; it "knew" each next token exactly.

Perplexity can be less than 1.

False — since

0 \leq P_{i} \leq 1

, each surprisal

- lo g_{b} P_{i} \geq 0

, so

H \geq 0

and

PPL = b^{H} \geq b^{0} = 1

; 1 is the hard floor.

Lower cross-entropy always means lower perplexity.

True —

PPL = b^{H}

is monotonically increasing in

H

(for base

b > 1

), so they move in the same direction; less average surprise means fewer effective choices.

Perplexity is measured in bits.

False — cross-entropy

H

has units (bits when

b = 2

, nats when

b = e

), but perplexity

b^{H}

is a dimensionless count: the effective number of equally-likely tokens.

A model with perplexity 50 on English is choosing among exactly 50 words at every step.

False — 50 is the effective (geometric-mean) branching factor across the whole text, not a literal count at any single position; some steps are far more or less certain.

If you double the length

N

of the test text, perplexity roughly doubles.

False — perplexity is a per-token geometric mean, so it is designed to be length-independent; doubling

N

leaves it roughly unchanged if the text is statistically similar.

Two models on the same data: the one with lower per-token negative log-likelihood has lower perplexity.

True — perplexity is a monotonic (exponential) function of the mean NLL, which is exactly

H

, so lower mean NLL always yields lower PPL; they rank models identically.

Switching from

lo g_{2}

ln

changes the numeric value of perplexity.

False — as long as you exponentiate with the matching base,

e^{H_{nats}} = 2^{H_{bits}}

; the base

b

cancels and perplexity is base-independent.

A larger vocabulary always gives a model higher perplexity.

False — a bigger vocabulary only raises the ceiling (

PPL \leq V

for a uniform model over

V

tokens); a good model can still be very confident and have low perplexity regardless of

V

Spot the error

"I computed mean

ln

-loss = 2.0, so perplexity =

2^{2.0} = 4

Error: base mismatch — the loss is in nats (

b = e

), so you must exponentiate with

e

, giving

e^{2.0} \approx 7.39

, not

2^{2.0}

"Model A has PPL 30 with word tokens; Model B has PPL 12 with sub-word tokens; B is clearly better."

Error: the tokenizations differ, so

N

and the per-token distribution

P_{i}

differ; per-token perplexities are not comparable across tokenizers — use bits-per-byte instead. See Bits-Per-Character (BPC).

"Entropy and perplexity are opposites, so a confident (low-entropy) model has high perplexity."

Error: they are not opposites —

PPL = b^{H}

increases with

H

, so low entropy gives low perplexity, same direction.

"To get sequence perplexity I add the per-token probabilities and take a root."

Error: probabilities are multiplied (chain rule

P = \prod_{i} P_{i}

), then you take the

N

-th root of the reciprocal; addition would be meaningless here.

"The model gave probability 0 to one rare word, but the other 999 words were great, so PPL is still fine."

Error: one

P_{i} = 0

makes

- lo g_{b} 0 = + \infty

, so the sum,

H

, and thus PPL all blow up to

\infty

— a single impossible-but-real token destroys the score.

"Perplexity 1.0 on my training set proves my model is excellent."

Error: perplexity near 1 on training data usually signals memorization/overfitting; the metric is only meaningful on held-out data the model has not seen.

"Since PPL = exp(loss) and loss ≥ 0, perplexity ranges from 0 to ∞."

Error: because

H \geq 0

, perplexity ranges from 1 (at

H = 0

) to

\infty

, never from 0;

e^{0} = 1

is the minimum.

Why questions

Why do we take the geometric mean of

1/ P_{i}

rather than the arithmetic mean?

Because sequence probability is a product

\prod_{i} P_{i}

; the geometric mean is the natural per-factor summary of a product, and its log is exactly

H

, the arithmetic mean of the log-probabilities. See Cross-Entropy Loss.

Why does the exponent carry a negative sign in

(\prod_{i} P_{i})^{- 1/ N}

The negative flips "probability" (bigger = better) into a cost (bigger = worse), so perplexity behaves like a loss you want to minimize.

Why does a uniform model over

V

tokens have perplexity exactly

V

Each

P_{i} = 1/ V

gives surprisal

lo g_{b} V

, so

H = lo g_{b} V

and

PPL = b^{l o g_{b} V} = V

; it literally recovers the vocabulary size — the worst-case branching factor.

Why is perplexity only a proxy for text quality, not quality itself?

It measures probability calibration on real text, rewarding the safe/frequent next token; a model can score low PPL yet produce dull, repetitive prose because "likely" is not the same as "good."

Why must probabilities come from a proper distribution (e.g. softmax) for perplexity to be valid?

Perplexity assumes each

P_{i} = P (w_{i} ∣ w_{< i})

is a normalized probability summing to 1 over the vocabulary; unnormalized scores would make

- lo g_{b} P_{i}

and the geometric mean meaningless.

Why does minimizing the language-modeling loss automatically minimize perplexity?

The training loss is the average negative log-likelihood, which equals

H

, and perplexity is just

b^{H}

; lowering one monotonically lowers the other.

Why can't perplexity ever be negative?

It is an exponential

b^{H}

with

b > 1

, and exponentials of real numbers are always positive; moreover

H \geq 0

forces

PPL \geq 1

Edge cases

What is the perplexity if the model assigns probability 1 to every actual next token?

Exactly 1 — every surprisal is

- lo g_{b} 1 = 0

, so

H = 0

and

PPL = b^{0} = 1

; the perfect, never-surprised model.

What is the perplexity if the model assigns probability 0 to any actual token?

\infty

— that token's surprisal

- lo g_{b} 0 = + \infty

dominates the sum, which is exactly why we smooth or clamp probabilities away from zero.

For a single-token sequence (

N = 1

), what does perplexity reduce to?

Just

1/ P_{1}

— the geometric mean of one number is that number, so perplexity is the reciprocal of the model's probability for that one token.

What perplexity does a completely random guesser over a 50,000-token vocabulary achieve?

50,000 — a uniform model gives

PPL = V

; this is the ceiling that any trained model must beat to show it learned anything.

If a model is worse than uniform (assigns below-average probability to real tokens), can its perplexity exceed

V

Yes — perplexity is only bounded below by 1, not above by

V

; a systematically miscalibrated model that favors wrong tokens can have

PPL > V

What happens to perplexity as the test text approaches infinite length (from the same source)?

It converges toward

b^{H_{\infty}}

, where

H_{\infty}

is the true per-token cross-entropy between the model and the language — the metric stabilizes rather than drifting with

N

. See Entropy and Information Content.

Connections

4.2.09 Perplexity as a metric — the parent concept this bank probes; reuse its definition of $PPL = b^{H}$ when you reason through each trap.
Cross-Entropy Loss — perplexity is $b^{H}$ with $H$ the cross-entropy, so this page's traps are all disguised statements about that loss.
Entropy and Information Content — supplies the surprisal $- lo g_{b} P_{i}$ that makes "why $\geq 0$ " and the $\infty$ edge case work.
Language Modeling Objective — minimizing its average NLL is minimizing $H$ , hence perplexity; explains the "same direction" traps.
Tokenization — sets $N$ and the per-token distribution, which is why cross-tokenizer comparison of PPL fails.
Bits-Per-Character (BPC) — the tokenizer-independent fix invoked whenever tokenizations differ.
Softmax and Probability Distributions — guarantees each $P_{i}$ is a normalized probability, the precondition for perplexity to be defined at all.