Perplexity as a metric: Exercises — AI-ML deep dive

Before we start, one shared toolkit. Every problem below uses exactly two moves, so let us name them once.

Level 1 — Recognition

L1.1 A model predicts each of exactly $8$ tokens with equal probability $1/8$ at every step. Without computing $H$ , state its perplexity and say why.

L1.2 True or false: "If model A has cross-entropy $H_{A} = 3$ bits and model B has $H_{B} = 5$ bits on the same text, then A has the higher perplexity."

L1.3 Match each quantity to its formula: (a) surprisal of one event, (b) average per-token cross-entropy, (c) perplexity — from the list ${b^{H}, - lo g_{b} p, - \frac{1}{N} \sum_{i} lo g_{b} P_{i}}$ .

Recall Solution L1

L1.1 For a uniform model over $V$ options, every $P_{i} = 1/ V$ , so $H = - lo g_{2} \frac{1}{8} = lo g_{2} 8 = 3 bits, PPL = 2^{3} = 8.$ Perplexity of a uniform model is exactly the vocabulary size $V = 8$ — the branching factor. The model is genuinely torn between 8 equally-good tokens.

L1.2 False. $PPL = b^{H}$ is monotonically increasing in $H$ : bigger entropy → bigger perplexity. Since $H_{A} = 3 < 5 = H_{B}$ , model A has the lower perplexity ( $2^{3} = 8$ vs $2^{5} = 32$ ). A is the better model.

L1.3 (a) surprisal $= - lo g_{b} p$ (b) average cross-entropy $= - \frac{1}{N} \sum_{i} lo g_{b} P_{i}$ (c) perplexity $= b^{H}$

Level 2 — Application

L2.1 A model assigns these probabilities to the four real tokens of a sentence: $P_{1} = 0.5, P_{2} = 0.5, P_{3} = 0.25, P_{4} = 0.125$ . Compute $H$ in bits and then $PPL$ .

L2.2 In code the same sentence is scored with natural log. Compute the mean negative log-likelihood $NLL = - \frac{1}{N} \sum_{i} ln P_{i}$ and confirm $exp (NLL)$ gives the same perplexity as L2.1.

L2.3 A model gives every real token the same probability $0.2$ . What is its perplexity, for any sentence length?

Recall Solution L2

L2.1 Surprisals in bits ( $- lo g_{2} P_{i}$ ): $- lo g_{2} 0.5 = 1$ , $1$ , $- lo g_{2} 0.25 = 2$ , $- lo g_{2} 0.125 = 3$ . $H = \frac{1 + 1 + 2 + 3}{4} = \frac{7}{4} = 1.75 bits, PPL = 2^{1.75} \approx 3.364.$

L2.2 $ln 0.5 = - 0.693147$ (twice), $ln 0.25 = - 1.386294$ , $ln 0.125 = - 2.079442$ . $NLL = \frac{0.693147 + 0.693147 + 1.386294 + 2.079442}{4} = \frac{4.852030}{4} = 1.213008 nats .$ $exp (1.213008) \approx 3.364.$ Identical to L2.1 — the base cancels once you exponentiate with the matching base. This is the meaning of $e^{H_{nats}} = 2^{H_{bits}}$ .

L2.3 Uniform-per-token: every $P_{i} = 0.2 = 1/5$ , so $PPL = 1/0.2 = 5$ regardless of $N$ . Constant per-token probability ⇒ the geometric mean is that same number ⇒ perplexity is its reciprocal.

Level 3 — Analysis

L3.1 Two three-token predictions from the parent note. Model A: ${0.5, 0.25, 0.1}$ . Model C: ${0.3, 0.3, 0.3}$ . Which has lower perplexity? Compute both and explain which token dominates each score.

L3.2 A model is almost perfect: on $99$ tokens it assigns $P = 0.9$ , but on one real token it assigns $P = 1 0^{- 6}$ . Compute the perplexity and comment on how much that single token hurts.

L3.3 Take the limit: a model assigns probability 0 to a token that actually occurred. What is the perplexity, and what does the geometry of $- lo g p$ (see figure) tell you about why?

Recall Solution L3

L3.1 Model A: product $= 0.5 \cdot 0.25 \cdot 0.1 = 0.0125$ , so $PPL_{A} = 0.012 5^{- 1/3}$ . In nats: $ln 0.0125 = - 4.382027$ ; divide by 3 → $- 1.460676$ ; negate → $1.460676$ ; $e^{1.460676} \approx 4.310$ . Model C: product $= 0. 3^{3} = 0.027$ , so $PPL_{C} = 0.02 7^{- 1/3} = 1/0.3 = 3.333$ . Model C is lower (3.333 < 4.310). Even though A is very confident on token 1 ( $0.5$ ), its worst token ( $0.1$ ) drags the geometric mean down. Perplexity is a geometric mean, so the smallest probabilities do the most damage — one bad guess costs more than one good guess saves.

L3.2 $NLL = \frac{99 \cdot ( - ln 0.9 ) + 1 \cdot ( - ln 1 0 ^{- 6} )}{100}$ . Now $- ln 0.9 = 0.105361$ and $- ln 1 0^{- 6} = 6 ln 10 = 13.815511$ . $NLL = \frac{99 ( 0.105361 ) + 13.815511}{100} = \frac{10.430726 + 13.815511}{100} = 0.242462.$ $PPL = e^{0.242462} \approx 1.2744$ . Compare: without the bad token, $PPL = 1/0.9 \approx 1.111$ . One catastrophic token in a hundred pushes perplexity from $1.11$ to $1.27$ — a large jump from a single rare event, because surprisal $- lo g p$ explodes as $p \to 0$ .

L3.3 As $p \to 0^{+}$ , $- lo g p \to + \infty$ (look at the red curve shooting up on the left of the figure). One infinite surprisal makes the sum infinite, so $H = \infty$ and $PPL = b^{\infty} = \infty$ . A single impossible-but-real token destroys the entire score. This is exactly why real models smooth probabilities so no token ever gets exactly $0$ .

Level 4 — Synthesis

L4.1 The same sentence "the cat sat" is scored by two systems on different tokenizations:

System W (word-level): tokens [the, cat, sat], $N = 3$ , probabilities ${0.4, 0.2, 0.5}$ .
System S (sub-word): tokens [the, ca, t, sat], $N = 4$ , probabilities ${0.4, 0.5, 0.6, 0.5}$ .

Compute both perplexities. Are they comparable? What is comparable?

L4.2 For System W and System S above, compute the total sentence log-probability $ln P (sentence)$ for each. Which quantity is legitimately comparable across the two tokenizers, and why?

Recall Solution L4

L4.1 System W: product $= 0.4 \cdot 0.2 \cdot 0.5 = 0.04$ ; $PPL_{W} = 0.0 4^{- 1/3}$ . $ln 0.04 = - 3.218876$ ; $/3 = - 1.072959$ ; negate $1.072959$ ; $e^{1.072959} \approx 2.9240$ . System S: product $= 0.4 \cdot 0.5 \cdot 0.6 \cdot 0.5 = 0.06$ ; $PPL_{S} = 0.0 6^{- 1/4}$ . $ln 0.06 = - 2.813411$ ; $/4 = - 0.703353$ ; negate $0.703353$ ; $e^{0.703353} \approx 2.0205$ . Not comparable. $N$ differs ( $3$ vs $4$ ) and the per-token distributions differ, so these perplexities live on different scales. A finer tokenizer packs the same sentence into more, individually-easier tokens, artificially lowering per-token perplexity. To compare fairly, normalize by a tokenizer-independent unit like bits-per-character — see Bits-Per-Character (BPC).

L4.2 System W: $ln 0.04 = - 3.218876$ . System S: $ln 0.06 = - 2.813411$ . The total log-probability of the identical text is comparable (higher = better model of that string), because it does not depend on how you cut the string into tokens — the product of conditionals telescopes to $P (whole string)$ . Here System S assigns higher probability ( $- 2.81 > - 3.22$ ), so it models the string better, even though you cannot legitimately compare their per-token perplexities directly. Dividing this total by the number of characters (not tokens) gives the tokenizer-fair metric.

Level 5 — Mastery

L5.1 (Design a smoothing floor). A model outputs a probability of $0$ for one real token out of $N = 50$ , giving all other $49$ tokens probability $0.5$ . You add smoothing: replace any probability below a floor $ε$ with $ε = 1 0^{- 3}$ (leaving the $0.5$ s unchanged for simplicity). Compute the perplexity after smoothing, and state the perplexity before smoothing.

L5.2 (Invert the metric). A benchmark reports $PPL = 20$ over a text of $N = 100$ tokens using natural log. Recover (a) the average per-token cross-entropy in nats, (b) the same in bits, and (c) the total sentence log-probability $ln P (sequence)$ .

L5.3 (Prove the branching-factor bound). Show that for any model over a vocabulary of size $V$ , the perplexity on held-out text satisfies $1 \leq PPL \leq V$ only when the model is uniform at the top; more precisely prove $PPL \geq 1$ always, with equality iff the model is perfectly certain ( $P_{i} = 1$ for every real token). State the reasoning cleanly.

Recall Solution L5

L5.1 Before smoothing: one $P_{i} = 0$ ⇒ surprisal $= + \infty$ ⇒ $PPL = \infty$ . After smoothing: $49$ tokens at $0.5$ and $1$ token at $ε = 1 0^{- 3}$ . $NLL = \frac{49 ( - l n 0.5 ) + 1 ( - l n 1 0 ^{- 3} )}{50} = \frac{49 ( 0.693147 ) + 6.907755}{50} = \frac{33.964220 + 6.907755}{50} = 0.817439.$ $PPL = e^{0.817439} \approx 2.2647$ . Smoothing turns an infinite, useless score into a finite, meaningful one — the whole point of never allowing zero probability.

L5.2 With $PPL = b^{H}$ and $b = e$ : (a) $H_{nats} = ln 20 = 2.995732$ nats. (b) $H_{bits} = lo g_{2} 20 = \frac{ln 20}{ln 2} = \frac{2.995732}{0.693147} = 4.321928$ bits. (c) $H_{nats} = - \frac{1}{N} ln P (seq)$ , so $ln P (seq) = - N H_{nats} = - 100 \times 2.995732 = - 299.5732$ .

L5.3 Every real token has $0 < P_{i} \leq 1$ , so $- lo g_{b} P_{i} \geq 0$ (log of a number $\leq 1$ is $\leq 0$ ; the minus makes it $\geq 0$ ). The average of non-negative numbers is non-negative: $H \geq 0$ . Then $PPL = b^{H} \geq b^{0} = 1$ . Equality $PPL = 1$ holds iff every surprisal is $0$ , i.e. every $P_{i} = 1$ : the model assigns all its mass to the token that actually occurs at every step — perfect certainty, zero confusion. The upper bound $PPL \leq V$ is reached at the opposite extreme, the uniform model (worst case), where $P_{i} = 1/ V$ and $PPL = V$ ; any non-uniform model that still respects a $V$ -way vocabulary can exceed $V$ only if it wastes mass on wrong tokens, so $V$ is the uniform benchmark, not a hard ceiling. Hence the clean, always-true statement is $PPL \geq 1$ .

Connections

Cross-Entropy Loss — every $H$ above is exactly this loss.
Entropy and Information Content — the $- lo g p$ curve driving L3.
Language Modeling Objective — minimizing these NLLs is the training goal.
Tokenization — the reason L4's two systems aren't comparable.
Bits-Per-Character (BPC) — the tokenizer-fair fix for L4.
Softmax and Probability Distributions — where each $P_{i}$ comes from.