Perplexity as a metric

WHAT is perplexity?

WHY the weird power of $-1/N$ ?

The raw sequence probability $P(w_1,\dots,w_N)$ shrinks toward 0 as $N$ grows (you multiply many numbers $<1$ ). So it can't compare texts of different lengths.
Taking the == $N$ -th root== turns the product into a per-token average (geometric mean).
The negative exponent flips "probability" (bigger = better) into "perplexity" (bigger = worse), so it behaves like a cost.

HOW to derive it from cross-entropy (first principles)

We build perplexity from information theory, step by step.

Step 1 — Surprise of one event. Information content (surprisal) of an event with probability $p$ is $-\log_2 p$ bits. Why? Rare events (small $p$ ) carry more information; $-\log$ makes surprise large when $p$ is small and $0$ when $p=1$ .

Step 2 — Average surprise = cross-entropy. Using the chain rule, $P(w_1,\dots,w_N) = \prod_{i=1}^{N} P(w_i \mid w_{<i})$ . The average per-token cross-entropy (in bits) is $H = -\frac{1}{N}\sum_{i=1}^{N} \log_2 P(w_i \mid w_{<i})$ Why the average? We want a length-independent "typical surprise per token."

Step 3 — Perplexity is just $2$ raised to that entropy. $\text{PPL} = 2^{H} = 2^{-\frac{1}{N}\sum_i \log_2 P(w_i\mid w_{<i})}$

Step 4 — Show this equals the definition.

2^{-\frac{1}{N}\sum_i \log_2 P_i} = 2^{\log_2 \left(\prod_i P_i\right)^{-1/N}} = \left(\prod_{i} P_i\right)^{-1/N} = P(w_1,\dots,w_N)^{-1/N}

So the two definitions are identical. ✅

Worked examples

Worked example 2) Three-token sentence, concrete numbers

The model gives $P(w_1)=0.5,\ P(w_2\mid w_1)=0.25,\ P(w_3\mid w_{<3})=0.1$ .

Product $P = 0.5 \cdot 0.25 \cdot 0.1 = 0.0125$ . Why? Chain rule.
$N=3$ , so $\text{PPL} = 0.0125^{-1/3}$ .
$\log$ : $\ln(0.0125) \approx -4.382$ ; divide by $3 \to -1.4608$ ; negate $\to 1.4608$ ; $e^{1.4608}\approx 4.31$ .
$\text{PPL} \approx 4.31$ . Why this step? The geometric mean of $\{0.5,0.25,0.1\}$ is $\approx 0.232$ , and $1/0.232 \approx 4.31$ — perplexity is the reciprocal of the typical per-token probability.

Common mistakes (steel-manned)

The 80/20 core

Recall Feynman: explain to a 12-year-old

Imagine you're guessing the next word in a story. If you're a great guesser, the real word is usually one you thought was likely — you're rarely shocked. Perplexity is like counting how many words you were "torn between" each time. If you were torn between 2 words, perplexity is 2. If you were sure and always right, it's 1 (no confusion). A bad guesser is torn between hundreds of words, so their perplexity is huge. Lower number = smarter guesser.

Flashcards

What is perplexity in one line?

The exponentiated average per-token negative log-likelihood; the effective number of equally-likely tokens the model is choosing among.

Formula for perplexity from sequence probability?

\text{PPL}=P(w_1,\dots,w_N)^{-1/N}

Relation between perplexity and cross-entropy

H

\text{PPL}=b^{H}

where

H

is the average per-token cross-entropy in base-

b

logs.

Why take the

-1/N

power?

The

N

-th root makes it a per-token geometric mean (length-independent); the negative flips probability into a cost.

Is lower or higher perplexity better?

Lower is better — the model is less surprised by the real text.

Perplexity of a uniform model over vocab size

V

Exactly

V

(worst case; recovers the branching factor).

What happens if the model gives probability 0 to a real token?

Perplexity becomes

\infty

; this motivates smoothing / avoiding zero probabilities.

Can you compare perplexities across different tokenizers?

No — PPL is per-token, so

N

and the distribution change; only compare on identical text and tokenization.

In code, how is PPL computed with natural log?

\text{PPL}=\exp\left(\frac{1}{N}\sum_i -\ln P(w_i\mid w_{<i})\right)

Does the log base affect the final perplexity?

No, as long as you exponentiate with the same base (

e^{H_{nats}}=2^{H_{bits}}

Connections

Cross-Entropy Loss — perplexity is literally $e^{\text{cross-entropy}}$ .
Entropy and Information Content — the surprisal $-\log p$ foundation.
Language Modeling Objective — minimizing NLL = minimizing perplexity.
Tokenization — vocabulary and token count change $N$ , affecting PPL.
Bits-Per-Character (BPC) — tokenizer-independent alternative.
Softmax and Probability Distributions — where per-token $P_i$ comes from.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Perplexity ka matlab simple hai: model kitna "surprise" ho raha hai real text dekh ke. Agar model ko pata hai ki next word kya aayega — yaani woh us word ko high probability deta hai — toh woh surprised nahi hota, aur perplexity kam aati hai. Kam perplexity = accha model. Isko aise socho: har step pe model kitne equally-likely options ke beech confuse hai — agar 2 options ke beech confuse hai toh PPL=2, agar sure hai toh PPL=1.

Formula ka dil ye hai: $\text{PPL}=P(\text{poora sequence})^{-1/N}$ . Yahan $N$ -th root isliye lete hain taaki har token ka average (geometric mean) mil jaaye, aur alag-alag length ke texts compare ho saken. Aur ek important connection: perplexity basically cross-entropy ka exponential hai — $\text{PPL}=e^{\text{average negative log-likelihood}}$ . Deep learning training mein hum cross-entropy loss minimize karte hain, matlab hum indirectly perplexity hi kam kar rahe hote hain.

Do practical baatein yaad rakho. Ek: agar model kisi real token ko probability 0 de de, toh $-\log 0 = \infty$ , aur perplexity infinite ho jaati hai — isiliye smoothing zaroori hai. Do: perplexity ko sirf same tokenizer aur same text pe compare karo. Alag tokenizer matlab alag number of tokens ( $N$ ), toh comparison galat ho jaayega. Iss case mein bits-per-character use karo.

Exam ya interview mein bas ye 80/20 pakdo: lower PPL = better, PPL = effective number of choices (branching factor), aur PPL = $2^H$ (entropy se jud़ा hua). Bas itna clear rahe toh perplexity poora samajh gaye.

WHAT is perplexity?

HOW to derive it from cross-entropy (first principles)

Worked examples

Common mistakes (steel-manned)

The 80/20 core

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Tokenization & Language Modeling

Connections