Perplexity as a metric — Hinglish notes

4.2.9 · AI-ML › Tokenization & Language Modeling

Perplexity KYA hai?

$- 1/ N$ ki yeh ajeeb power kyun?

Raw sequence probability $P (w_{1}, \dots, w_{N})$ jaise-jaise $N$ badhta hai, 0 ki taraf shrink hoti jaati hai (aap bahut se numbers $< 1$ multiply karte ho). Toh yeh alag-alag length ke texts ko compare nahi kar sakti.
== $N$ -th root== lene se product ek per-token average (geometric mean) ban jaata hai.
Negative exponent "probability" (bada = better) ko "perplexity" (bada = worse) mein flip karta hai, toh yeh ek cost ki tarah behave karta hai.

Cross-entropy se isko HOW derive karte hain (first principles)

Hum perplexity ko information theory se step by step build karte hain.

Step 1 — Ek event ki Surprise. $p$ probability wale event ka information content (surprisal) $- lo g_{2} p$ bits hota hai. Kyun? Rare events (chhota $p$ ) zyada information carry karte hain; $- lo g$ surprise ko bada banata hai jab $p$ chhota ho aur $0$ jab $p = 1$ ho.

Step 2 — Average surprise = cross-entropy. Chain rule use karke, $P (w_{1}, \dots, w_{N}) = \prod_{i = 1}^{N} P (w_{i} ∣ w_{< i})$ . Average per-token cross-entropy (bits mein) hai $H = - \frac{1}{N} \sum_{i = 1}^{N} lo g_{2} P (w_{i} ∣ w_{< i})$ Average kyun? Hum ek length-independent "typical surprise per token" chahte hain.

Step 3 — Perplexity bas $2$ ko us entropy par raise karna hai. $PPL = 2^{H} = 2^{- \frac{1}{N} \sum_{i} l o g_{2} P (w_{i} ∣ w_{< i})}$

Step 4 — Dikhao ki yeh definition ke barabar hai.

2^{- \frac{1}{N} \sum_{i} l o g_{2} P_{i}} = 2^{l o g_{2} (\prod_{i} P_{i})^{- 1/ N}} = (i \prod P_{i})^{- 1/ N} = P (w_{1}, \dots, w_{N})^{- 1/ N}

Toh dono definitions identical hain. ✅

Worked examples

Common mistakes (steel-manned)

80/20 core

Recall Feynman: 12-saal ke bachche ko samjhao

Socho tum ek story mein agla word guess kar rahe ho. Agar tum ek great guesser ho, real word usually wahi hota hai jo tumhe likely laga tha — tum rarely shocked hote ho. Perplexity yeh count karna jaisa hai ki tum har baar kitne words ke beech "torn" the. Agar tum 2 words ke beech torn the, perplexity 2 hai. Agar tum sure the aur hamesha sahi rahe, toh 1 hai (koi confusion nahi). Ek bura guesser hazaaron words ke beech torn hota hai, toh unki perplexity bahut badi hoti hai. Chhota number = smarter guesser.

Flashcards

Ek line mein perplexity kya hai?

Exponentiated average per-token negative log-likelihood; equally-likely tokens ki effective number jinmein se model choose kar raha hota hai.

Sequence probability se perplexity ka formula?

PPL = P (w_{1}, \dots, w_{N})^{- 1/ N}

Perplexity aur cross-entropy

H

ka relation?

PPL = b^{H}

jahan

H

base-

b

logs mein average per-token cross-entropy hai.

- 1/ N

power kyun lete hain?

N

-th root isse per-token geometric mean banata hai (length-independent); negative probability ko cost mein flip karta hai.

Lower ya higher perplexity better hai?

Lower better hai — model real text se kam surprised hota hai.

Vocab size

V

pe uniform model ki perplexity?

Exactly

V

(worst case; branching factor recover karta hai).

Agar model ek real token ko probability 0 deta hai toh kya hoga?

Perplexity

\infty

ho jaati hai; yahi smoothing / zero probabilities avoid karne ki motivation hai.

Alag tokenizers mein perplexities compare kar sakte hain?

Nahi — PPL per-token hai, toh

N

aur distribution change ho jaata hai; sirf identical text aur tokenization pe compare karo.

Code mein natural log se PPL kaise compute hoti hai?

PPL = exp (\frac{1}{N} \sum_{i} - ln P (w_{i} ∣ w_{< i}))

Kya log base final perplexity affect karta hai?

Nahi, jab tak same base se exponentiate karo (

e^{H_{na t s}} = 2^{H_{bi t s}}

Connections

Cross-Entropy Loss — perplexity literally $e^{cross-entropy}$ hai.
Entropy and Information Content — surprisal $- lo g p$ ka foundation.
Language Modeling Objective — NLL minimize karna = perplexity minimize karna.
Tokenization — vocabulary aur token count $N$ change karte hain, PPL affect hoti hai.
Bits-Per-Character (BPC) — tokenizer-independent alternative.
Softmax and Probability Distributions — jahan se per-token $P_{i}$ aata hai.