Entropy and KL divergence: Worked examples — Hinglish deep dive

1.3.18 · D3 · AI-ML › Probability & Statistics › Entropy and KL divergence

Yeh page parent topic ki workbook hai. Hum formulas ko scratch se re-derive nahi karenge — balki hum har tarah ke inputs ko dhundhenge jo yeh formulas encounter kar sakti hain, aur unhe last decimal tak work out karenge. Isko ek checklist ki tarah socho: is page ke baad, koi bhi probability distribution tumhe kabhi surprise nahi karni chahiye.

Shuru karne se pehle, un do symbols ko ek baar plain words mein phir se anchor karte hain jo hum baar baar use karenge.

Neeche har jagah, $lo g$ bina base ke matlab hai $lo g_{2}$ (answers bits mein) jab tak hum "nats" na bolein, jis case mein yeh natural log $ln$ hai.

Scenario matrix

Yeh is topic ki problems mein milne wale sabhi cases ka poora landscape hai. Neeche har worked example ko us cell ke saath tag kiya gaya hai jo woh cover karta hai, aur milke yeh poori grid fill karte hain.

Cell	Kya special hai isme	Covered by
A. Uniform	sabhi outcomes equally likely → entropy hits its maximum $lo g_{2} n$	Ex 1
B. Skewed / biased	ek outcome dominate karta hai → entropy shrinks	Ex 2
C. Degenerate (certain)	ek probability $= 1$ , baaki $= 0$ → entropy $= 0$ , aur $0 lo g 0$ trap	Ex 2, Ex 3
D. KL both directions	$D_{K L} (p ∥ q)$ vs $D_{K L} (q ∥ p)$ → asymmetry made concrete	Ex 4
E. KL degenerate	kuch $q (x) = 0$ jahan $p (x) > 0$ → divergence blows up to $\infty$	Ex 5
F. Identity	$p = q$ → KL $= 0$ (sanity floor)	Ex 4 (check)
G. Cross-entropy = KL + H	one-hot label vs prediction → classification loss	Ex 6
H. Limiting behaviour	continuous knob $p \to 0$ ya $p \to 1$ : entropy ka peak kahan hai?	Ex 7
I. Word problem	real-world compression / surprise story	Ex 8
J. Exam twist	additivity + independence ek hi shot mein combine karo	Ex 9

Upar wala blue curve (isko binary entropy curve $H_{2} (p)$ kaho) woh map hai jis par hum baar baar return karte rahenge. Isko left se right padho: edges $p = 0$ aur $p = 1$ par curve floor ko touch karta hai (certainty, cell C), aur middle mein $p = 0.5$ par yeh $1$ bit ki ceiling tak pahunchta hai (cell A). Cell B slopes par rehta hai; cell H poore curve ki shape hai.

Cell A — uniform maximum

Worked example Example 1 — Fair four-sided die (Cell A)

Ek perfectly balanced tetrahedral die faces ${1, 2, 3, 4}$ dikhata hai, har ek $\frac{1}{4}$ probability ke saath. $H (X)$ bits mein nikalo.

Forecast: Padhne se pehle guess karo. Chaar equally likely outcomes — kaunsa face hai yeh jaanne ke liye kitne yes/no questions chahiye? Ek number joto karo.

Sum likho: $H (X) = - \sum_{x = 1}^{4} \frac{1}{4} lo g_{2} \frac{1}{4} .$ Yeh step kyun? Entropy sirf surprisals $- lo g_{2} p (x)$ ka average hai. Yahan har term identical hai, toh average ek term ke barabar hai.
Har ek surprisal: $- lo g_{2} \frac{1}{4} = - (- 2) = 2$ bits. Yeh step kyun? $\frac{1}{4} = 2^{- 2}$ , aur $lo g_{2} 2^{- 2} = - 2$ . Do bits = do yes/no questions ("kya yeh ${1, 2}$ mein hai?" phir "dono mein se kaunsa?").
Average: $H (X) = 4 \times \frac{1}{4} \times 2 = 2$ bits. Yeh step kyun? Sabhi chaar weighted terms $\frac{1}{4} \cdot 2$ hain; unka sum $2$ hai.

Verify: Maximum-entropy rule kehta hai $H \leq lo g_{2} n = lo g_{2} 4 = 2$ , equality ke saath hit hua kyunki die uniform hai. Curve figure par, chaar outcomes peak ko generalize karte hain: uniform hamesha ceiling par hoti hai. ✓

Cells B & C — skew aur zero-probability trap

Worked example Example 2 — Loaded coin, phir ek stuck coin (Cells B, C)

(B) Ek coin jisme $p (H) = 0.9, p (T) = 0.1$ . (C) Ek trick coin jo hamesha heads pe girta hai: $p (H) = 1, p (T) = 0$ . Dono entropies bits mein nikalo.

Forecast: Kya loaded coin ki entropy $1$ bit ke zyada kareeb hogi ya $0$ ke? Aur stuck coin $0 lo g_{2} 0$ term ke saath kya karta hai?

Loaded coin surprisals: $- lo g_{2} 0.9 = 0.152$ , $- lo g_{2} 0.1 = 3.322$ bits. Yeh step kyun? Rare events (tail) zyada surprise carry karte hain — yahi $- lo g$ shape ka poora point hai.
Weighted average: $H = 0.9 (0.152) + 0.1 (3.322) = 0.137 + 0.332 = 0.469$ bits. Yeh step kyun? Hum har surprise ko weight karte hain iske actually hone ki frequency se; rare-but-shocking tail ko uski rarity se discount kiya jaata hai.
Stuck coin (C): $H = - [1 \cdot lo g_{2} 1 + 0 \cdot lo g_{2} 0]$ . Pehla term $= 0$ kyunki $lo g_{2} 1 = 0$ . Doosra term convention $0 lo g_{2} 0 := 0$ use karta hai. Yeh step kyun? $lo g_{2} 0$ hai $- \infty$ , par ise $0$ probability se multiply kiya gaya hai. Limit $lim_{p \to 0^{+}} p lo g_{2} p = 0$ (yeh outcome kabhi hota hi nahi, toh yeh surprise add nahi kar sakta). Isliye $H = 0$ .

Verify: $0.469$ blue curve ke slope par $p = 0.9$ tick aur ceiling ke beech hota hai — $1$ se below, $0$ se above, exactly jaisa skew predict karta hai. Stuck coin $p = 1$ par floor pe land karta hai. ✓

Cell C phir se — true floor

Cells D & F — KL dono directions mein aur identity floor

Worked example Example 4 — Truth vs model, aur wapas (Cells D, F)

True coin $p = (0.5, 0.5)$ . Model coin $q = (0.7, 0.3)$ . $D_{K L} (p ∥ q)$ compute karo, phir $D_{K L} (q ∥ p)$ , bits mein. Yeh bhi confirm karo ki $D_{K L} (p ∥ p) = 0$ .

Forecast: Kya dono directions same number denge? Compute karne se pehle haan ya nahi predict karo.

$D_{K L} (p ∥ q) = 0.5 lo g_{2} \frac{0.5}{0.7} + 0.5 lo g_{2} \frac{0.5}{0.3}$ . Yeh step kyun? KL log-ratio $lo g_{2} \frac{p}{q}$ ko true frequency $p (x)$ se weight karta hai — hum apni galti ki surprise ko reality ke under average karte hain.
Ratios compute karo: $lo g_{2} \frac{0.5}{0.7} = lo g_{2} 0.714 = - 0.485$ ; $lo g_{2} \frac{0.5}{0.3} = lo g_{2} 1.667 = 0.737$ .
$D_{K L} (p ∥ q) = 0.5 (- 0.485) + 0.5 (0.737) = - 0.243 + 0.369 = 0.126$ bits. Yeh step kyun? Individual terms negative ho sakte hain, par total KL kabhi nahi ho sakta — Gibbs' inequality guarantee karta hai $\geq 0$ .
Ab flip karo. $D_{K L} (q ∥ p) = 0.7 lo g_{2} \frac{0.7}{0.5} + 0.3 lo g_{2} \frac{0.3}{0.5} = 0.7 (0.485) + 0.3 (- 0.737) = 0.340 - 0.221 = 0.119$ bits. Yeh step kyun? Ab weights hain $q (x)$ , $p (x)$ nahi — ek alag average, isliye ek alag number.
Identity check: $D_{K L} (p ∥ p) = \sum p lo g_{2} \frac{p}{p} = \sum p lo g_{2} 1 = 0$ .

Verify: $0.126 \neq = 0.119$ , toh KL asymmetric hai — ise kabhi distance treat mat karo. Dono strictly positive hain (models differ karte hain), aur $p ∥ p$ exactly floor $0$ deta hai. ✓

Cell E — divergence explode ho jaati hai

Worked example Example 5 — Jab model kuch rule out kar de (Cell E)

True $p = (0.5, 0.5)$ over ${H, T}$ , par ek overconfident model kehta hai $q = (1, 0)$ — "heads haarna impossible hai". $D_{K L} (p ∥ q)$ compute karo.

Forecast: Model probability $0$ assign karta hai kisi cheez ko jo aadhe time hoti hai. Finite waste, ya catastrophe?

$T$ ka term: $0.5 lo g_{2} \frac{0.5}{0} = 0.5 lo g_{2} (\infty) = + \infty$ . Yeh step kyun? Yahan $p (T) = 0.5 > 0$ par $q (T) = 0$ . Convention $D_{K L} = \infty$ jab $p > 0, q = 0$ fire karta hai. Intuitively: ek aisi event ko encode karna jise tumhare code ne impossible maana tha, infinitely many bits cost karta hai.
Toh $D_{K L} (p ∥ q) = \infty$ , heads term ki parwah kiye bina. Yeh step kyun? Ek infinite term poore sum ko dominate karta hai.

Verify: Cell D se contrast karo jahan har $q (x) > 0$ ne finite answer diya. ML ke liye lesson: model ko kabhi hard $0$ probability output mat karne do (isliye label smoothing aur softmax, jo sabhi $q (x) > 0$ rakhte hain). ✓

Cell G — cross-entropy, KL, aur classification loss

Worked example Example 6 — One-hot label vs prediction (Cell G)

True label (class 2 of 3): $p = [0, 1, 0]$ . Model prediction $q = [0.1, 0.7, 0.2]$ . Cross-entropy loss $H (p, q)$ nats mein nikalo, aur dikhao ki yeh yahan $D_{K L} (p ∥ q)$ ke barabar hai.

Forecast: Kyunki $p$ one-hot hai, sum ke actually kitne terms survive karenge?

$H (p, q) = - \sum_{x} p (x) ln q (x) = - [0 \cdot ln 0.1 + 1 \cdot ln 0.7 + 0 \cdot ln 0.2]$ . Yeh step kyun? Cross-entropy model ko score karta hai us surprise se jo woh true outcome ko assign karta hai, true distribution se weighted. Ek one-hot $p$ har wrong-class term ko zero kar deta hai.
Survivor: $H (p, q) = - ln 0.7 = 0.357$ nats. Yeh step kyun? Sirf class 2 mein $p > 0$ hai, toh sirf $- ln q (2)$ bachta hai — correct class ka negative log-likelihood.
KL se link: $D_{K L} (p ∥ q) = H (p, q) - H (p)$ . Yahan $H (p) = - 1 \cdot ln 1 = 0$ (one-hot certain hai). Yeh step kyun? Ek one-hot label ka zero entropy hota hai, toh cross-entropy minimize karna hai KL minimize karna — constant $H (p)$ training ke dauran drop out ho jaata hai.
Isliye $D_{K L} (p ∥ q) = 0.357 - 0 = 0.357$ nats.

Verify: Cross-entropy $\geq H (p) = 0$ hamesha, aur $0$ sirf tab equal hoti hai jab perfect $q (2) = 1$ ho. Hamara $0.357$ nats woh penalty hai jo model $30%$ mass wrong classes par spread karne ke liye pay karta hai. Yahi exactly Cross-Entropy Loss hai jo classifiers train karne ke liye use hoti hai. ✓

Cell H — limiting behaviour, peak kahan hai

Worked example Example 7 — Bias knob ghoomana (Cell H)

Ek coin ke liye jisme $p (H) = p$ , binary entropy hai $H_{2} (p) = - p lo g_{2} p - (1 - p) lo g_{2} (1 - p)$ . Yeh maximum kahan hai, aur $p \to 0$ aur $p \to 1$ ke limits kya hain?

Forecast: Figure s01 mein blue curve dekho. Compute karne se pehle uske sabse unche spot aur dono endpoints point karo.

Endpoints: jaise $p \to 0^{+}$ , $- p lo g_{2} p \to 0$ aur $- (1 - p) lo g_{2} (1 - p) \to 0$ , toh $H_{2} \to 0$ . Same $p \to 1^{-}$ par. Yeh step kyun? Dono edges near-certainty hain (almost hamesha tails, ya almost hamesha heads) — koi surprise nahi, curve floor ko kiss karta hai.
Maximum: symmetry aur parent note se concavity ke zariye, peak $p = 0.5$ par hai, deta hai $H_{2} (0.5) = - 0.5 lo g_{2} 0.5 - 0.5 lo g_{2} 0.5 = 1$ bit. Yeh step kyun? Maximum uncertainty = maximally spread distribution = $n = 2$ ke liye uniform case (cell A).

Verify: $H_{2} (0.5) = 1$ , $lo g_{2} n = lo g_{2} 2 = 1$ ceiling se match karta hai. Curve $0.5$ ke baare mein symmetric hai kyunki heads↔tails relabel karna change nahi kar sakta ki tum kitne uncertain ho. ✓

Cell I — ek word problem

Worked example Example 8 — Weather forecasts compress karna (Cell I)

Ek town ka daily weather hai $p = (sun 0.7, rain 0.2, snow 0.1)$ . Ek lazy engineer compress karta hai ek uniform codebook $q = (\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ use karke. (a) True entropy $H (p)$ kya hai? (b) Uniform code har din kitne extra bits waste karta hai, yaani $D_{K L} (p ∥ q)$ ?

Forecast: Uniform code snow ko sun jitna hi treat karta hai — kya yeh bahut waste karega ya thoda?

Entropy: $H (p) = - [0.7 lo g_{2} 0.7 + 0.2 lo g_{2} 0.2 + 0.1 lo g_{2} 0.1]$ .
- $0.7 lo g_{2} 0.7 = 0.7 (- 0.5146) = - 0.3602$
- $0.2 lo g_{2} 0.2 = 0.2 (- 2.3219) = - 0.4644$
- $0.1 lo g_{2} 0.1 = 0.1 (- 3.3219) = - 0.3322$ Toh $H (p) = 0.3602 + 0.4644 + 0.3322 = 1.157$ bits. Yeh step kyun? Yeh is weather ke liye best possible average code length hai.
KL to uniform: $D_{K L} (p ∥ q) = \sum_{x} p (x) lo g_{2} \frac{p ( x )}{1/3} = \sum_{x} p (x) (lo g_{2} p (x) + lo g_{2} 3)$ . Yeh step kyun? $lo g_{2} \frac{p}{1/3} = lo g_{2} p + lo g_{2} 3$ ; constant $lo g_{2} 3$ sum se bahar aa jaata hai.
Kyunki $\sum_{x} p (x) = 1$ : $D_{K L} = lo g_{2} 3 - H (p) = 1.585 - 1.157 = 0.428$ bits. Yeh step kyun? $\sum_{x} p (x) lo g_{2} p (x) = - H (p)$ , toh poori cheez neatly collapse ho jaati hai. Uniform code ka KL hamesha $lo g_{2} n - H (p)$ ke barabar hota hai.

Verify: Lazy code $1.157 + 0.428 = 1.585 = lo g_{2} 3$ bits/day cost karta hai — exactly 3 equally-likely symbols encode karne ki cost, jo uniform code weather ko pretend karta hai. $0.428$ -bit waste positive hai, jaisa har KL hona chahiye. ✓

Cell J — exam twist

Worked example Example 9 — Ek saath do independent dice (Cell J, additivity)

Tum Ex 1 wala fair four-sided die roll karte ho aur parent note wala fair coin flip karte ho, independently. Joint entropy $H (X, Y)$ kya hai?

Forecast: Independent sources of surprise — kya unki entropies add hoti hain, multiply hoti hain, ya kuch aur messier?

Independence ka matlab hai $p (x, y) = p (x) p (y)$ , toh $lo g_{2} p (x, y) = lo g_{2} p (x) + lo g_{2} p (y)$ . Yeh step kyun? Log independent probabilities ke product ko surprisals ka sum bana deta hai — yahi additivity property hai jisne $lo g$ ko pehli jagah sahi tool banaya.
Isliye $H (X, Y) = H (X) + H (Y) = 2 + 1 = 3$ bits. Yeh step kyun? Expectation linear hai, toh surprisals ke sum ka average, averages ka sum hota hai. Die contribute karta hai $2$ bits (Ex 1), coin $1$ bit.
Joint uniform ke through sanity check: pair ke $4 \times 2 = 8$ equally likely outcomes hain, aur $lo g_{2} 8 = 3$ . Yeh step kyun? Do independent uniforms ek bada uniform banate hain, jiska entropy hai $lo g_{2} (total outcomes)$ .

Verify: $H (X) + H (Y) = 3 = lo g_{2} 8$ . Additivity sirf isliye hold karta hai kyunki yeh independent hain; agar coin die par depend karta, toh $H (X, Y) < H (X) + H (Y)$ — gap hai Mutual Information. ✓

Yahan se kahan jaate hain

Waste-bits idea generalize hota hai: KL F-divergences family ka ek member hai, aur uska symmetrized cousin hai Jensen-Shannon Divergence.
Constraints ke subject mein maximum entropy wali distribution choose karna hai Maximum Entropy Principle.
Deep generative models mein, ek prior ke against KL term minimize karna Variational Autoencoders aur Evidence Lower Bound (ELBO) ka core hai.
Decision trees entropy drop par split karte hain jise Information Gain kehte hain.

Recall Self-test (jawab dene ke baad reveal karo)

Fair 4-sided die entropy ::: $2$ bits ( $= lo g_{2} 4$ ) Loaded coin $p (H) = 0.9$ entropy ::: $0.469$ bits $D_{K L} (p ∥ q)$ for $p = (.5, .5), q = (.7, .3)$ ::: $0.126$ bits Same but reversed $D_{K L} (q ∥ p)$ ::: $0.119$ bits (asymmetric!) KL jab $q$ ek aisi event ko $0$ assign kare jisme $p > 0$ ho ::: $+ \infty$ Cross-entropy of $p = [0, 1, 0]$ , $q = [0.1, 0.7, 0.2]$ ::: $- ln 0.7 = 0.357$ nats Weather $H (p)$ for $(0.7, 0.2, 0.1)$ ::: $1.157$ bits Us weather ka KL to uniform ::: $0.428$ bits Joint entropy of fair 4-die + fair coin ::: $3$ bits