Foundations — Entropy and KL divergence

1.3.18 · D1 · AI-ML › Probability & Statistics › Entropy and KL divergence

Ye page parent note topic note mein use hone wale notation ke har piece ko build karta hai, ek smart 12-saal ke bache se shuru hokar jisne kabhi $\sum$ sign nahi dekha. Ise upar se neeche padho; har symbol earn hota hai pehle use hone se.

1. Ek probability $p(x)$ — certainty ka ek tukda

Kisi bhi formula se pehle, hume raw material chahiye: probability.

Picture. Ek bar imagine karo jiska total length $1$ hai. Tum ise pieces mein kaat lo, har possible outcome ke liye ek piece. Ek piece ki width uski probability hai. Saari widths milke exactly bar bhar deni chahiye — na kuch missing, na kuch overflow.

Topic ko isko kyun chahiye. Entropy aur KL divergence averages hain jo in widths se weighted hain. Ek wide (likely) outcome average ko apni value ki taraf kheenchta hai; ek thin (rare) outcome use bahut kam hilata hai. Probabilities ke bina weigh karne ke liye kuch hota hi nahi.

2. Random variable $X$ aur uska alphabet $\mathcal{X}$

$x\in \mathcal{X}$ ko padho "$x$ list $\mathcal{X}$ mein se chuna gaya ek item hai." Symbol $\in$ ka matlab bas "belongs to" hai.

Picture. $\mathcal{X}$ wo labels ka set hai jo §1 ke certainty bar ke har slice ke neeche likhe hain. Agar bar mein teen slices hain, toh $\mathcal{X}=\{\text{sun},\text{rain},\text{snow}\}$ aur $|\mathcal{X}|=3$ — vertical bars $|\cdots |$ ka matlab hai "andar kitne items hain count karo."

Topic ko isko kyun chahiye. Maximum-entropy property $H(X)\le {\log }_2|\mathcal{X}|$ literally count karti hai kitne outcomes exist karte hain. Woh $|\mathcal{X}|$ exactly yahi list-length hai.

3. Sum sign $\sum$ — "har slice pe add karo"

Parent page ke har formula ki shuruaat $\sum$ se hoti hai. Ye scary nahi hai.

$\mathcal{X}=\{a,b,c\}$ ke liye: ${\sum }_{x\in \mathcal{X}}f(x)=f(a)+f(b)+f(c)$

Picture. Certainty bar ke slice-by-slice jao, har slice ka contribution ek bucket mein daalo, aur end mein bucket padho.

4. Logarithm $\log$ — "kitni halvings?" ko ek number mein banana

Ye woh tool hai jis par ye poora topic bana hai, toh hum ise carefully install karte hain.

Examples jo tum haath se check kar sakte ho:

• ${\log }_{2}1=0$ (already wahan, koi doubling nahi).
• ${\log }_{2}2=1$, ${\log }_{2}4=2$, ${\log }_{2}8=3$.
• ${\log }_2\frac{1}{2}=-1$ (ek halving — negative kyunki hum shrink hue).

Additivity jo log earn karta hai. Agar event $x$ ki probability $p$ hai aur independent event $y$ ki probability $q$ hai, toh pair probability $p\cdot q$ se hota hai. Tab $-{\log }_2(p\cdot q)=-{\log }_{2}p+(-{\log }_{2}q).$ Pair ki surprise = dono ki surprise, add hui. Ye single line hai kyun $-{\log }_{2}p$ (jise surprisal kehte hain) sahi meter hai, aur kyun $\log$ entropy ke dil mein baith ta hai.

5. Surprisal $I(x)=-{\log }_{2}p(x)$ — ek slice ki shock ki height

Picture. Rare slice (thin, chhota $p$) → tall surprise spike. Certain slice ($p=1$) → flat zero. Surprisal ko probability ke against plot karo aur tumhe ek curve milta hai jo $p=1$ par $0$ tak dive karta hai aur $p\to 0$ par $+\infty$ tak shoot karta hai.

Topic ko isko kyun chahiye. Entropy in spike heights ka average hai, aur cross-entropy un spikes ka average hai jo galat meter se compute ki gayi hain — toh surprisal wo atom hai jis se dono bane hain.

6. Expectation ${\mathbb{E}}_{x\sim p}[\cdot ]$ — weighted average

Picture. See-saw ka balance point: har outcome ke liye position $f(x)$ par weight $p(x)$ rakho; expectation wahan hai jahan plank balance karti hai. Wide slices zyada bhaari hoti hain aur balance ko apni value ki taraf kheenchti hain.

Topic ko isko kyun chahiye. Is ek symbol se parent ke do headline formulas plain English mein collapse ho jaate hain: $H(X)={\mathbb{E}}_{x\sim p}[I(x)]\text{(average surprise)},D_{KL}(p\Vert q)={\mathbb{E}}_{x\sim p}\left[\log \frac{p(x)}{q(x)}\right]\text{(average wasted surprise)}.$

7. Do distributions $p$ aur $q$ — sach vs. tumhara guess

Picture. Do certainty bars ek ke upar ek. Jahan $q$-slice, $p$-slice se narrower hai, tumhara model ek common event ko under-expect kar raha hai — aur tum iske liye pay karte ho. KL divergence uss payment ka running tally hai.

8. Wo conventions jo formulas ko explode hone se rokti hain

Picture. Jaise ek slice ki width zero ki taraf shrink hoti hai, uska surprisal spike tall hota jaata hai, lekin use multiply karne wali width aur tez si shrink hoti hai — product (ek thin, tall sliver of area) kuch nahi ban jaata. See-saw ko zero-width slice se koi weight feel nahi hota.

Prerequisite map

Ye map upar use kiya gaya dependency order dikhata hai: probabilities aur logs roots hain; expectation surprisal ko entropy mein fuse karta hai; do distributions plus log-ratio KL divergence dete hain; dono parent topic ko feed karte hain. Downstream, yahi tools Cross-Entropy Loss, Mutual Information, Information Gain, Maximum Entropy Principle, Jensen-Shannon Divergence, F-divergences, aur Variational Autoencoders mein use hone wala Evidence Lower Bound (ELBO) power karte hain.

Equipment checklist

Right side cover karo aur khud ko test karo. Agar tum inhe sab answer kar sako, toh tum parent note ke liye ready ho.