Foundations — Cross-entropy concept

1.3.19 · D1 · AI-ML › Probability & Statistics › Cross-entropy concept

Parent note ki ek bhi line padhne se pehle, tumhe uske har symbol par pakad honi chahiye. Yeh page assume karta hai ki tumne unme se koi nahi dekha. Hum har ek ko ek picture se build karte hain, us order mein jisme ek doosre par depend karta hai.

0. "Probability" kya hoti hai aur hum ise kaise picture karte hain?

Probability sirf $0$ aur $1$ ke beech ka ek number hai jo kehta hai ki kuch kitna likely hai.

• $0$ matlab "kabhi nahi hota"
• $1$ matlab "hamesha hota hai"
• $0.7$ matlab "lambe run mein 10 mein se 7 baar hota hai"

Picture yeh hai: ek bar socho jisme length $1$ hai aur woh pieces mein kata hua hai. Har piece ek possible outcome hai, aur piece ki width uski probability hai. Kyunki kuch na kuch toh hona hi chahiye, saare pieces milke poori bar bhar dete hain — widths $1$ tak add ho jaati hain.

Topic ko iska kyun zaroorat hai: cross-entropy do aise sliced bars compare karti hai — reality ka bar aur model ka bar.

1. Symbols $x$, $p(x)$, $q(x)$

Ab hum pieces ko naam de sakte hain.

Picture yeh hai: do bars ek doosre ke upar stacked hain. Same slices ("cat", "dog", "bird"), lekin widths alag hain — kyunki model reality ko perfectly nahi jaanta.

Topic ko iska kyun zaroorat hai: cross-entropy ka poora khel yeh hai ki "reality $p(x)$ frequency ke saath hoti hai, lekin hamare model ne sirf $q(x)$ believe kiya tha." Do bars, ek truth, ek guess.

2. Summation symbol $\sum$

${\sum }_{x}p(x)$

Picture yeh hai: tum bar ki har slice ki taraf ek ek karke point karte ho, uska number padhte ho, aur ek running total mein daal dete ho.

${\sum }_{x}p(x)=p(\text{cat})+p(\text{dog})+p(\text{bird})$

Topic ko iska kyun zaroorat hai: cross-entropy saare outcomes par ek average hai. Sigma yeh kehne ka tarika hai ki "har slice par loop karo aur surprise ka total karo, weight karke ki kitni baar hota hai."

3. Logarithm $\log$

Yeh woh symbol hai jisse zyaadatar log darte hain, isliye hum ise slowly ek picture se build karte hain.

Yeh tool kyun, say square root ya plain fraction nahi? Kyunki hum chahte hain ek aisi function jo multiply karne ko add karne mein convert kare. Woh akela property — $\log (a\times b)=\log (a)+\log (b)$ — exactly wahi hai jo humein baad mein chahiye: do independent events ke surprises add hone chahiye, lekin unki probabilities multiply hoti hain. Log ek akela tool hai jo ek ko doosre mein convert karta hai.

Picture yeh hai: probabilities ke liye (numbers $0$ aur $1$ ke beech), log curve neeche ki taraf jaata hai. Jaise $p\to 0$ (rarer aur rarer), $\log p\to -\infty$. Jaise $p\to 1$ (certain), $\log p\to 0$.

Topic ko iska kyun zaroorat hai: log chhoti probabilities ko minus sign ke saath bade numbers mein convert karta hai — "surprise" ka raw material.

4. Surprise: minus-log $-\log p(x)$

Dhyaan do ki probability ka log hamesha negative ya zero hota hai (curve axis ke neeche rehti hai). Surprise ek positive feeling honi chahiye, isliye hum sign flip karte hain.

Picture yeh hai: figure s03 se diving log curve lo aur usse axis ke upar flip karo. Ab woh $p\to 0$ ke saath infinity ki taraf uthti hai — "jitna rarer hai, utna zyada shocked hoon."

Cases ka sanity check:

• $p(x)=1$ → $-\log 1=0$. Koi surprise nahi, tumhe pata tha. ✔
• $p(x)=0.5$ → $-\ln 0.5=0.69$ nats. Thoda surprise. ✔
• $p(x)=0.01$ → $-\ln 0.01=4.6$ nats. Bada surprise. ✔
• $p(x)\to 0$ → surprise $\to \infty$. Ek "impossible" event ka hona infinitely shocking hai. ✔

Topic ko iska kyun zaroorat hai: cross-entropy average surprise hai. Yahi woh cheez hai jo average ki ja rahi hai. Shannon Entropy dekho us case ke liye jahan tum same distribution se surprise average karte ho jisse surprised ho rahe ho.

5. Surprise ko average karna: $H(p)$ aur $H(p,q)$

Ab $\sum$ aur surprise ko ek saath silte hain.

$H(p,q)$ ki shape left to right padhna:

1. Har slice $x$ par jao (yahi ${\sum }_x$ hai).
2. Poochho "yeh slice actually kitni baar hoti hai?" → weight $p(x)$.
3. Poochho "model kitna surprised tha?" → $-\log q(x)$.
4. Weight × surprise multiply karo, aur sab total karo.

Result hai reality ke across model ka average surprise, jo poora topic ek formula mein hai.

6. Unke beech ka gap: $D_{KL}$

Picture yeh hai: do stacked bars — total height $H(p,q)$ (model ka surprise) chhote $H(p)$ (reality ka unavoidable surprise) ke upar baitha hai. Upar wala extra sliver $D_{KL}$ hai.

Topic ko iska kyun zaroorat hai: $H(p)$ fixed hai (tum reality nahi badal sakte). Toh ek model ko train karna taaki $H(p,q)$ shrink ho, woh same hai jaise woh sliver $D_{KL}$ shrinkana — apna guess bar reality ke bar se match karwana. KL Divergence dekho.

7. Guesses kahan se aate hain: one-hot labels aur softmax

Do aakhri symbols jinhe parent lean karta hai.

Topic ko iska kyun zaroorat hai: classification mein, true bar one-hot hai aur guessed bar ek softmax output hai. Unke beech cross-entropy woh loss hai jo tum minimise karte ho (dekho Categorical Cross-Entropy aur Logistic Regression).

Prerequisite map

Ise upar se neeche padhna: probabilities distributions deti hain; logs surprise dete hain; sigma surprise ko average karke entropy aur cross-entropy mein laata hai; unka gap KL hai; softmax aur one-hot do bars supply karte hain; sab kuch final loss mein jaata hai.

Equipment checklist

Self-test: kya tum har ek ka jawab reveal karne se pehle de sakte ho?