Cross-entropy concept: Visual walkthrough — Hinglish deep dive

1.3.19 · D2 · AI-ML › Probability & Statistics › Cross-entropy concept

Yeh page cross-entropy ko scratch se build karta hai. Koi bhi probability formula pehle se assume nahi kiya gaya. Hum sirf ek sawaal se shuru karte hain — "mujhe kitna surprised hona chahiye?" — aur ek-ek picture ke saath chalte hain, jab tak hum us loss function tak nahi pahunch jaate jo almost har classifier ko train karta hai.

Do conventions pehle se fix, jo neeche har jagah use honge.

Kaun sa logarithm? Is poore page mein $lo g$ ka matlab hai natural logarithm — wahi jo $ln$ likha jaata hai, jiska base $e \approx 2.718$ hai. Hum ise isliye choose karte hain kyunki yahi log calculus aur gradient-based training mein built-in hai. Natural log ke saath, surprise ek unit mein measure hoti hai jise nat kehte hain (short for "natural unit of information"). Agar hum base- $2$ logarithm use karte, toh same quantities bits mein aate; shapes aur conclusions bilkul same hain, sirf number-per-unit badalta hai.
Kaun sa outcome space? Hum ek finite discrete set of possible outcomes assume karte hain — ek fixed chhoti list jaise ${cat, dog, bird}$ — taaki neeche har " $\sum_{x}$ " ek ordinary finite sum ho jo guarantee se sense make kare. Agar outcomes continuous hote (ek real number jo kuch bhi ho sakta), toh sum ek integral ban jaata, $H (p, q) = - \int p (x) lo g q (x) d x$ , aur $p, q$ probability densities hote na ki probabilities; is page ki har intuition unchanged rehti hai, bas hum "list par sum" ko "curve ke neeche area" se replace kar dete hain.

Baaki sab kuch game ke do players se nikalta hai:

ek true distribution — reality actually kya karti hai,
ek predicted distribution — hamaara model reality ke baare mein kya andaza lagaata hai.

Hum har symbol use karne se pehle use earn karenge.

Step 1 — Surprise ek number hai

KYA. Kisi bhi formula se pehle, hum ek idea par agree karte hain: kuch events hume shock karte hain, kuch nahi. Agar main aapko boluun "aaj subah suraj nikla," toh aap kandhey uchkaate hain. Agar main boluun "desert mein barf padi," toh aap hairan ho jaate hain. Surprise bas yeh hai ki woh hairaani kitni badi hai, ek number mein convert ki gayi.

Ek probability ko $q$ likhte hain — ek number $0$ aur $1$ ke beech jo batata hai ki model ko lagta tha ek event kitna likely tha. Hum ek aisi machine banaane wale hain jo is $q$ ko leti hai aur ek surprise-number deti hai. Us machine ko ek naam dete hain: $I (q)$ likhte hain "woh surprise jo aap tab feel karte hain jab ek event jo model ne $q$ rate kiya tha actually happen ho jaaye." Abhi $I$ bas ek blank box hai — humne abhi uska formula decide nahi kiya, sirf uska naam. Yeh step yeh decide karne ke baare mein hai ki box ko kaun se rules follow karne chahiye.

KYUN. Hum measure karna chahte hain ki prediction kitni achi hai. Iska ek hi honest measure hai: jab sach reveal hota hai, toh hum kitne shocked the? Ek accha predictor rarely shocked hota hai; ek bura predictor constantly shocked rehta hai.

Hum blank box $I (q)$ se teen cheezein maangte hain:

Agar model certain tha ki yeh hoga ( $q = 1$ ), toh surprise zero honi chahiye — koi shock nahi.
Agar model ne socha tha yeh impossible hai ( $q \to 0$ ) lekin ho gaya, toh surprise enormous honi chahiye.
Do independent events ki surprises add honi chahiye: do independent shocks dekhna ek shock plus doosra hai.

PICTURE. Neeche ke figure mein, horizontal axis model ki probability $q$ hai (left par $0$ se right par $1$ tak) aur vertical axis surprise $I (q)$ in nats hai. Curve left par high hai (rare event = shocking), zero tak slide karta hai right par (certain event = no shock) — exactly woh shape jo hamaari teen demands force karti hain.

Intuition Kyun yeh teen demands sirf logarithm ko chhod deti hain

Teesri demand par focus karo: independent events ke liye, probabilities multiply hoti hain ( $q_{1} q_{2}$ ) lekin surprises add honi chahiye. Toh hume ek aisa box chahiye jo multiply-inside ko add-outside mein convert kare: $I (q_{1} q_{2}) = I (q_{1}) + I (q_{2})$ . Try karo $q = a \cdot b$ ko bahut saare equal pieces se banate hue: agar $I (q^{n}) = n I (q)$ har whole number $n$ ke liye (jo adding rule force karti hai), toh ek probability ki surprise us proportion mein badhti hai ki aapne kitni baar multiply kiya — aur "kitni baar multiply kiya $q$ tak pahunchne ke liye" exactly wahi hai jo ek logarithm count karta hai. Ek hidden hypothesis zaroori hai: hum require karte hain ki $I$ , $q$ mein continuous ho (equivalently, monotonic). Iske bina, weird "pathological" solutions exist karti hain jo adding rule sirf rational points par satisfy karti hain aur baaqi jagah chaotically jump karti hain. Ek baar jab hum insist karte hain ki $I$ smoothly vary kare, adding rule "multiply-in becomes add-out" force karti hai ki $I (q) = c lo g q$ kisi constant $c$ ke liye; pehli do demands phir sign aur scale fix karti hain. Isi sense mein shape choose nahi ki jaati balki forced hoti hai.

Step 2 — Logarithm hi woh akela surprise hai jo add karta hai

KYA. Ab hum Step 1 ke blank box ko uske forced formula se fill karte hain. Ek event ki surprise jise model ne probability $q$ di woh hai:

$surprise box I (q) = flip sign so it’s positive - natural log: turns \times into + lo g model’s probability q$

KYUN logarithm aur, say, $1 - q$ nahi? Kyunki additivity demand ki wajah se (saath mein Step 1 mein stated continuity assumption ke). Independent probabilities multiply hoti hain: do independent events jinki probabilities $q_{1}$ aur $q_{2}$ hain, saath mein $q_{1} q_{2}$ probability se hote hain. Hum chahte the ki unki surprises add hon. " $\times$ inside" ko " $+$ outside" mein convert karne wala akela continuous function logarithm hai:

$lo g (q_{1} q_{2}) = lo g q_{1} + lo g q_{2} \Rightarrow I (q_{1} q_{2}) = I (q_{1}) + I (q_{2}) .$

$1 - q$ jaisi rule isse instantly fail karti hai ( $1 - q_{1} q_{2} \neq = (1 - q_{1}) + (1 - q_{2})$ ), toh yeh surprise nahi ho sakti.

KYUN minus sign? Ek probability at most $1$ hai, aur $1$ se chhote number ka $lo g$ $\leq 0$ hota hai. Surprise positive honi chahiye, isliye hum sign $-$ se flip karte hain. Figure dikhata hai ki $- lo g q$ infinity tak chahda jaata hai jab $q \to 0$ aur exactly $q = 1$ par zero ko touch karta hai — woh do anchors jo humne Step 1 mein demand ki thi.

PICTURE.

Step 3 — True distribution par average surprise

KYA. Ek event ek surprise deta hai. Lekin reality hum par bahut saare events throw karti hai time ke saath. Hum chahte hain average surprise per event across poore din ki forecasts.

Ab doosra player enter karta hai. Reality ki apni probabilities hain: $p (x)$ likhte hain ki outcome $x$ actually kitni baar hoti hai, jahaan $x$ hamare fixed finite list of outcomes par range karta hai. Averaging ka matlab hai: har outcome ki surprise ko weight karo us frequency se jitni reality use serve karti hai.

$H (p, q) = \sum_{x} how often x truly happens p (x) \times surprise we feel when it happens (- lo g q (x))$

KYUN $p$ se weight karo aur $q$ se nahi? Kyunki reality — hamaara model nahi — decide karti hai ki har outcome kitni frequently aati hai. Agar red actually 70% time aata hai, toh hamaari 70% surprises red-surprises hain. Hum woh surprise feel karte hain jo model $q$ assign karta hai, lekin hum use utni baar feel karte hain jitni reality $p$ dictate karti hai. Yeh split — $q$ feel karo, $p$ se weight — cross-entropy ka poora dil hai.

PICTURE. Bar chart mein x-axis par outcomes hain. Har bar ki height true frequency $p (x)$ hai; har bar ki color intensity surprise $- lo g q (x)$ hai. Cross-entropy total shaded area hai — frequency times surprise, summed.

Step 4 — Best possible score: entropy

KYA. Aap kabhi bhi sabse kum score kab paa sakte hain? Yeh tab hota hai jab aapka model perfect ho: $q (x) = p (x)$ har jagah. Tab cross-entropy true frequencies ke against surprise ban jaati hai:

$H (p) = - \sum_{x} true freq p (x) lo g same true freq p (x)$

Yeh reality ki Shannon Entropy hai — duniya mein built-in irreducible surprise, again nats mein.

KYUN floor exist karta hai? Even ek perfect forecaster kabhi kabhi surprised hota hai, kyunki reality genuinely random hai. Agar ek fair coin 50/50 hai, koi bhi forecast agle flip ke shock ko nahi hata sakta. Woh leftover shock $H (p)$ hai — woh best score jo physics allow karta hai.

PICTURE. "Model kitna acha hai" ke upar do curves: cross-entropy $H (p, q)$ ek valley hai jiska lowest point exactly horizontal line $H (p)$ par baithta hai, sirf tab pahuncha jaata hai jab $q = p$ .

Step 5 — Floor ke upar ka gap KL divergence hai

KYA. Cross-entropy floor plus ek penalty hai galat hone ki. Penalty isolate karne ke liye floor subtract karo:

$wasted surprise D_{K L} (p ∥ q) = aapka score H (p, q) - best possible H (p) = - \sum_{x} p (x) lo g \frac{q ( x )}{p ( x )}$

Rearranged, star equation:

$H (p, q) = H (p) + D_{K L} (p ∥ q)$

$H (p)$ — fixed; reality ki apni randomness. Aap kuch bhi karo yeh nahi badlega.
$D_{K L} (p ∥ q)$ — KL Divergence; woh extra surprise jo aap purely galat guess karne ke liye pay karte hain. Zero sirf tab jab $q = p$ , warna positive.

KYUN dono forms agree karte hain. Divergence ko ratio $\frac{q ( x )}{p ( x )}$ inside aur leading minus ke saath likhna $+ \sum_{x} p (x) lo g \frac{p ( x )}{q ( x )}$ ke same hai, kyunki ek fraction ko log ke andar flip karna sign flip karta hai: $- lo g \frac{q}{p} = lo g \frac{p}{q}$ . Woh log split karo, $lo g \frac{p}{q} = lo g p - lo g q$ , aur sum karne par exactly $H (p, q) - H (p)$ milta hai — isi tarah box equation janam leti hai.

KYUN training ke liye yeh matter karta hai. Jab aap model train karte hain toh sirf $q$ move kar sakte hain. Kyunki $H (p)$ ek constant hai, $H (p, q)$ minimize karna exactly $D_{K L} (p ∥ q)$ minimize karna hai — apne guess ko truth ki taraf drag karna. Yahi wajah hai ki cross-entropy minimize karna Maximum Likelihood Estimation ke equal hai.

PICTURE. Ek stacked bar: neeche ek fixed grey block $H (p)$ on the bottom, ek shrinking colored block $D_{K L}$ on top. Training top block ko melt karti hai; bottom kabhi nahi hiltaa.

Step 6 — Degenerate case: one-hot labels

KYA. Classification mein, reality usually certain hoti hai: image hai ek cat. Tab $p$ one-hot hai — ek single $1$ true class par, baaki sab $0$ , e.g. $p = [1, 0, 0]$ .

Ise sum mein plug karo. Har term jahan $p (x) = 0$ die ho jaata hai (zero times anything is zero). Sirf true class bachti hai:

$H (p, q) = - \sum_{x} p (x) lo g q (x) = - 1 \cdot lo g q (true) - 0 - 0 = correct class ki surprise - lo g q_{true}$

KYUN collapse hota hai. Ek one-hot truth ke saath, sirf ek outcome kabhi hota hai, toh hum sirf wahi surprise feel karte hain jo us ek outcome se attached hai. Poora sum us single log tak reduce ho jaata hai jo model ne sahi answer ko diya tha.

KYUN entropy floor bhi gayab ho jaata hai. One-hot $p$ ke liye, $H (p) = - 1 lo g 1 = 0$ : certain reality ki zero built-in randomness hoti hai. Toh yahan $H (p, q) = D_{K L} (p ∥ q)$ exactly — cross-entropy is KL divergence.

PICTURE. Step 3 ka bar chart, lekin ab ek ko chhod kar saari bars ki height $0$ hai. Sirf true-class bar contribute karta hai; uska shaded area hi poora loss hai.

Step 7 — Edge cases: zeros, certainty, aur asymmetry

KYA. Teen corners dikhane zaroori hain taaki aap kabhi koi unshown scenario na hit karo.

Model kehta hai impossible, lekin ho jaata hai ( $q_{true} \to 0$ ): $- lo g q_{true} \to + \infty$ . Loss unbounded hai. Yahi wajah hai ki real code $q$ ko $0$ se door clamp karta hai (e.g. ek tiny $ε$ add karta hai).
Ek truly impossible outcome ( $p (x) = 0$ ): uska term $0 \cdot lo g q (x) = 0$ ho jaata hai, drop out — chahe $q (x)$ kitna bhi tiny ho. Reality ise kabhi serve nahi karti, toh hum kabhi uski surprise feel nahi karte. (Hum $0 lo g 0 = 0$ limit se define karte hain.)
Asymmetry ( $H (p, q) \neq = H (q, p)$ ): players ko swap karna ek alag sawaal poochhtaa hai. $H (p, q)$ surprise ko reality se weight karta hai; $H (q, p)$ model se weight karta hai. Alag weights, alag number.

KYUN dikhaaen. Ek learner jo sirf tidy case dekhe code mein zero se divide karega ya galat $p$ aur $q$ swap karega. Corner 1 mein infinity koi bug nahi — yeh cross-entropy ka cheekh ke kehna hai "tumne truth ko rule out kar diya."

PICTURE. Left panel: $- lo g q$ wall rockets to infinity jab $q \to 0$ . Right panel: do side-by-side bars dikhate hain ki $H (p, q)$ aur $H (q, p)$ same pair ke liye alag heights rakhte hain.

Ek-picture summary

Sab kuch ek canvas par: reality ki frequencies $p$ (bar heights) model ki surprise $- lo g q$ (curve) se milti hain. Unka weighted total $H (p, q)$ hai, jo cleanly split hota hai immovable floor $H (p)$ mein plus shrinkable gap $D_{K L} (p ∥ q)$ mein jise training zero tak drive karti hai.

Recall Poore walkthrough ki Feynman retelling

Ek bag of colored balls imagine karo. Reality mix decide karti hai — maan lo 70% red, 30% blue; woh mix hai $p$ . Har draw se pehle aap apna bet chillate hain: "80% red!"; woh bet hai $q$ .

Jab ek ball aata hai, aapka shock hai $- lo g$ (natural log) us chance ka jo aapne us color ko diya tha. Kisi color ki chance near zero guess karo aur woh aa jaaye? Aap gasp karo. Near one guess karo aur aa jaaye? Aap barely blink karo. Hum $lo g$ use karte hain kyunki do separate shocks simply add up hone chahiye, aur sirf $lo g$ "chances multiply" ko "shocks add" mein convert karta hai.

Poora din karo aur apna shock average karo, lekin har color ko us frequency se weight karo jitni bag (reality $p$ ) actually use deta hai. Woh weighted-average shock hai cross-entropy $H (p, q)$ , aur uski unit nat hai.

Sabse kum score jo aap kabhi hit kar sakte hain woh tab hai jab aapke bets exactly bag se match karein — lekin tab bhi aap thoda surprised rehte hain, kyunki bag genuinely random hai. Woh leftover shock hai entropy $H (p)$ , ek floor jiske neeche aap nahi khod sakte. Jo bhi surprise aap us floor ke upar pay karte hain woh pure waste hai galat bet karne se — wahi hai KL divergence. Kyunki floor kabhi nahi hiltaa, "guessing mein better hona" literally ka matlab hai "KL waste ko zero pe pighlaana," jo exactly wahi hai jo ek classifier train karna karta hai.

Quick self-check

Neeche, " $lo g$ " natural log hai; har ek ko haath se karo taaki numbers kabhi magic na lagein.

Coin: reality

p = 0.7

heads, model

q = 0.8

heads.

H (p, q)

compute karo.

- [0.7 lo g 0.8 + 0.3 lo g 0.2] = - [0.7 (- 0.223) + 0.3 (- 1.609)] = - [- 0.156 - 0.483] = 0.639

nats.

Uska entropy floor

H (p)

- [0.7 lo g 0.7 + 0.3 lo g 0.3] = - [0.7 (- 0.357) + 0.3 (- 1.204)] = - [- 0.250 - 0.361] = 0.611

nats.

Wasted bits

D_{K L}

H (p, q) - H (p) = 0.639 - 0.611 = 0.028

nats.

One-hot

p

sum ko kyun collapse karta hai?

Har

p (x) = 0

term die ho jaata hai, sirf

- lo g q_{true}

bachta hai.

lo g

(na ki

1 - q

) surprise rule kyun hai?

Sirf

lo g

independent surprises ko add karwaata hai.