Cross-entropy concept: Worked examples — Hinglish deep dive

1.3.19 · D3 · AI-ML › Probability & Statistics › Cross-entropy concept

Yeh page parent note ki "brute-force" companion hai. Wahan humne definition banai thi. Yahan hum har tarah ka input dhundte hain jo cross-entropy face kar sakti hai, aur har ek ko poora work karte hain — taaki koi bhi case aisa na ho jo tumne pehle na dekha ho.

Shuru karne se pehle, ek reminder us machine ka jisme hum numbers daal rahe hain, aur do supporting symbols jo hum baar baar use karenge.

Neeche sab kuch yeh hai: $p$ chuno, $q$ chuno, surprise ko reality se weight karo, add up karo. Bar-chart figures dikhate hain ki $p$ (blue) aur $q$ (orange) actually side by side kaisi dikhti hain.

Scenario matrix

Cross-entropy do probability vectors leti hai. "Scenarios" is baat se aate hain ki woh vectors kaisi dikh sakti hain — hard vs soft truth, good vs bad guess, aur woh dangerous edges jahan probability $0$ ya $1$ ho.

#	Cell (case class)	Kya special hai	Example
A	Soft truth, soft guess, close	Dono distributions spread out, model near-correct	Ex 1
B	Soft truth, soft guess, mismatched	Model galat direction mein confident	Ex 2
C	Hard (one-hot) truth, decent guess	True vector jaisa $[1, 0, 0]$ , model reasonable	Ex 3
D	Hard truth, confidently WRONG guess	Huge-penalty wala case	Ex 4
E	Degenerate: $q (x) = 0$ jahan $p (x) > 0$	$ln 0 = - \infty$ , blow-up	Ex 5
F	Degenerate: $p (x) = 0$ term	$0 \cdot ln q$ — kya yeh vanish ho jaata hai?	Ex 5
G	Perfect match $q = p$ (lower bound)	Cross-entropy apne floor $H (p)$ tak pahunchti hai	Ex 6
H	Real-world word problem	Ek story ko $p, q$ mein translate karna	Ex 7
I	Exam twist: label smoothing	Truth deliberately soften ki gayi hai	Ex 8
J	Soft truth, HARD (one-hot) guess	Model ek class pe commit karta hai; finite hai jab tak woh ek $p = 0$ class pe commit na kare	Ex 8b
K	Ek single term ka limiting behaviour	$- ln q$ kaise badhta hai jab $q \to 0$ ya $q \to 1$	Ex 9 (figure)

Ab hum har row sweep karte hain.

Bar chart dekho: blue truth bars aur orange guess bars almost overlap kar rahe hain — yahi visual near-coincidence hai isliye hum expect karte hain ki loss floor se barely upar baithega.

Step 1 — Surprises likhо. $- ln (0.65) = 0.4308$ , $- ln (0.35) = 1.0498$ . Yeh step kyun? Surprise per-outcome hai aur model ki probability $q$ use karta hai — yeh woh "code length" hai jise humne commit kiya tha.

Step 2 — Reality $p$ se weight karo aur sum karo. $H (p, q) = 0.7 (0.4308) + 0.3 (1.0498) = 0.3016 + 0.3149 = 0.6165 nats$ Yeh step kyun? Reality decide karti hai ki har surprise actually kitni baar hoti hai, isliye hum $q$ se nahi, $p$ se weight karte hain.

Verify. True entropy $H (p) = - [0.7 ln 0.7 + 0.3 ln 0.3] = 0.6109$ . Kyunki $H (p, q) \geq H (p)$ hamesha hota hai ( $D_{K L} \geq 0$ rule definition se), aur $0.6165 \geq 0.6109$ ✓. Gap $D_{K L} = 0.0056$ nats bahut chhota hai — "close guess" se match karta hai. Units: nats ✓.

Bar chart mein orange guess ab blue truth ke relative flipped hai — wahan tall hai jahan truth chhoti hai. Yeh crossing mismatched guess ka visual signature hai aur warn karta hai ki loss bada hoga.

Step 1 — Surprises. $- ln (0.2) = 1.6094$ , $- ln (0.8) = 0.2231$ . Yeh step kyun? Same machine; sirf $q$ change hua.

Step 2 — True $p$ se weight karo. $H (p, q) = 0.7 (1.6094) + 0.3 (0.2231) = 1.1266 + 0.0669 = 1.1935 nats$ Yeh step kyun? Barish 70% time hoti hai, aur har rainy day model bahut surprised hota hai — us badi surprise par heavy $0.7$ weight loss ko inflate karta hai.

Verify. $H (p, q) = 1.19 ≫ 0.62$ Ex 1 se ✓. $D_{K L} = 1.1935 - 0.6109 = 0.5826$ nats — bada mismatch cost, jaisa expect tha. Phir bhi $\geq H (p)$ ✓.

Chart mein notice karo: blue truth "cat" par ek single full bar hai (yahi one-hot dikhta hai). Har orange bar jo zero blue bar ke upar baitha hai woh multiply away ho jayega.

Step 1 — Zero-weight terms hatao. $H (p, q) = - [1 \cdot ln 0.7 + 0 \cdot ln 0.2 + 0 \cdot ln 0.1]$ Yeh step kyun? $p (dog) = p (bird) = 0$ , toh woh surprises actually kabhi hoti nahi — unka contribution hai $0 \cdot (kuch finite) = 0$ .

Step 2 — Ek survivor evaluate karo. $H (p, q) = - ln 0.7 = 0.3567 nats$

Verify. One-hot truth ke liye, $H (p) = 0$ (ek known cat ke baare mein koi uncertainty nahi), isliye $D_{K L} = H (p, q) = 0.3567$ — poora loss hi mismatch hai. Sanity: perfect $q (cat) = 1$ deta $- ln 1 = 0$ ✓. Yeh exactly negative-log-likelihood shortcut hai aur MLE se link karta hai.

Step 1 — Sirf true class matter karti hai. $H (p, q) = - ln q (cat) = - ln 0.05$ Yeh step kyun? One-hot truth phir se dog/bird terms ko zero kar deti hai — cross-entropy sirf woh probability padhti hai jo model ne jo actually hua usse di thi.

Step 2 — Evaluate karo. $H (p, q) = - ln 0.05 = 2.9957 nats$

Verify. Ex 3 ( $0.357$ ) vs yahan ( $2.996$ ) compare karo: true-class probability ko $0.7$ se $0.05$ tak drop karne se loss ~8× multiply ho gaya. Confident wrongness ke liye yeh steep penalty hi wajah hai ki hum MSE par cross-entropy prefer karte hain. Note karo ki model ki "dog" mein confidence ( $0.90$ ) kabhi enter nahi hui — cross-entropy galat cheez ke baare mein sure hone ko reward nahi karta ✓.

Sub-case E — $q$ zero jahan $p$ positive hai. Step 1. $H (p, q) = - ln q (cat) = - ln 0$ . Yeh step kyun? True class probability $1$ se hoti hai, lekin model ne swear kiya ki yeh impossible hai. Step 2. $- ln 0 = + \infty$ . Infinite loss. Yeh kyun matter karta hai: ek model jo reality ko rule out karta hai infinitely surprised hota hai. Isliye real code ek tiny floor $ϵ$ add karta hai (e.g. $q \to max (q, 1 0^{- 12})$ ) — ya Label Smoothing use karta hai taaki koi probability exactly $0$ kabhi na ho.

Sub-case F — Ek term ke liye $p$ zero hai. Yahan dog term hai $p (dog) ln q (dog) = 0 \cdot ln (0.2)$ . Kyunki $ln (0.2)$ ek finite number hai ( $- 1.609$ ), product genuinely $0$ hai. Yeh step kyun? Danger $0 \cdot (- \infty)$ tabhi aata hai jab $q$ wahan bhi $0$ ho. Jab tak $q (dog) > 0$ hai, zero weight term ko cleanly delete karta hai. (Convention ke anusaar $0 ln 0 := 0$ , iska limit.)

Verify. E: $lim_{q \to 0^{+}} (- ln q) = + \infty$ ✓. F: $0 \cdot ln 0.2 = 0$ exactly ✓. Toh sirf dangerous zero hai $q \to 0$ true class par.

Step 1 — Plug in karo. $H (p, q) = - [0.6 ln 0.6 + 0.4 ln 0.4]$ Yeh step kyun? Jab $q = p$ hota hai toh code lengths pehle se reality ke liye optimal hain — zero wasted bits.

Step 2 — Evaluate karo. $- ln 0.6 = 0.5108$ , $- ln 0.4 = 0.9163$ . $H (p, q) = 0.6 (0.5108) + 0.4 (0.9163) = 0.3065 + 0.3665 = 0.6730 nats$

Verify. Yeh construction se Shannon Entropy $H (p)$ ke equal hai, isliye $D_{K L} (p ∥ q) = 0$ ✓. Aur $0.673$ ek lower bound hai: koi bhi $q$ isse beat nahi kar sakta ( $D_{K L} \geq 0$ rule phir se). Yeh floor isliye hai ki hum classification loss ko $0$ tak drive nahi kar sakte jab tak truth one-hot na ho ( $H (p) = 0$ ).

Step 1 — Story ko vectors mein translate karo. Truth $p = [0.3, 0.7]$ ; guess $q = [0.5, 0.5]$ . Yahan $N = 2$ (spam / ham). Yeh step kyun? "30% truly spam hain" frequency hai — yeh $p$ hai. "Model hamesha 50-50 kehta hai" $q$ hai.

Step 2 — Surprises equal hain. $- ln 0.5 = 0.6931$ dono classes ke liye. Yeh step kyun? Ek uniform guess har jagah same code length assign karta hai.

Step 3 — Weighted sum. $H (p, q) = 0.3 (0.6931) + 0.7 (0.6931) = 0.6931 nats$ Yeh step kyun? Jab $q$ $N$ outcomes par uniform ho, toh har surprise $- ln (1/ N) = ln N$ hoti hai, aur $p$ -weights sum to $1$ hote hain, isliye $H (p, q) = ln N$ regardless of $p$ (yahan $ln 2$ ).

Verify. $ln 2 = 0.6931$ ✓. Compare karo $H (p) = - [0.3 ln 0.3 + 0.7 ln 0.7] = 0.6109$ . Wasted bits $D_{K L} = 0.0822$ nats — commit karne se inkaar karne ki price. Ek smarter model jo $q = p$ output kare exactly itna save kar leta.

Step 1 — Note karo ki ab har term count karti hai. Yeh step kyun? Yeh exactly parent ka Pitfall 1 hai: soft $p$ ka matlab hai ki hum terms drop nahi kar sakte. Dekho Label Smoothing.

Step 2 — Surprises. $- ln 0.8 = 0.2231$ , $- ln 0.1 = 2.3026$ (do baar). Yeh step kyun? Har surprise "is outcome ka mujhe kitne nats cost honge" model ke $q$ se padha jaata hai — aur kyunki $0.1$ chhota hai, uski surprise ( $2.30$ ) badi hai, confident $0.8$ bar ki surprise se das guna. Bar jitna chhota, surprise utni badi; yeh inverse relationship hi $- ln$ ki poori geometry hai.

Step 3 — Weighted sum. $H (p, q) = 0.9 (0.2231) + 0.05 (2.3026) + 0.05 (2.3026)$ $= 0.2008 + 0.1151 + 0.1151 = 0.4310 nats$ Yeh step kyun? Other classes par do chhote $0.05$ weights ab model ko wahan thodi probability rakhne ke liye nudge karte hain — yahi label smoothing kharidta hai (calibration, kam overconfidence).

Verify. Same $q$ par one-hot loss hota $- ln 0.8 = 0.2231$ . Smoothed loss $0.4310 > 0.2231$ ✓ — extra $0.208$ do side terms se aata hai. Floor check: $H (p)$ for $[0.9, 0.05, 0.05] = 0.3944$ , aur $0.4310 \geq 0.3944$ ✓.

Step 1 — Near-one-hot guess, finite case. Surprises: $- ln 0.98 = 0.0202$ , $- ln 0.01 = 4.6052$ (do baar). Yeh step kyun? Hard-ish guess "galat" surprises huge banata hai ( $4.6$ ), lekin truth unhe sirf $0.05$ each weight karti hai — toh woh contribute karte hain lekin explode nahi karte.

Step 2 — Weighted sum. $H (p, q) = 0.9 (0.0202) + 0.05 (4.6052) + 0.05 (4.6052) = 0.0182 + 0.2303 + 0.2303 = 0.4787 nats$ Yeh step kyun? Kyunki $p$ phir bhi har side class par $0.05$ rakhta hai, over-confidence penalised hoti hai — ek soft truth one-hot guesses se ladti hai. Yahi exactly wajah hai ki label smoothing arrogant outputs discourage karta hai.

Step 3 — Degenerate hard-wrong guess. Agar $q = [0, 1, 0]$ ho, toh cat term hai $p (cat) ln q (cat) = 0.9 \cdot ln 0 = - \infty$ , toh $H (p, q) = + \infty$ . Yeh step kyun? Same rule jaisa Cell E: ek positive truth weight ( $0.9$ ) times $ln 0$ blow up karta hai. Ek one-hot guess tabhi safe hai jab woh exactly aise classes par land kare jis par truth bhi care karti hai.

Verify. Finite case $0.4787 > 0.4310$ Ex 8 se (arrogance mild guess se zyada cost karti hai) ✓, aur $\geq H (p) = 0.3944$ ✓. Degenerate case: $lim_{q \to 0^{+}} 0.9 (- ln q) = + \infty$ ✓.

Yeh figure kaise padhen. Horizontal axis $q$ hai, woh probability jo tumhara model us class ko assign karta hai jo actually hua — yeh $0$ (model ne swear kiya ki impossible tha) se $1$ (model fully certain aur correct tha) tak jaati hai. Vertical axis resulting cross-entropy loss $L = - ln q$ nats mein hai. Blue curve ko left-to-right follow karo: yeh top-left ( $+ \infty$ ) se right edge par $0$ tak plunge karta hai. Teen coloured dots neeche ke checkpoints mark karte hain.

Step 1 — Endpoints padho. $q \to 1^{-}$ : $- ln q \to 0$ (perfectly confident aur correct, koi surprise nahi). $q \to 0^{+}$ : $- ln q \to + \infty$ (confident aur impossibly wrong — Ex 5 ka blow-up). Yeh step kyun? Yeh woh extreme corners hain jinke beech har classifier rehta hai.

Step 2 — Middle checkpoints padho. $q = 0.5 \Rightarrow - ln 0.5 = 0.693$ ; $q = 0.1 \Rightarrow 2.303$ ; $q = 0.9 \Rightarrow 0.105$ . Yeh step kyun? Notice karo ki curve convex hai aur left par steep hai — slope hai $- 1/ q$ , isliye $q = 0.1$ par gradient magnitude $10$ hai, versus $q = 0.9$ par $1.11$ . Yahi "confidently wrong hone par strong gradient" property hai parent ke MSE comparison se.

Verify. Slope $\frac{d}{d q} (- ln q) = - \frac{1}{q}$ : $q = 0.1$ par yeh $- 10$ hai, $q = 0.9$ par $- 1.111$ ✓. Convexity ( $\frac{1}{q ^{2}} > 0$ ) guarantee karta hai $q = 1$ par single minimum ✓. Yeh geometric root hai isliye cross-entropy fast train karta hai — dekho Logistic Regression.

Recall Quick self-check

One-hot truth $[1, 0, 0]$ , model $[0.5, 0.3, 0.2]$ — loss kya hai? ::: $- ln 0.5 = 0.6931$ nats (sirf true-class term bachta hai). Kaun sa single zero cross-entropy ko infinite banata hai? ::: $q = 0$ us class par jahan $p > 0$ ho (model reality ko rule out kar deta hai). Agar $q = p$ ho, toh cross-entropy kiske equal hoti hai? ::: Shannon entropy $H (p)$ — iska minimum, $D_{K L} = 0$ ke saath. Uniform guess $ln N$ kyun deta hai regardless of $p$ ? ::: Har surprise equal hoti hai $- ln (1/ N) = ln N$ , aur $p$ -weights sum to $1$ karte hain.

Related deep concepts: KL Divergence, Categorical Cross-Entropy, Mutual Information.