Cross-entropy concept — Hinglish notes

1.3.19 · AI-ML › Probability & Statistics

Overview

Cross-entropy do probability distributions ke beech ka difference measure karta hai — yeh quantify karta hai ki average mein kitne bits chahiye honge ek distribution ke samples encode karne ke liye agar hum galat distribution assume kar lein. Yeh machine learning mein classification ke liye workhorse loss function hai kyunki yeh directly confident galat predictions ko penalize karta hai.

Yeh kyun zaroori hai: Cross-entropy information theory aur machine learning ko bridge karta hai. Yeh batata hai ki "hum kitne surprised hain" jab reality hamari model ki beliefs se match nahi karti.

[!intuition] Core Idea

Imagine karo tum ek weather forecaster ho. Har roz tum rain probability predict karte ho, aur nature sach reveal karta hai.

Agar tum kaho 90% rain aur rain ho → chhoti surprise (tum confident the aur sahi the)
Agar tum kaho 10% rain aur rain ho → bahut badi surprise (tum confident the aur galat the)

Cross-entropy tumhari predictions ki average surprise hai, weighted by jo actually hota hai. Jitna better tumhari predictions reality se match karengi, utna lower cross-entropy hogi.

Key insight: Cross-entropy = "Kitne extra bits waste kar raha hoon mein galat probability distribution use karke?"

[!definition] Mathematical Definition

Diya gaya hai:

$p (x)$ = true probability distribution (reality, ground truth)
$q (x)$ = predicted probability distribution (model ki belief)

$p$ se $q$ ki taraf cross-entropy hai:

$H (p, q) = - \sum_{x} p (x) lo g q (x)$

Continuous distributions ke liye:

$H (p, q) = - \int p (x) lo g q (x) d x$

Units: bits (agar $lo g = lo g_{2}$ ) ya nats (agar $lo g = ln$ ). ML mein hum natural log use karte hain.

[!formula] First Principles se Derivation

Step 1: Shannon Entropy (Self-Information)

Question: Jab event $x$ probability $p (x)$ ke saath occur hota hai, toh mujhe kitni surprise hogi?

Answer: Surprise (self-information) hai:

$I (x) = - lo g p (x)$

Yeh formula kyun?

Rare events ( $p (x)$ chhota) → badi surprise
Certain events ( $p (x) = 1$ ) → zero surprise
Independent events ki surprise additive honi chahiye: $I (x, y) = I (x) + I (y)$ , matlab humein $lo g$ chahiye

Expected surprise jab $p (x)$ se sampling ho:

$H (p) = - \sum_{x} p (x) lo g p (x)$

Yeh $p$ ki Shannon entropy hai — minimum average bits jo chahiye $p$ ke samples encode karne ke liye agar hum $p$ perfectly jaante hain.

Step 2: Agar Hum Galat Code Use Karein Toh?

Maan lo reality $p (x)$ follow karti hai lekin hum apni encoding design karte hain distribution $q (x)$ assume karke.

Hum event $x$ ko code length $- lo g q (x)$ assign karte hain (optimal for $q$ )
Lekin events frequency $p (x)$ ke saath occur hote hain
Average code length = $- \sum_{x} p (x) lo g q (x)$

Yeh hai cross-entropy $H (p, q)$ — average bits jo hum actually use karte hain galat distribution ke saath.

Yeh step kyun? Cross-entropy batata hai galat hone ki cost kya hai. Agar $q = p$ , toh hum minimum possible bits use karte hain. $p$ se koi bhi deviation bits waste karta hai.

Step 3: KL Divergence se Connection

Extra bits waste hote hain $q$ use karke $p$ ki jagah:

$D_{K L} (p ∥ q) = H (p, q) - H (p) = - \sum_{x} p (x) lo g \frac{q ( x )}{p ( x )}$

Yeh hai Kullback-Leibler divergence.

Key relationship:

$H (p, q) = H (p) + D_{K L} (p ∥ q)$

$H (p)$ = irreducible uncertainty (truth ki entropy)
$D_{K L} (p ∥ q)$ = model mismatch ki extra cost

ML mein cross-entropy minimize kyun karein? Kyunki $H (p)$ fixed hai (hum reality nahi badal sakte), $H (p, q)$ minimize karna equivalent hai $D_{K L} (p ∥ q)$ minimize karne ke — matlab apni model ko reality se match karana.

[!example] Example 1: Binary Classification (Coin Flip)

Setup: True distribution: coin heads $p = 0.7$ ke saath aata hai. Model predict karta hai $q = 0.8$ .

Truth: $p (heads) = 0.7$ , $p (tails) = 0.3$

Model: $q (heads) = 0.8$ , $q (tails) = 0.2$

Cross-entropy:

$H (p, q) = - [0.7 lo g (0.8) + 0.3 lo g (0.2)]$

Yeh step kyun? Hum har outcome ko uski true probability $p$ se weight karte hain, model ki belief $q$ se nahi. Reality decide karti hai ki har surprise kitni baar occur hogi.

$H (p, q) = - [0.7 \times (- 0.223) + 0.3 \times (- 1.609)]$ $= - [- 0.156 - 0.483] = 0.639 nats$

Truth ki entropy se compare karo:

$H (p) = - [0.7 lo g (0.7) + 0.3 lo g (0.3)] = 0.611 nats$

Wasted bits: $D_{K L} = 0.639 - 0.611 = 0.028$ nats.

Interpretation: Hamari model heads mein thodi overconfident hai, jisse chhoti information loss ho rahi hai.

[!example] Example 2: Multiclass Classification (3 classes)

Setup: Image classification — cat, dog, bird.

Truth (one-hot): $p = [1, 0, 0]$ (yeh definitely cat hai)

Model predict karta hai: $q = [0.7, 0.2, 0.1]$

Cross-entropy:

$H (p, q) = - [1 \times lo g (0.7) + 0 \times lo g (0.2) + 0 \times lo g (0.1)]$

Baaki terms zero kyun hain? Kyunki $p (dog) = p (bird) = 0$ — yeh classes is sample mein occur nahi karte. Hum sirf us probability ki care karte hain jo actually hua usse assign ki gayi hai.

$H (p, q) = - lo g (0.7) = 0.357 nats$

Agar model predict kare $q = [0.1, 0.8, 0.1]$ (galat class)?

$H (p, q) = - lo g (0.1) = 2.303 nats$

Itni badi jump kyun? Cross-entropy confident galat predictions ko heavily penalize karta hai. Model 80% sure tha ki yeh dog hai, lekin yeh cat tha — massive surprise.

[!example] Example 3: MSE Kyun Nahi?

Question: Classification ke liye MSE ki jagah cross-entropy kyun use karein?

Setup: Binary classification, true label $y = 1$ (positive class).

Model output: $\overset{y}{^} = 0.1$ (negative predict karta hai).

MSE: $L_{M S E} = (1 - 0.1)^{2} = 0.81$

Cross-entropy: $L_{C E} = - lo g (0.1) = 2.303$

Yeh difference kyun matter karta hai:

Agar model improve ho $\overset{y}{^} = 0.2$ tak:

MSE: $(1 - 0.2)^{2} = 0.64$ → decrease of $0.17$
Cross-entropy: $- lo g (0.2) = 1.609$ → decrease of $0.694$

Cross-entropy better kyun hai: Yeh stronger gradient signal deta hai jab model confidently galat ho. MSE gradient error mein linear hai; cross-entropy gradient predicted probability ke inversely proportional hai, toh galat predictions zyada jaldi correct hoti hain.

[!mistake] Common Pitfalls

Mistake 1: "Cross-entropy bas negative log probability hai"

Kyun sahi lagta hai: Practice mein hum likhte hain loss = -log(p_correct_class), jo lagta hai hum sum ignore kar rahe hain.

Kyun yeh incomplete hai: Yeh tabhi true hai jab true distribution one-hot ho (hard labels). Poori definition $p (x)$ se weight karke sab outcomes ke upar sum karti hai.

Fix:

One-hot $p$ → $H (p, q) = - lo g q (y_{true})$ (baaki sab terms vanish ho jaate hain)
Soft labels $p$ → $H (p, q) = - \sum_{i} p_{i} lo g q_{i}$ (sab terms matter karte hain)

Kab matter karta hai: Label smoothing, knowledge distillation, multi-label classification.

Mistake 2: "Cross-entropy minimize karna = entropy minimize karna"

Kyun sahi lagta hai: Dono mein "entropy" naam hai aur dono $- lo g$ use karte hain.

Kyun galat hai:

$H (p)$ = true distribution ki entropy (fixed, model par depend nahi karta)
$H (p, q)$ = cross-entropy (model ke $q$ par depend karta hai)

Fix: Tum $H (p, q)$ minimize karte ho $q$ ko $p$ se match karake. Kyunki $H (p, q) = H (p) + D_{K L} (p ∥ q)$ aur $H (p)$ constant hai, tum actually $D_{K L} (p ∥ q)$ minimize kar rahe ho.

Mistake 3: "Cross-entropy symmetric hai"

Kyun sahi lagta hai: Formula mein dono $p$ aur $q$ appear karte hain.

Kyun galat hai: $H (p, q) \neq = H (q, p)$ generally.

Proof: $H (p, q) = - \sum p (x) lo g q (x)$ $H (q, p) = - \sum q (x) lo g p (x)$

Alag weights, alag sums.

Fix: Cross-entropy measure karta hai "surprise jab reality $p$ hai lekin hum $q$ assume karte hain". Inhe swap karna ek alag question poochta hai. Hamesha $p$ ko ground truth rakho, $q$ ko model.

[!recall]- Feynman Explanation (ELI12)

Imagine karo tum ek guessing game khel rahe ho. Har round mein, mein secretly ek colored ball ek bag se pick karta hoon (maybe 70% red, 30% blue). Tumhe color guess karna hai pehle ki mein dikhaaun.

Ab, tumhe points milte hain based on kitne confident the tum:

Agar tum kaho "90% sure hai ki red hai" aur woh red IS → bahut points milte hain (tum sahi the aur confident bhi!)
Agar tum kaho "90% sure hai ki red hai" aur woh BLUE hai → TONS of points jaate hain (tum galat the aur super confident bhi)

Cross-entropy kaafi rounds mein tumhare average point loss track karne jaisa hai. Agar tumhare guesses true bag contents (70% red) se match karein, toh bahut kam points jaate hain. Agar tum hamesha 50-50 guess karo jab bag actually 70-30 ho, toh zyada points jaate hain kyunki tum zyada surprised hote ho.

Machine learning mein, model tum ho guessing karte hue, aur reality bag hai. Hum model ko train karte hain cross-entropy minimize karne ke liye = average surprise minimize karo = better guesses karo jo reality se match karein!

[!mnemonic] Memory Aid

CROSS = Compare Reality's Occurrence to System's Speculation

Compare: Do distributions
Reality: True $p (x)$ (ground truth)
Occurrence: True probabilities se weighted
System: Model ka predicted $q (x)$
Surprise: $- lo g q (x)$ (information content)

Formula mnemonic: "People Love Quality" → $p (x) lo g q (x)$ negative sign ke saath sum kiya gaya.

Connections

Shannon Entropy — cross-entropy self-entropy ko generalize karta hai jab $p \neq = q$
KL Divergence — $H (p, q) = H (p) + D_{K L} (p ∥ q)$
Softmax Function — logits ko probabilities $q$ mein convert karta hai cross-entropy loss ke liye
Logistic Regression — binary cross-entropy iska loss function hai
Categorical Cross-Entropy — neural networks mein use hone wala multiclass extension
Label Smoothing — regularization technique jo $p$ ko modify karti hai overconfidence rokne ke liye
Maximum Likelihood Estimation — cross-entropy minimize karna = likelihood maximize karna
Mutual Information — variables ke beech shared information ka related measure

Flashcards

Cross-entropy kya measure karta hai?

Distribution

p

ke samples encode karne ke liye average bits ki number, ek code use karke jo distribution

q

ke liye optimize hai; equivalently, "surprise" jab reality

p

hai lekin hum

q

assume karte hain.

Cross-entropy

H (p, q)

ka formula?

H (p, q) = - \sum_{x} p (x) lo g q (x)

(discrete) ya

- \int p (x) lo g q (x) d x

(continuous).

Cross-entropy mein

- lo g

kyun use karte hain?

Kyunki surprise (self-information) probability ke inversely related hai: rare events zyada surprising hote hain. Log independent events ke liye additive property ensure karta hai.

Cross-entropy aur KL divergence ka relationship?

H (p, q) = H (p) + D_{K L} (p ∥ q)

, jahan

H (p)

true distribution ki entropy hai aur

D_{K L}

extra bits hain jo waste hote hain

p

ki jagah

q

use karke.

Classification mein cross-entropy minimize kyun karein?

Kyunki

H (p)

(true distribution ki entropy) fixed hai,

H (p, q)

minimize karna equivalent hai

D_{K L} (p ∥ q)

minimize karne ke, matlab model ko reality se match karana.

One-hot true label

y

aur predicted probabilities

q

ke liye cross-entropy?

H (p, q) = - lo g q_{y}

(sirf true class ki predicted probability matter karti hai; baaki terms vanish ho jaate hain kyunki

p_{i} = 0

for

i \neq = y

Classification ke liye cross-entropy MSE se better kyun hai?

Cross-entropy mein confidently galat hone par stronger gradients hote hain (gradient

\propto 1/ q

vs. MSE ka linear gradient), jisse mistakes zyada jaldi correct hoti hain.

Kya cross-entropy symmetric hai?

Nahi.

H (p, q) \neq = H (q, p)

generally, kyunki yeh alag distributions (

p

vs.

q

) se weight karte hain. Hamesha

p

= ground truth,

q

= model rakho.

Cross-entropy ka kya hoga jab

q

approach kare

p

ko?

H (p, q) \to H (p)

(true distribution ki entropy), aur

D_{K L} (p ∥ q) \to 0

. Yeh minimum achievable cross-entropy hai.

Cross-entropy explode kyun karta hai jab

q (x) \to 0

x

ke liye jahan

p (x) > 0

ho?

Kyunki

- lo g (q (x)) \to \infty

jab

q (x) \to 0

. Model near-zero probability assign karta hai kisi cheez ko jo actually happen karti hai — infinite surprise.