Entropy and KL divergence: Question bank — Hinglish deep dive

1.3.18 · D5 · AI-ML › Probability & Statistics › Entropy and KL divergence

Yeh Entropy and KL divergence ke liye ek rapid-fire misconception hunter hai. Har line ek Question ::: Answer reveal hai — question padho, apna jawab ZOOR SE bolo, phir reveal karo. Agar tum jawab ko ek sentence mein justify nahi kar sakte, toh abhi concept tumhara nahi hua. Yahaan koi bhari arithmetic nahi hai (woh D3/D4 mein hai) — yeh page ideas aur edges ko target karta hai.

True or false — justify karo

Entropy ek discrete distribution ke liye negative ho sakti hai.

False. Har term

- p (x) lo g p (x)

ka value

\geq 0

hota hai kyunki

0 \leq p (x) \leq 1

lo g p (x) \leq 0

hota hai; non-negative values ka sum

H (X) \geq 0

deta hai.

n

outcomes wale variable ki maximum possible entropy

lo g_{2} n

bits hoti hai.

True. Gibbs' inequality ke hisaab se uniform distribution entropy ko maximise karti hai, jo

H = lo g_{2} n

deta hai; koi bhi bias isse kam kar deta hai.

Ek naaya outcome add karna jis ki probability exactly

0

ho, entropy ko change karta hai.

False. Convention

0 lo g 0 = 0

ke zariye woh outcome kuch contribute nahi karta, isliye entropy unchanged rehti hai.

D_{K L} (p ∥ q) = D_{K L} (q ∥ p)

sabhi distributions ke liye.

False. KL asymmetric hai; dono directions alag sawaal poochte hain ("

q

use karne ki cost jab truth

p

hai" vs. uska ulta) aur generally alag numbers dete hain.

D_{K L} (p ∥ q)

negative ho sakta hai agar

q

bahut bura model ho.

False. Gibbs'/Jensen's inequality ensure karti hai ki

D_{K L} \geq 0

hamesha; bura

q

ise bada banata hai, kabhi negative nahi.

Cross-entropy

H (p, q)

hamesha entropy

H (p)

se kam se kam utni badi hoti hai.

True. Kyunki

H (p, q) = H (p) + D_{K L} (p ∥ q)

aur

D_{K L} \geq 0

, cross-entropy kabhi optimal code length

H (p)

se neeche nahi jaati.

Ek deterministic variable (

p = 1

ek outcome par) ki entropy

0

hoti hai.

True. Koi surprise nahi hai:

- 1 \cdot lo g 1 = 0

, isliye jab tum ise observe karte ho toh zero information milti hai.

Agar

D_{K L} (p ∥ q) = 0

toh

p

aur

q

identical hone chahiye (jahaan

p > 0

ho).

True. Gibbs' inequality mein equality tab hi hoti hai jab

p (x) = q (x)

har us outcome ke liye ho jis ki positive probability ho.

Nats mein maapi gayi entropy aur bits mein maapi gayi entropy alag amounts of uncertainty describe karti hain.

False. Woh same uncertainty ko alag units mein measure karti hain; nats

=

bits

\times ln 2

, bas log base ka change hai.

Error dhundho

"Kyunki KL difference measure karta hai,

D_{K L} (p ∥ r) \leq D_{K L} (p ∥ q) + D_{K L} (q ∥ r)

Galat — KL ek metric nahi hai aur triangle inequality violate karta hai; agar tumhe ek symmetric, triangle-respecting quantity chahiye toh Jensen-Shannon Divergence use karo.

H (X) = - \sum p (x) lo g p (x)

, isliye agar koi

p (x) = 0

ho toh entropy undefined hai."

Galat — limit

lim_{p \to 0^{+}} p lo g p = 0

ise cleanly patch kar deta hai; ek kabhi na hone wala event zero surprise add karta hai.

"Classification loss minimise karne ke liye hum labels ki entropy

H (p)

minimise karte hain."

Galat — hum model

q

par cross-entropy

H (p, q)

minimise karte hain;

H (p)

data se fixed hai aur ise change nahi kiya ja sakta. Dekho Cross-Entropy Loss.

"Cross-entropy minimise karna KL divergence minimise karne se unrelated hai."

Galat —

H (p, q) = H (p) + D_{K L} (p ∥ q)

aur

H (p)

q

mein constant hai, isliye ek ko minimise karna exactly doosre ko bhi minimise karta hai.

"Agar

p (x) > 0

lekin

q (x) = 0

, toh KL bas ek bada finite number hai."

Galat — term

p (x) lo g \frac{p ( x )}{0}

diverge karta hai, isliye

D_{K L} (p ∥ q) = \infty

;

q

ko har woh outcome "cover" karna chahiye jo

p

produce kar sakta hai.

"Entropy distribution

p

ki ek convex function hai."

Galat — entropy

p

mein concave hai; do distributions ka average entropy kabhi kam nahi karta. (KL, iske contrast mein, pair

(p, q)

mein convex hai.)

"Ek one-hot label ki cross-entropy sabhi class probabilities use karti hai."

Galat — jab

p

one-hot hoti hai toh sirf true class bachti hai, isliye

H (p, q) = - lo g q (true class)

; baaki predictions drop out ho jaati hain.

Why wale sawaal

Surprisal

I (x) = - lo g p (x)

mein

1/ p (x)

ki jagah

lo g

kyun use karte hain?

Log independent events ki information ko add karta hai:

I (x, y) = I (x) + I (y)

, jo hamari intuition se match karta hai ki do independent surprises stack hote hain, jo

1/ p

karna fail ho jaata hai.

Uniform distribution — koi peaked distribution nahi — entropy kyun maximise karti hai?

Uniform probability ko utni hi evenly spread karti hai jitni ho sake, isliye koi bhi outcome predictable nahi hota; maximum unpredictability ka matlab hai maximum average surprise. Yahi Maximum Entropy Principle ki neenv hai.

D_{K L} (p ∥ q)

ko "extra bits wasted" kyun interpret karte hain?

Yeh cross-entropy minus entropy ke barabar hai,

H (p, q) - H (p)

—

q

ke (galat) code se coding aur optimal

p

-code ke beech ka fark, yaani galat hone ki inefficiency.

Practice mein KL ki direction kyun matter karti hai (jaise Variational Autoencoders mein)?

D_{K L} (q ∥ p)

(reverse) "mode-seeking" hai — yeh

q

ko un regions ko ignore karne deta hai jahaan

p

bada hai, jabki

D_{K L} (p ∥ q)

(forward) "mean-covering" hai aur

p

ke kisi bhi mass ko miss karne par punish karta hai. Alag directions fitted

q

ko alag shape dete hain.

KL ko optimisation ke liye loss ki tarah kyun use kar sakte hain even though yeh distance nahi hai?

Humein bas ek non-negative quantity chahiye jo

0

ho jab distributions match karein aur shrink ho jab woh approach karein — KL yeh sab karta hai; triangle inequality gradient descent ke liye irrelevant hai.

Zyada possible outcomes add karne se maximum achievable entropy kyun tend to badhti hai?

Ceiling

lo g_{2} n

, outcomes ki sankhya

n

ke saath badhti hai; zyada distinguishable possibilities ka matlab hai zyada potential uncertainty jo resolve ho sakti hai.

Edge cases

Jab ek outcome ki probability

1

ho toh

H (X)

kya hota hai?

Exactly

0

bits — ek certain variable koi information carry nahi karta, entropy ki degenerate lower bound.

Jab

p = q

exactly ho toh

D_{K L} (p ∥ q)

kya hota hai?

Exactly

0

— har ratio

p (x) / q (x) = 1

hota hai aur

lo g 1 = 0

; koi wasted coding cost nahi hai.

Agar

q

kisi aisi outcome ko zero probability assign kare jo

p

produce kar sakta hai, toh

D_{K L} (p ∥ q)

ka kya hota hai?

Yeh

+ \infty

ho jaata hai — ek infinitely bura model jo kuch aisa rule out karta hai jo actually hota hai.

p (x) = 0

wale outcome ka entropy contribution kya hota hai?

Zero,

0 lo g 0 = 0

convention ke zariye; yeh simply sum mein participate nahi karta.

Jab coin ka bias

p (H)

0.5

0

1

ki taraf jaata hai, toh uski entropy ka kya hota hai?

Yeh monotonically maximum

1

bit se

0

ki taraf girती hai — zyada predictability, kam surprise; curve symmetric hai aur

p = 0.5

par peak karti hai.

Agar model

q

ek classification task mein true class ko

0

probability assign kare toh

H (p, q)

kya hota hai?

Yeh

+ \infty

tak blow up ho jaata hai (

- lo g 0

se) — isliye practical implementations predictions ko exact

0

se door clip ya smooth karti hain.

Ek continuous variable ke liye, kya "differential entropy" negative ho sakti hai?

Haan — differential entropy sum ko ek density par integral se replace kar deta hai aur negative ho sakta hai (jaise ek narrow Gaussian), discrete entropy ke unlike jo hamesha

\geq 0

hoti hai.

Recall Quick self-test

Kaun sa KL direction "mode-seeking" behaviour deta hai? ::: Reverse form $D_{K L} (q ∥ p)$ , jo variational inference mein use hota hai. Entropy $p$ mein concave hai ya convex? ::: Concave. $D_{K L} (p ∥ q) = \infty$ kab hota hai? ::: Jab kisi outcome mein $p > 0$ lekin $q = 0$ ho.

Related: Mutual Information · Information Gain · Evidence Lower Bound (ELBO) · F-divergences · Cross-Entropy Loss