Cross-entropy concept: Exercises — Hinglish deep dive

1.3.19 · D4 · AI-ML › Probability & Statistics › Cross-entropy concept

Yeh parent Cross-Entropy note ke liye ek self-testing worksheet hai. Har problem apna level batati hai (L1 Recognition → L5 Mastery). Pehle khud try karo, phir collapsible solution kholna. Poore worksheet mein natural log ( $ln$ , units = nats) use karo jab tak problem "bits" ( $lo g_{2}$ ) na kahe.

Shuru karne se pehle, ek reminder un sabhi symbols ka jo hum use karenge, saral shabdon mein:

Poore page ke liye dimag mein rakhne wali picture — surprise as a curve:

Dhyan do: jaise predicted probability $q$ $0$ ki taraf slide hoti hai, surprise $- lo g q$ infinity ki taraf shoot karti hai. Yeh akela curve explain karta hai ki confident-wrong predictions ko itni buri tarah kyun punish kiya jaata hai — yahi poore worksheet ki theme hai.

Level 1 — Recognition

Exercise 1.1

$p$ relative to $q$ ki cross-entropy kaun sa formula hai? (a) $- \sum_{x} q (x) lo g p (x)$ (b) $- \sum_{x} p (x) lo g q (x)$ (c) $\sum_{x} p (x) lo g p (x)$ (d) $\sum_{x} (p (x) - q (x))^{2}$

Recall Solution

(b). Left to right padho: "reality $p$ se weight karo, guess $q$ ki surprise lo." Option (a) roles swap karta hai (woh $H (q, p)$ hoga, ek alag number). Option (c) $- H (p)$ hai (galat sign, aur koi $q$ nahi). Option (d) mean squared error hai, cross-entropy nahi.

Exercise 1.2

True ya false: ordinary probability distributions ke liye cross-entropy negative ho sakti hai (jab sabhi $p (x), q (x) \in [0, 1]$ ).

Recall Solution

False. Har $q (x) \leq 1$ hai, isliye $lo g q (x) \leq 0$ , isliye $- lo g q (x) \geq 0$ . Non-negative surprises ka sum jo non-negative $p (x)$ se weighted hai, woh $\geq 0$ hoga. (Discrete distributions ke liye yeh sirf $0$ tab pohonch sakta hai jab $q$ probability $1$ us ek outcome par daalta ho jiske baare mein $p$ certain hai.)

Level 2 — Application

Exercise 2.1

Ek biased coin heads par true probability $p = 0.6$ ke saath girti hai. Tumhara model $q = 0.6$ guess karta hai (perfect). $H (p, q)$ nats mein compute karo aur confirm karo ki yeh $H (p)$ ke barabar hai.

Recall Solution

Do outcomes: heads $p = 0.6$ , tails $0.4$ . $H (p, q) = - [0.6 ln 0.6 + 0.4 ln 0.4] = - [0.6 (- 0.5108) + 0.4 (- 0.9163)]$ $= - [- 0.3065 - 0.3665] = 0.6730 nats.$ Kyunki $q = p$ hai, "log ke andar" aur "bahar" ke numbers identical hain, isliye yeh $H (p)$ hi hai. Waste $D_{K L} = 0$ . Yeh floor hai: koi bhi guess perfect se behtar nahi ho sakta.

Exercise 2.2

Wahi true coin ( $p = 0.6$ heads), lekin ab model $q = 0.9$ heads guess karta hai. $H (p, q)$ aur wasted bits $D_{K L} (p ∥ q)$ compute karo.

Recall Solution

$H (p, q) = - [0.6 ln 0.9 + 0.4 ln 0.1] = - [0.6 (- 0.10536) + 0.4 (- 2.30259)]$ $= - [- 0.06322 - 0.92103] = 0.98425 nats.$ Waste: $D_{K L} = H (p, q) - H (p) = 0.98425 - 0.67301 = 0.31124$ nats. Isko padhna: model heads ke baare mein over-confident hai. Jab tails aata hai (40% time) toh woh huge $- ln 0.1 = 2.30$ surprise pay karta hai, jo average ko floor se bahut upar kheench deta hai.

Exercise 2.3

Teen-class problem, one-hot truth $p = [0, 1, 0]$ (answer class 2 hai). Model predict karta hai $q = [0.2, 0.5, 0.3]$ . Loss compute karo.

Recall Solution

One-hot truth har term ko zero kar deta hai sirf true class ko chodkar: $H (p, q) = - [0 \cdot ln 0.2 + 1 \cdot ln 0.5 + 0 \cdot ln 0.3] = - ln 0.5 = 0.6931 nats.$ Sirf woh probability matter karti hai jo model ne correct class ko di. Dekho Categorical Cross-Entropy.

Level 3 — Analysis

Exercise 3.1

Soft (smoothed) labels. Truth one-hot nahi hai: $p = [0.1, 0.8, 0.1]$ (ek label-smoothed cat, dekho Label Smoothing). Model predict karta hai $q = [0.1, 0.8, 0.1]$ (exactly match). $H (p, q)$ aur $D_{K L} (p ∥ q)$ compute karo.

Recall Solution

$H (p, q) = - [0.1 ln 0.1 + 0.8 ln 0.8 + 0.1 ln 0.1]$ $= - [0.1 (- 2.30259) + 0.8 (- 0.22314) + 0.1 (- 2.30259)] = - [- 0.23026 - 0.17851 - 0.23026] = 0.63903 nats.$ Kyunki $q = p$ hai, $D_{K L} = 0$ . Key lesson: one-hot case ke unlike, yahan loss floor $0$ nahi hai — ek perfect model bhi $H (p) = 0.639$ nats pay karta hai kyunki labels khud uncertain hain. Yahi wajah hai ki label smoothing minimum achievable loss ko change karta hai.

Exercise 3.2

Numerically dikhao ki cross-entropy symmetric nahi hai: $p = [0.7, 0.3]$ aur $q = [0.4, 0.6]$ ke saath, dono $H (p, q)$ aur $H (q, p)$ compute karo aur confirm karo ki woh alag hain.

Recall Solution

$H (p, q) = - [0.7 ln 0.4 + 0.3 ln 0.6] = - [0.7 (- 0.91629) + 0.3 (- 0.51083)] = 0.64140 + 0.15325 = 0.79465.$ Ruko — sign dhyan se: $- 0.7 (- 0.91629) = + 0.64140$ , $- 0.3 (- 0.51083) = + 0.15325$ , sum $= 0.79465$ nats. $H (q, p) = - [0.4 ln 0.7 + 0.6 ln 0.3] = - [0.4 (- 0.35667) + 0.6 (- 1.20397)] = 0.14267 + 0.72238 = 0.86505 nats.$ $0.79465 \neq = 0.86505$ , isliye $H (p, q) \neq = H (q, p)$ . Dono sawaal "surprised jab reality $p$ ho lekin maine $q$ assume kiya" versus uska swap genuinely alag hain.

Exercise 3.3

$H (p, q) = H (p) + D_{K L} (p ∥ q)$ aur fact $D_{K L} \geq 0$ use karte hue, prove karo ki fixed truth $p$ ke liye sabse chhoti possible cross-entropy $H (p)$ hai, jo sirf tab achieve hoti hai jab $q = p$ .

Recall Solution

$H (p)$ mein $q$ nahi hai, isliye jab hum model vary karte hain toh yeh ek constant hai. Tab $H (p, q) = constant H (p) + \geq 0 D_{K L} (p ∥ q) .$ Ek constant plus ek non-negative quantity tab minimize hoti hai jab non-negative part apne minimum $0$ par ho. Aur $D_{K L} (p ∥ q) = 0$ iff $q = p$ (Gibbs' inequality). Isliye $min_{q} H (p, q) = H (p)$ at $q = p$ . Yahi wajah hai ki cross-entropy minimize karna model ko reality ki taraf kheenchta hai (equivalently, Maximum Likelihood Estimation ki taraf).

Level 4 — Synthesis

Exercise 4.1

Ek logistic-regression model (dekho Logistic Regression) positive class ke liye $\overset{y}{^} = σ (z)$ output karta hai, aur true label $y = 1$ hai. Per-sample loss hai $L = - ln \overset{y}{^}$ . $\frac{d L}{d z}$ derive karo aur dikhao ki yeh $\overset{y}{^} - y$ ke barabar hai, jahan $σ (z) = \frac{1}{1 + e ^{- z}}$ aur $σ^{'} (z) = σ (z) (1 - σ (z))$ .

Recall Solution

Chain rule, teen clearly dikhne wale steps mein. Pehla: $L = - ln \overset{y}{^}$ , isliye $\frac{d L}{d y ^} = - \frac{1}{y ^}$ . Doosra: $\frac{d y ^}{d z} = σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ . Combine: $\frac{d L}{d z} = - \frac{1}{y ^} \cdot \overset{y}{^} (1 - \overset{y}{^}) = - (1 - \overset{y}{^}) = \overset{y}{^} - 1 = \overset{y}{^} - y .$ Kyun yeh beautiful hai: log se aane wala messy $1/ \overset{y}{^}$ sigmoid derivative ke $\overset{y}{^}$ ko exactly cancel kar deta hai, aur clean prediction minus target bachi rahti hai. Woh cancellation hi asli wajah hai ki cross-entropy sigmoid/softmax ke saath itni naturally pair kyun hoti hai — jab model confidently galat ho tab koi vanishing gradient nahi.

Exercise 4.2

Gradient signal contrast karo. Maano true $y = 1$ lekin $\overset{y}{^} = 0.01$ (confidently galat). Cross-entropy gradient magnitude $∣ \overset{y}{^} - y ∣$ compute karo aur ise MSE gradient magnitude se compare karo, jahan loss $L_{M S E} = (y - \overset{y}{^})^{2}$ same sigmoid ke saath hai, jo hai $∣2 (\overset{y}{^} - y) \overset{y}{^} (1 - \overset{y}{^}) ∣$ .

Recall Solution

Cross-entropy: $∣ \overset{y}{^} - y ∣ = ∣0.01 - 1∣ = 0.99$ . Ek strong push. MSE: $∣2 (0.01 - 1) (0.01) (0.99) ∣ = ∣2 (- 0.99) (0.0099) ∣ = 0.0196$ . MSE gradient roughly $0.02$ hai, lagbhag $50 \times$ kamzor, kyunki $\overset{y}{^} (1 - \overset{y}{^})$ factor exactly tab $0$ ki taraf collapse karta hai jab model saturate hota hai. Conclusion: MSE confident mistakes par stall karta hai; cross-entropy push karta rehta hai. Yeh parent note ke Example 3 ka quantitative version hai.

Level 5 — Mastery

Exercise 5.1

Full loss decompose karo. Truth $p = [0.2, 0.8]$ (soft label), model $q = [0.5, 0.5]$ (maximally unsure). $H (p, q)$ , $H (p)$ , aur $D_{K L} (p ∥ q)$ compute karo, aur identity $H (p, q) = H (p) + D_{K L}$ verify karo.

Recall Solution

$H (p, q) = - [0.2 ln 0.5 + 0.8 ln 0.5] = - ln 0.5 = 0.69315 nats.$ (Dono classes ko $q = 0.5$ milta hai, isliye loss sirf $- ln 0.5$ hai chahe $p$ kuch bhi ho.) $H (p) = - [0.2 ln 0.2 + 0.8 ln 0.8] = - [0.2 (- 1.60944) + 0.8 (- 0.22314)] = 0.32189 + 0.17851 = 0.50040 nats.$ $D_{K L} = 0.69315 - 0.50040 = 0.19275 nats.$ Identity check karo: $H (p) + D_{K L} = 0.50040 + 0.19275 = 0.69315 = H (p, q) .$ ✓ Interpretation: model class 2 ki taraf commit karne se inkaar karke $0.193$ nats waste karta hai, upar se $0.500$ nats irreducible label uncertainty ke.

Exercise 5.2

Optimal constant predictor. Ek bade dataset par empirical class frequencies $p = [0.25, 0.75]$ hain. Agar ek model ko har sample ke liye same $q = [q_{1}, 1 - q_{1}]$ output karne pe majboor kiya jaaye, toh average cross-entropy minimize karne wala $q_{1}$ kaun sa hai, aur minimum value kya hai?

Recall Solution

Exercise 3.3 se, average cross-entropy tab minimize hoti hai jab $q = p$ ho. Isliye $q_{1} = 0.25$ . Minimum loss $= H (p)$ : $H (p) = - [0.25 ln 0.25 + 0.75 ln 0.75] = - [0.25 (- 1.38629) + 0.75 (- 0.28768)]$ $= 0.34657 + 0.21576 = 0.56233 nats.$ Kyun yeh mastery insight hai: data par trained cross-entropy model ki output distribution ko empirical class frequencies ki taraf drive karti hai — yahi maximum-likelihood answer hai. Koi bhi $q_{1} \neq = 0.25$ ek positive $D_{K L}$ add karta hai.

Exercise 5.3

Bits vs nats. Exercise 5.2 ke answer ko bits mein convert karo.

Recall Solution

$H (p)_{bits} = \frac{0.56233}{l n 2} = \frac{0.56233}{0.69315} = 0.81128 bits.$ Same information content, alag ruler. Nats base $e$ use karte hain; bits base $2$ use karte hain; nats ko bits mein convert karne ke liye $ln 2$ se divide karo.

[!recall]- One-line self-check

Soft labels par ek perfect model ki cross-entropy barabar hoti hai

labels ki Shannon entropy

H (p)

ke, zero nahi.

- ln \overset{y}{^}

ka gradient ek sigmoid ke through logit ke w.r.t. hai

\overset{y}{^} - y

Nats ko bits mein convert karne ke liye tum

ln 2 \approx 0.6931

se divide karo.