Log-loss - binary cross-entropy — Hinglish notes

2.2.10 · AI-ML › Linear & Logistic Regression

KYUN chahiye nayi loss? (Pehle MSE ko steel-man karo)

Linear regression mein humne Mean Squared Error use ki thi. Ise reuse karna tempting lagta hai: $(\overset{y}{^} - y)^{2}$ .

KIYA hai loss? — first principles se derive kiya

HOW probability ko loss mein convert karte hain (derivation):

Hum chahte hain woh parameters jo observed data ko sabse zyada likely banayein (Maximum Likelihood).
$N$ independent points ki likelihood: $L = \prod_{i = 1}^{N} \overset{y}{^}_{i}^{y_{i}} (1 - \overset{y}{^}_{i})^{1 - y_{i}}$ . Kyun product? Independence ⇒ joint probability multiply hoti hai.
Products painful hote hain; $lo g$ lo (monotone hai, toh maximizer unchanged rehta hai): $lo g L = \sum_{i} [y_{i} lo g \overset{y}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{y}{^}_{i})]$ Kyun log? Product ko sum mein convert karta hai, aur tiny probabilities ka underflow avoid karta hai.
Log-likelihood maximize karna = uska negative minimize karna. Scale-independent banane ke liye $N$ se average lo:

Figure — Log-loss - binary cross-entropy

Gradient — jahan magic cancellation hoti hai

Maano $z = w^{⊤} x + b$ aur $\overset{y}{^} = σ (z) = \frac{1}{1 + e ^{- z}}$ . Sigmoid ka key fact: $σ^{'} (z) = σ (z) (1 - σ (z)) = \overset{y}{^} (1 - \overset{y}{^})$

Single-example loss $ℓ = - [y lo g \overset{y}{^} + (1 - y) lo g (1 - \overset{y}{^})]$ lo aur $z$ ke w.r.t. differentiate karo:

= \frac{\hat y - y}{\hat y(1-\hat y)}$$ *Kyun yeh step?* Bas har log term par chain rule; common denominator par combine karo. $$\frac{\partial \ell}{\partial z} = \frac{\partial \ell}{\partial \hat y}\cdot\frac{d\hat y}{dz} = \frac{\hat y - y}{\hat y(1-\hat y)}\cdot \hat y(1-\hat y) = \boxed{\hat y - y}$$ > [!intuition] Yeh kyun beautiful hai > Uljha hua $\hat y(1-\hat y)$ **cancel** ho jaata hai. Gradient bas **prediction − truth** hai, exactly linear regression jaise. Toh: > $$\frac{\partial J}{\partial w_j} = \frac{1}{N}\sum_i (\hat y_i - y_i)\,x_{ij}, \qquad \frac{\partial J}{\partial b} = \frac{1}{N}\sum_i(\hat y_i - y_i)$$ > Confidently galat hone par koi vanishing gradient nahi — error term $(\hat y - y)$ bada rehta hai. --- ## Worked examples > [!example] Confident aur sahi > True $y=1$, model kehta hai $\hat y = 0.99$. > $\ell = -[1\cdot\log 0.99 + 0] = -\log 0.99 \approx 0.01$. > **Kyun yeh step?** $y=1$ $(1-y)$ term ko khatam kar deta hai. Chhoti loss = achha. > [!example] Confident aur GALAT (saza) > True $y=1$, model kehta hai $\hat y = 0.01$. > $\ell = -\log 0.01 = \log 100 \approx 4.6$. > **Kyun yeh step?** Same surviving term, lekin ab ek tiny number ka $\log$ bada negative hai → badi positive loss. Jaise $\hat y\to 0$, $\ell\to\infty$: kabhi pakka aur galat mat bano. > [!example] Full-batch computation > Do points: $(y=1,\hat y=0.8)$, $(y=0,\hat y=0.3)$. > $\ell_1 = -\log 0.8 = 0.223$. $\ell_2 = -\log(1-0.3)=-\log 0.7 = 0.357$. > $J = \tfrac12(0.223+0.357) = 0.29$. > **Kyun yeh step?** Per-example losses average karo; har ek sirf apne label se match karne wala term use karta hai. --- ## Forecast-then-Verify > [!recall]- Forecast: us model ki log-loss kya hai jo hamesha $\hat y = 0.5$ predict karta hai? > Har point $-\log 0.5 = \log 2 \approx 0.693$ contribute karta hai, label se independent. Toh $J = \ln 2 \approx 0.693$. Yeh **baseline** hai — koi bhi useful model $0.693$ nats (ya $\log_2$ use karne par $1$ bit) beat kare. --- > [!mistake] "Log-loss negative ho sakti hai." > **Kyun sahi lagta hai:** aage minus sign hai. > **Fix:** $\hat y\in(0,1)\Rightarrow \log\hat y < 0$, aur leading minus use positive kar deta hai. Log-loss hamesha $\ge 0$ hoti hai, 0 sirf true class par perfect confidence ki (unreachable) limit mein hit hoti hai. > [!mistake] "Loss mein exactly 0 ya 1 feed karo." > **Kyun sahi lagta hai:** ek perfect prediction *hoti hi* 0 ya 1 hai. > **Fix:** $\log 0 = -\infty$ → NaN. Practice mein $\hat y$ ko $[\varepsilon, 1-\varepsilon]$ par clip karo (jaise $\varepsilon=10^{-7}$), ya better, numerically stable `logits`-based implementation use karo. --- ## Feynman > [!recall]- Ek 12-saal ke bache ko explain karo > Socho tum points bet kar rahe ho ki barish hogi ya nahi. Agar tum kaho "90% sure hai barish hogi" aur barish hoti hai, tum bahut kam lose karte ho. Agar tum chillao "Mujhe 99% PAKKA hai barish nahi hogi!" aur baarish aa jaaye, tum *bahut zyada* points lose karte ho. Log-loss wahi point system hai: humble rehna kam cost karta hai, loud-and-galat rehna bahut cost karta hai. Computer apne guesses ko adjust karta hai taaki kam se kam points lose kare. > [!mnemonic] **"Log the Right one."** > Log-loss = **−log(probability jo tumne RIGHT answer ko di)**. Sahi answer ko high prob mili → chhoti loss. Sahi answer ko tiny prob mili → badi loss. --- ## Active-recall flashcards #flashcards/ai-ml Sigmoid output ke saath MSE kyun use nahi karte? ::: MSE + sigmoid non-convex hai aur iske gradient mein ek $\sigma'=\hat y(1-\hat y)$ factor hai jo confidently galat hone par vanish ho jaata hai, learning stall kar deta hai. Log-loss kis principle se derive hoti hai? ::: Bernoulli distribution ka Maximum likelihood (phir negative log lo aur average karo). Ek label ki Bernoulli likelihood likho. ::: $P(y\mid\hat y)=\hat y^{y}(1-\hat y)^{1-y}$. Single-example log-loss batao. ::: $\ell=-[y\log\hat y+(1-y)\log(1-\hat y)]$. Log-loss words mein kya equal hai? ::: Model ne true class ko jo probability di, uska minus log. Sigmoid + log-loss ke liye $\partial\ell/\partial z$ kya hai? ::: $\hat y - y$ ($\hat y(1-\hat y)$ cancel ho jaata hai). $J$ ka $w_j$ ke w.r.t. gradient? ::: $\frac{1}{N}\sum_i(\hat y_i-y_i)x_{ij}$. Hamesha-0.5 model ki log-loss? ::: $\ln 2\approx 0.693$ (beat karne wala baseline). Kya log-loss negative ho sakti hai? ::: Nahi; hamesha $\ge 0$, kyunki $-\log(\text{prob in }(0,1))\ge0$. $\hat y$ ko clip kyun karte hain? ::: $\log 0=-\infty$ / NaN avoid karne ke liye; $[\varepsilon,1-\varepsilon]$ par clip karo. --- ## Connections - [[Sigmoid function]] — $\hat y=\sigma(z)$ aur $\sigma'=\hat y(1-\hat y)$ identity provide karta hai. - [[Logistic Regression]] — woh model jiska yeh loss hai. - [[Maximum Likelihood Estimation]] — poore formula ki origin. - [[Cross-Entropy and KL Divergence]] — log-loss true label distribution aur predicted distribution ke beech cross-entropy hai. - [[Gradient Descent]] — clean $(\hat y - y)x$ gradient consume karta hai. - [[Mean Squared Error]] — woh loss jo humne yahan reject ki, aur kyun. - [[Softmax and Categorical Cross-Entropy]] — multi-class generalization. ## 🖼️ Concept Map ```mermaid flowchart TD LR[Logistic regression outputs prob y-hat] MSE[MSE with sigmoid] NC[Non-convex + vanishing gradient] MLE[Maximum Likelihood] BER[Bernoulli model] LIK[Likelihood = product] LOGL[Log-likelihood = sum] BCE[Binary Cross-Entropy loss] SINGLE[-log prob of true class] SIG[Sigmoid derivative y-hat 1-y-hat] GRAD[Gradient y-hat - y] LR -->|tempting reuse| MSE MSE -->|suffers from| NC NC -->|fix: switch to| MLE MLE -->|assumes| BER BER -->|independence gives| LIK LIK -->|take log| LOGL LOGL -->|negate + average| BCE BCE -->|one term survives| SINGLE SIG -->|cancels in| GRAD BCE -->|differentiate| GRAD GRAD -->|clean update, no stall| LR ```