Log-loss and calibration — Hinglish notes

2.6.11 · AI-ML › Model Evaluation & Selection

Overview

Log-loss (logarithmic loss, cross-entropy loss) measure karta hai ki predicted probabilities true outcomes se kitni acchi tarah match karti hain. Calibration measure karta hai ki predicted probabilities true frequencies ko reflect karti hain ya nahi: agar aap 100 events ke liye "70% confident" kehte hain, toh roughly 70 hone chahiye.

Yeh kyun important hai: Ek model ki accuracy high ho sakti hai lekin probabilities bilkul bekar ho sakti hain. Log-loss + calibration batate hain ki aapki uncertainties trustworthy hain ya nahi—yeh medical diagnosis, finance, aur uncertainty ke saath kisi bhi decision ke liye critical hai.

Core Intuition

Log-loss: Math Scratch Se

Binary Classification

Aapke paas true label $y \in {0, 1}$ hai aur class 1 ke liye predicted probability $\overset{p}{^} \in [0, 1]$ hai.

Step 1: Likelihood define karo. Agar $y = 1$ hai, toh aapne jo probability assign ki woh $\overset{p}{^}$ thi. Agar $y = 0$ hai, toh aapne jo probability assign ki woh $1 - \overset{p}{^}$ thi. Combined: $P (data ∣ model) = \overset{p}{^}^{y} \cdot (1 - \overset{p}{^})^{1 - y}$

Yeh step kyun? Yeh Bernoulli likelihood hai—aapki prediction given outcome $y$ observe karne ki basic probability.

Step 2: Negative log lo. Hum likelihood maximize karna chahte hain = negative log-likelihood minimize karna chahte hain: $Log-loss = - lo g P (data ∣ model) = - [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})]$

Negative log kyun?

Numerical stability: 0 ke paas probabilities underflow karti hain; logs numbers ko manageable rakhte hain.
Additive: logs multiple samples par products ko sums mein convert kar dete hain.
Convex: easier optimization (gradient descent cleanly kaam karta hai).

Step 3: Dataset par average karo. $N$ samples ke liye:

Multi-class Classification

True label: one-hot vector $y = [y_{1}, y_{2}, \dots, y_{K}]$ jahan exactly ek $y_{k} = 1$ hai. Predicted probabilities: $\hat{p} = [\overset{p}{^}_{1}, \dots, \overset{p}{^}_{K}]$ with $\sum \overset{p}{^}_{k} = 1$ .

Derivation: True class observe karne ki likelihood $\prod_{k = 1}^{K} \overset{p}{^}_{k}^{y_{k}}$ hai (sirf true class contribute karta hai kyunki baaki ke liye $y_{k} = 0$ hai). Negative log-likelihood:

Yeh step kyun? $k$ par sum ek single term (true class) par collapse ho jaata hai kyunki saare incorrect classes ke liye $y_{i, k} = 0$ hota hai.

Calibration: The Math

Definition

Calibration Measure Karna: Expected Calibration Error (ECE)

Step 1: Predictions bin karo. $[0, 1]$ ko $M$ bins mein divide karo (typically 10-20): $B_{1}, B_{2}, \dots, B_{M}$ .

Step 2: Har bin ke liye accuracy aur confidence compute karo. Bin $B_{m}$ ke liye:

Average predicted probability: $conf (B_{m}) = \frac{1}{∣ B _{m} ∣} \sum_{i \in B_{m}} \overset{p}{^}_{i}$
Actual accuracy: $acc (B_{m}) = \frac{1}{∣ B _{m} ∣} \sum_{i \in B_{m}} y_{i}$

Step 3: Bin size se weight karo aur average karo.

Yeh step kyun? Hum bin size se weight karte hain kyunki 2 samples waale bin ko 200 samples waale bin par dominate nahi karna chahiye. Absolute difference $∣ acc - conf ∣$ calibration gap hai.

Reliability Diagrams

Predicted confidence (x-axis) vs. observed accuracy (y-axis) plot karo. Perfect calibration = diagonal line $y = x$ .

Common patterns:

Overconfident: curve diagonal se neeche (aap 80% kehte hain, reality 60% hai)
Underconfident: curve diagonal se upar (aap 60% kehte hain, reality 80% hai)

Worked Examples

Worked example Example 1: Log-loss Calculation

Setup: 4 binary predictions.

Sample	True $y$	Predicted $\overset{p}{^}$
1	1	0.9
2	1	0.6
3	0	0.2
4	0	0.8
Log-loss calculate karo:
$L_{1} = - [1 \cdot lo g (0.9) + 0 \cdot lo g (0.1)] = - lo g (0.9) = 0.105$
$L_{2} = - lo g (0.6) = 0.511$
$L_{3} = - lo g (1 - 0.2) = - lo g (0.8) = 0.223$
$L_{4} = - lo g (1 - 0.8) = - lo g (0.2) = 1.609$

Yeh step kyun? $y = 1$ ke liye, loss $- lo g (\overset{p}{^})$ hai. $y = 0$ ke liye, loss $- lo g (1 - \overset{p}{^})$ hai.

Average: $L = \frac{0.105 + 0.511 + 0.223 + 1.609}{4} = 0.612$

Insight: Sample 4 (confident wrong prediction) loss par dominate karta hai—yeh by design hai. Log-loss overconfidence ko errors par severely punish karta hai.

Worked example Example 2: Calibration Analysis

Setup: 100 predictions, confidence se binned.

Bin Range	Count	Avg Confidence	Avg Accuracy
0.0-0.2	10	0.15	0.10
0.2-0.4	20	0.30	0.25
0.4-0.6	30	0.50	0.50
0.6-0.8	25	0.70	0.60
0.8-1.0	15	0.90	0.80

ECE calculate karo: $ECE = \frac{10}{100} ∣0.10 - 0.15∣ + \frac{20}{100} ∣0.25 - 0.30∣ + \frac{30}{100} ∣0.50 - 0.50∣$ $+ \frac{25}{100} ∣0.60 - 0.70∣ + \frac{15}{100} ∣0.80 - 0.90∣$ $= 0.1 (0.05) + 0.2 (0.05) + 0.3 (0) + 0.25 (0.1) + 0.15 (0.1)$ $= 0.005 + 0.01 + 0 + 0.025 + 0.015 = 0.055$

Yeh step kyun? Har bin apne size ke proportion mein contribute karta hai. Extremes par bins (0.6-1.0) overconfidence dikhate hain—model 70-90% kehta hai, reality 60-80% hai.

Interpretation: ECE = 5.5%. Model thoda overconfident hai, khaaskar high confidence par. Yeh disastrous nahi hai, lekin calibration fixes (temperature scaling) help karenge.

Common Mistakes

Common mistake Mistake 2: "Softmax outputs probabilities hain"

Kyun sahi lagta hai: Softmax $[0, 1]$ mein values produce karta hai jo 1 sum hoti hain—probabilities jaisi lagti hain!

Steel-man: Mathematically, softmax model ki belief given classes par ek probability distribution hai. Ek perfectly trained, well-specified model ke liye, yeh calibrated hote.

Sach: Neural networks often overconfident softmax scores output karte hain. Logits ka scale arbitrary hai—agar aap sabhi logits ko 10 se multiply karo, softmax sharper (zyada confident) ban jaata hai, lekin predictions change nahi hoti. Model calibrate hone ke liye train nahi hua tha; woh classify karne ke liye train hua tha.

Fix: Raw softmax ko "confidence scores" ki tarah treat karo, probabilities ki tarah nahi. Temperature scaling apply karo: $softmax (z)$ ko $softmax (z / T)$ se replace karo, jahan $T > 1$ smooth karta hai (confidence kam karta hai). ECE minimize karne ke liye validation set par $T$ tune karo.

Calibration Fixes

Temperature Scaling

Problem: Neural net outputs $z$ (logits). Softmax $σ (z)$ overconfident hai.

Solution: Temperature $T$ add karo: $\overset{p}{^}_{k} = \frac{e ^{z_{k} / T}}{\sum _{j} e ^{z_{j} / T}}$

$T$ kaise find karein: Model weights freeze karo. NLL ya ECE minimize karne ke liye validation set par $T$ tune karo. Typically $T \in [1, 5]$ .

Yeh kyun kaam karta hai: Logits ko $T > 1$ se divide karna distribution ko "soften" karta hai—probabilities 0 aur 1 se door uniform ki taraf move karti hain. Yeh retraining ke bina overconfidence correct karta hai.

Platt Scaling (Binary)

Model scores ke upar ek logistic regression fit karo: $P (y = 1) = σ (a \cdot s + b)$ jahan $s$ model ka output hai aur $a, b$ validation data par seekhe jaate hain.

Yeh kyun kaam karta hai: Arbitrary scores ko calibrated probabilities mein map karne ke liye ek affine transformation seekhta hai.

Isotonic Regression (Non-parametric)

Scores se probabilities tak ek piecewise-constant, monotonically increasing function fit karo. Flexible hai lekin chhote validation sets par overfit ho sakta hai.

Connections to Other Concepts

Cross-Entropy Loss: Log-loss true distribution (one-hot) aur predicted distribution ke beech cross-entropy hai.
Information Theory: Log-loss "surprise" measure karta hai—predictions given outcomes encode karne ke liye average bits.
Brier Score: Log-loss ka alternative: $\frac{1}{N} \sum (\overset{p}{^}_{i} - y_{i})^{2}$ . Confident errors ke liye kam sensitive, lekin nicely decompose nahi karta.
ROC-AUC: Ranking quality (discrimination) measure karta hai. Ek model ki perfect AUC ho sakti hai lekin terrible calibration.
Focal Loss: Log-loss ko hard examples par focus karne ke liye modify karta hai: $- (1 - \overset{p}{^})^{γ} lo g \overset{p}{^}$ . Imbalanced settings mein use hota hai.
Bayesian Neural Networks: Uncertainty estimates produce karte hain. Calibration check karta hai ki woh uncertainties honest hain ya nahi.
Proper Scoring Rules: Log-loss "proper" hai—maximize hota hai jab predictions = true probabilities. Honest forecasting encourage karta hai.
Temperature Scaling: Post-hoc calibration method. Logits ko scale karta hai overconfidence fix karne ke liye.
Reliability Diagrams: Calibration ke liye visual tool. Predicted vs. observed frequencies plot karo.

Feynman Technique

Recall Ek 12-saal ke bacche ko samjhao

Socho tum ek weather forecaster ho. Tum kehte ho "70% chance of rain." Agar tum yeh 100 baar kehte ho, toh roughly 70 baar baarish honi chahiye. Agar sirf 50 baar baarish hoti hai, toh tumhari probabilities kharaab hain—tum overconfident ho. Calibration check karta hai ki tumhare percentages honest hain ya nahi.

Ab socho, jab bhi tum galat hote ho, tumhare points kate jaate hain. Lekin sabhi mistakes equal nahi hain. Agar tum "99% no rain" kehte ho aur baarish ho jaati hai, toh tum BAHUT saare points khote ho—tum super confident the aur super galat the. Agar tumne "60% no rain" kaha hota aur baarish ho jaati, toh kam points kate kyunki tum sure nahi the. Yahi log-loss hai: yeh cocky mistakes ko humble ones se bahut zyada punish karta hai.

Kyun? Kyunki real life mein (medicine, driving, investing), galat hone par overconfident hona dangerous hai. Log-loss tumhare AI ko yeh admit karana sakta hai jab woh unsure ho, aur calibration sure karti hai ki uska "main 80% sure hoon" actually kuch maayane rakhta hai.

Mnemonic

Active Recall Flashcards

#flashcards/ai-ml

Log-loss kya hai aur hum logarithms kyun use karte hain?

Log-loss (cross-entropy) hai

- \frac{1}{N} \sum [y lo g \overset{p}{^} + (1 - y) lo g (1 - \overset{p}{^})]

. Hum logarithms isliye use karte hain kyunki: (1) probabilities multiply hoti hain, logs unhe add kara dete hain; (2) 0 ke paas numerical stability; (3) information-theoretic "surprise" measure karta hai; (4) optimization ke liye convex hai.

Binary log-loss formula

L = - \frac{1}{N} \sum_{i = 1}^{N} [y_{i} lo g \overset{p}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{p}{^}_{i})]

jahan

y_{i} \in {0, 1}

\overset{p}{^}_{i} \in [0, 1]

. Range:

[0, \infty)

Multi-class log-loss formula

L = - \frac{1}{N} \sum_{i = 1}^{N} \sum_{k = 1}^{K} y_{i, k} lo g \overset{p}{^}_{i, k}

jahan

y_{i}

one-hot hai,

\hat{p}_{i}

sum hokar 1 deta hai. Equivalent hai

- \frac{1}{N} \sum_{i} lo g \overset{p}{^}_{i, c_{i}}

(true class).

Ek model ke calibrated hone ka matlab kya hai?

Ek model calibrated hota hai agar, un sabhi predictions mein jinki confidence

p

hai, true positive rate

p

ho. Jaise, un sabhi "70% confident" predictions mein se, 70% sahi hone chahiye. Perfect calibration: observed frequency = predicted probability.

Expected Calibration Error (ECE) formula

ECE = \sum_{m = 1}^{M} \frac{∣ B _{m} ∣}{N} ∣ acc (B_{m}) - conf (B_{m}) ∣

jahan

B_{m}

predictions ke bins hain,

acc

bin mein actual accuracy hai,

conf

average predicted probability hai. Range:

[0, 1]

Ek model ki low log-loss lekin poor calibration kyun ho sakti hai?

Log-loss training distribution ke saath fit measure karta hai. Ek model training data par log-loss minimize kar sakta hai lekin test/OOD data par overconfident ho sakta hai, ya poorly extrapolate kar sakta hai. Hamesha calibration alag se ECE ya reliability diagrams se measure karo.

Softmax outputs automatically calibrated probabilities kyun nahi hote?

Softmax

[0, 1]

mein values produce karta hai jo 1 sum hoti hain, lekin logits ka scale arbitrary hai. Neural nets classify karne ke liye train hote hain (loss minimize karna), calibrated hone ke liye nahi. Woh often overconfident scores output karte hain. Fix: temperature scaling.

Temperature scaling kya hai aur yeh calibration kaise fix karta hai?

Temperature scaling

softmax (z)

softmax (z / T)

se replace karta hai jahan

T > 1

ko validation data par tune kiya jaata hai. Logits ko

T

se divide karna distribution ko "soften" karta hai, probabilities ko extremes (0 ya 1) se door le jaata hai, retraining ke bina overconfidence correct karta hai.

Log-loss vs. accuracy: kaun zyada informative hai?

Log-loss zyada informative hai kyunki yeh probability quality evaluate karta hai, sirf binary decisions nahi. Ek model jo sabhi positives ke liye 0.51 predict karta hai uski accuracy 100% hai lekin probabilities bekar hain. Log-loss high hoga, problem reveal karta hua.

Log-loss confident wrong predictions ko kaise penalize karta hai?

Log-loss

- lo g (\overset{p}{^})

hai jab sahi,

- lo g (1 - \overset{p}{^})

hai jab galat. Jaise

\overset{p}{^} \to 0

ek true positive ke liye,

- lo g (\overset{p}{^}) \to \infty

. Ek confident wrong prediction (jaise true class ke liye 0.01) massive loss (~4.6) deta hai, jabki ek timid wrong prediction (0.4) chhota loss (~0.92) deta hai. Exponential penalty.

Reliability diagram kya hota hai?

Predicted confidence (x-axis) vs. observed accuracy (y-axis) ka bins mein plot. Perfect calibration = diagonal line

y = x

. Diagonal se neeche curve = overconfident. Upar = underconfident.

Log-loss ko "cross-entropy" kyun kehte hain?

Cross-entropy

H (p, q) = - \sum p (x) lo g q (x)

woh average bits measure karta hai jo distribution

p

ke data ko

q

ke liye optimized code use karke encode karne mein lagte hain. Classification ke liye,

p

one-hot true label hai,

q

predicted distribution hai. Binary log-loss exactly

H (p, q)

hai Bernoulli distributions ke liye.