Log-loss (logarithmic loss, cross-entropy loss) measures how well predicted probabilities match true outcomes. Calibration measures whether predicted probabilities reflect true frequencies: if you say "70% confident" for 100 events, roughly 70 should occur.
Why this matters: A model can have high accuracy but terrible probabilities. Log-loss + calibration tell you if your uncertainties are trustworthy—critical for medical diagnosis, finance, any decision under uncertainty.
You have true label y∈{0,1} and predicted probability p^∈[0,1] for class 1.
Step 1: Define the likelihood.
If y=1, the probability you assigned was p^.
If y=0, the probability you assigned was 1−p^.
Combined:
P(data∣model)=p^y⋅(1−p^)1−y
Why this step? This is the Bernoulli likelihood—basic probability of observing outcome y given your prediction.
Step 2: Take the negative log.
We want to maximize likelihood = minimize negative log-likelihood:
Log-loss=−logP(data∣model)=−[ylogp^+(1−y)log(1−p^)]
Why negative log?
Numerical stability: probabilities near 0 cause underflow; logs keep numbers manageable.
Additive: logs turn products into sums over multiple samples.
Convex: easier optimization (gradient descent works cleanly).
True label: one-hot vector y=[y1,y2,…,yK] where exactly one yk=1.
Predicted probabilities: p^=[p^1,…,p^K] with ∑p^k=1.
Derivation:
The likelihood of observing the true class is ∏k=1Kp^kyk (only the true class contributes since others have yk=0).
Negative log-likelihood:
Why this step? The sum over k collapses to a single term (the true class) because yi,k=0 for all incorrect classes.
Step 1: Bin predictions.
Divide [0,1] into M bins (typically 10-20): B1,B2,…,BM.
Step 2: Compute accuracy and confidence per bin.
For bin Bm:
Average predicted probability: conf(Bm)=∣Bm∣1∑i∈Bmp^i
Actual accuracy: acc(Bm)=∣Bm∣1∑i∈Bmyi
Step 3: Weight by bin size and average.
Why this step? We weight by bin size because a bin with 2 samples shouldn't dominate a bin with 200. The absolute difference ∣acc−conf∣ is the calibration gap.
Problem: Neural net outputs z (logits). Softmax σ(z) is overconfident.
Solution: Add temperature T:
p^k=∑jezj/Tezk/T
How to find T: Freeze model weights. Tune T on validation set to minimize NLL or ECE. Typically T∈[1,5].
Why this works: Dividing logits by T>1 "softens" the distribution—probabilities move away from 0 and 1 toward uniform. This corrects overconfidence without retraining.
Imagine you're a weather forecaster. You say "70% chance of rain." If you say that 100 times, it should rain about 70 times. If it only rains 50 times, your probabilities are bad—you're overconfident. Calibration checks if your percentages are honest.
Now, imagine every time you're wrong, you lose points. But not all mistakes are equal. If you say "99% no rain" and it pours, you lose TONS of points—you were super confident and super wrong. If you said "60% no rain" and it rained, you lose fewer points because you weren't sure. That's log-loss: it punishes cocky mistakes way more than humble ones.
Why? Because in real life (medicine, driving, investing), being overconfident when wrong is dangerous. Log-loss makes your AI admit when it's unsure, and calibration makes sure its "I'm 80% sure" actually means something.
Log-loss (cross-entropy) is −N1∑[ylogp^+(1−y)log(1−p^)]. We use logarithms because: (1) probabilities multiply, logs make them add; (2) numerical stability near 0; (3) measures information-theoretic "surprise"; (4) convex for optimization.
Binary log-loss formula
L=−N1∑i=1N[yilogp^i+(1−yi)log(1−p^i)] where yi∈{0,1}, p^i∈[0,1]. Range: [0,∞).
Multi-class log-loss formula
L=−N1∑i=1N∑k=1Kyi,klogp^i,k where yi is one-hot, p^i sums to 1. Equivalent to −N1∑ilogp^i,ci (true class).
What does it mean for a model to be calibrated?
A model is calibrated if, among all predictions with confidence p, the true positive rate is p. E.g., of all "70% confident" predictions, 70% should be correct. Perfect calibration: observed frequency = predicted probability.
Expected Calibration Error (ECE) formula
ECE=∑m=1MN∣Bm∣∣acc(Bm)−conf(Bm)∣ where Bm are bins of predictions, acc is actual accuracy in bin, conf is average predicted probability. Range: [0,1].
Why can a model have low log-loss but poor calibration?
Log-loss measures fit to the training distribution. A model can minimize log-loss on training data but be overconfident on test/OOD data, or extrapolate poorly. Always measure calibration separately with ECE or reliability diagrams.
Why are softmax outputs not automatically calibrated probabilities?
Softmax produces values in [0,1] summing to 1, but the logits' scale is arbitrary. Neural nets are trained to classify (minimize loss), not to be calibrated. They often output overconfident scores. Fix: temperature scaling.
What is temperature scaling and how does it fix calibration?
Temperature scaling replaces softmax(z) with softmax(z/T) where T>1 is tuned on validation data. Dividing logits by T "softens" the distribution, moving probabilities away from extremes (0 or 1), correcting overconfidence without retraining.
Log-loss vs. accuracy: which is more informative?
Log-loss is more informative because it evaluates probability quality, not just binary decisions. A model predicting 0.51 for all positives has 100% accuracy but useless probabilities. Log-loss would be high, revealing the problem.
How does log-loss penalize confident wrong predictions?
Log-loss is −log(p^) when correct, −log(1−p^) when wrong. As p^→0 for a true positive, −log(p^)→∞. A confident wrong prediction (say 0.01 for true class) yields massive loss (~4.6), while a timid wrong prediction (0.4) yields smaller loss (~0.92). Exponential penalty.
What is a reliability diagram?
A plot of predicted confidence (x-axis) vs. observed accuracy (y-axis) across bins. Perfect calibration = diagonal line y=x. Curve below diagonal = overconfident. Curve above = underconfident.
Why is log-loss called "cross-entropy"?
Cross-entropy H(p,q)=−∑p(x)logq(x) measures the average bits needed to encode data from distribution p using code optimized for q. For classification, p is the one-hot true label, q is the predicted distribution. Binary log-loss is exactly H(p,q) for Bernoulli distributions.
Chalo isko simple tarike se samajhte hain. Socho tumhara model sirf haan/naa nahi bolta, balki confidence bhi deta hai - jaise "70% chance hai ki barish hogi". Ab log-loss ka kaam yeh check karna hai ki yeh probabilities kitni sahi hain. Agar tum bade confidence se (P=0.9) galat prediction karte ho, toh log-loss tumhe bahut zyada penalty deta hai, aur agar timid ho ke (P=0.6) galat hue toh kam penalty. Logarithm isliye use karte hain kyun ki probabilities multiply hoti hain, aur log unhe addition mein badal deta hai - isse math aasan ho jaata hai aur gradient descent smoothly kaam karta hai. Binary case ka formula bas Bernoulli likelihood ka negative log hai, aur multi-class mein sirf true class wala term bachta hai baaki sab zero ho jaate hain.
Ab calibration ka intuition alag hai lekin equally important. Ek model ki accuracy high ho sakti hai, par uski probabilities jhoothi ho sakti hain. Perfect calibration ka matlab hai ki jab tum 100 events ke liye "70% confident" bolte ho, toh actually kareeb 70 hone chahiye - na kam, na zyada. Ise measure karne ke liye hum predictions ko bins mein baant dete hain (jaise 0-0.1, 0.1-0.2...), phir har bin mein dekhte hain ki jitni confidence claim ki thi, utna actual accuracy mila ya nahi. In dono ka difference hi ECE (Expected Calibration Error) deta hai.
Yeh cheez matter kyun karti hai? Kyunki real world mein decisions probabilities pe based hote hain - medical diagnosis mein doctor ko pata hona chahiye ki "80% cancer chance" ka matlab sach mein 80% hai, ya finance mein risk estimate trustworthy hona chahiye. Sirf accuracy dekhna kaafi nahi, kyunki wahan tumhe bas final label sahi chahiye, lekin log-loss aur calibration tumhe batate hain ki tumhare model ki uncertainty bharosemand hai ya nahi. Yaad rakho - ek achha model wahi hai jo apni confidence ke saath honest ho.