Foundations — Log-loss and calibration

This is the ground floor for the parent topic. Read Hinglish version: 2.6.11 Log-loss and calibration (Hinglish). Nothing below assumes you have seen probability notation, logarithms, or summation signs before. We build each symbol, anchor it to a picture, and say why the topic needs it.

1. What is a probability? The symbol $p$

Picture a horizontal bar from left ($0$) to right ($1$). Sliding a marker along it is exactly "changing your confidence".

We write a probability with the letter $p$. When it is a probability the model predicts (a guess, not a truth), we put a little hat on it: $\hat{p}$ — read aloud as "p-hat". The hat always means "estimated / predicted by us", never "the real answer".

2. Interval vs set notation: $[0,1]$ and $\{0,1\}$

Before we use them, let us pin down two bracket conventions you will meet everywhere.

So $\hat{p}\in [0,1]$ says "a guess can be any value on the ruler", while $y\in \{0,1\}$ says "the truth is only ever one of two labels". The symbol $\in$ reads "is an element of / lives in".

3. The true outcome: the symbol $y$

Reality is not a probability — it either happened or it did not. We record that with $y$.

Contrast the two symbols so they never blur (both bracket conventions were defined above):

The whole game is comparing a soft guess $\hat{p}$ against a hard truth $y$.

4. The logarithm: the symbol $\log$

This is the tool the topic is named after, so we build it carefully.

The only facts we actually need are visual:

Read three things off the curve of $\log (x)$ for $x$ between 0 and 1:

• $\log (1)=0$. A perfect prediction earns zero penalty.
• As $x$ shrinks toward $0$, $\log (x)$ dives to $-\infty$ (negative infinity, just defined). A tiny probability produces a huge negative number.
• $\log (x)$ is negative for every $x$ in $(0,1)$. That is why the loss formulas always have a minus sign out front — to flip the penalty back to positive.

We will lean on two algebraic properties of logs, so let us name them explicitly.

5. Combining guess and truth: exponents as switches

The parent writes the Bernoulli likelihood ${\hat{p}}^y(1-\hat{p}{)}^{1-y}$. The trick is that $y$ (which is 0 or 1) acts as an on/off switch in the exponent.

Now we turn that likelihood into the per-sample log-loss, one justified algebra step at a time.

Each step had a named reason: product rule to split, power rule to drop exponents, negation to make the score positive. See Cross-Entropy Loss for where this exact expression comes from in Information Theory (it is "bits of surprise"), and Brier Score / Proper Scoring Rules for sibling ways to grade probabilities.

6. Summation and averaging: the symbols $\sum$ and $\frac{1}{N}$

We rarely grade one prediction; we grade a whole dataset of $N$ of them.

The subscript $i$ is just a name tag on each sample: $y_i$ is the truth of sample number $i$, ${\hat{p}}_i$ its predicted probability. Multiplying the whole sum by $\frac{1}{N}$ turns a total into an average — so datasets of different sizes stay comparable.

7. Multi-class notation: the extra index $k$ and one-hot vectors

With more than two classes we need a second name tag. The class index is $k$, running from $1$ to $K$ (the number of classes).

The bold letters $\mathbf{y},\hat{\mathbf{p}}$ signal "this is a whole list (vector)", not a single number.

8. Calibration symbols: bins $B_m$, accuracy, confidence

Calibration needs us to group predictions with similar confidence and check reality inside each group.

Inside a bin we compute two averages:

• $\text{conf}(B_m)$ = average of the model's guesses ${\hat{p}}_i$ in the bin (what it claimed).
• $\text{acc}(B_m)$ = average of the truths $y_i$ in the bin, i.e. the fraction that actually happened (what reality delivered).

The Expected Calibration Error rolls all (non-empty) bins into one number, weighting each by its share $\frac{|B_m|}{N}$ so a tiny bin can't dominate:

Plotting $\text{conf}$ on the x-axis and $\text{acc}$ on the y-axis gives a reliability diagram; the perfect diagonal is $y=x$. See Temperature Scaling for the standard fix when the curve sags below it, and ROC-AUC for a ranking-based metric that ignores calibration entirely.

9. Prerequisite map

Equipment checklist

Cover the right side and test yourself. If any line is fuzzy, re-read its section above.