Foundations — Log-loss - binary cross-entropy

Everything below builds those tools from scratch, in the exact order the parent note uses them. No symbol appears before its plain-words meaning and its picture.

1. A label $y$ — the ground truth

The curly braces $\{\}$ just list the only allowed values. There is no "0.5 truth" — reality already happened, so the answer is one or the other.

Why the topic needs it: log-loss compares a guess to this fixed 0-or-1 answer. Without a crisp truth there is nothing to score against.

2. A probability $\hat{y}$ — the model's guess

The little hat on $\hat{y}$ (read "y-hat") is a universal convention meaning "estimated version of $y$". It is not the truth; it is the model's opinion.

Round brackets $()$ = open interval (endpoints excluded); square brackets $[]$ = closed interval (endpoints included). This tiny distinction matters later: $\hat{y}$ can crawl close to 0 or 1 but is never allowed to touch them, because touching breaks the logarithm (§4).

Picture $\hat{y}$ as a slider from 0 to 1. If the slider sits at 0.9 the model is saying "I'm 90% sure the answer is 1." Look at the red marker: it can slide right up against the walls but the walls themselves are forbidden.

Why the topic needs it: the whole loss is a function of this one number. See Sigmoid function and Logistic Regression for where $\hat{y}$ comes from.

3. The set $\{0,1\}$ and the "only one term survives" trick

Because $y$ is exactly 0 or 1, any expression multiplied by $y$ or by $(1-y)$ acts like a switch:

This is why the parent's Bernoulli formula ${\hat{y}}^y(1-\hat{y}{)}^{1-y}$ collapses to a single case each time — the exponents 0 and 1 turn factors on and off ($a^1=a$, $a^0=1$).

Why the topic needs it: it lets one clean formula cover both label cases without an if statement.

4. The logarithm $\log$ — turning "unlikely" into "costly"

This is the star tool, so we build it slowly.

In machine learning "$\log$" with no base written means the natural log $\ln$, base $e\approx 2.718$. The choice of base only rescales the answer (natural log gives units called nats), so it never changes which model wins.

Why not just use the probability directly? Why introduce a new tool at all? Because we need a scoring rule with two specific shapes, and only $-\log$ delivers both:

1. As the correct-answer probability approaches 1, the cost should fall to 0. Indeed $-\log (1)=0$.
2. As that probability approaches 0, the cost should explode to $\infty$, punishing confident mistakes brutally. Indeed $-\log (x)\to \infty$ as $x\to 0^{+}$.

No polynomial does this — you need the log's blow-up near zero.

Look at the red curve of $-\log (\hat{y})$: flat and near zero on the right (you gave the right answer high probability → cheap), and rocketing up on the left (you gave it almost no chance → ruinously expensive). Trace the marked points: $-\log (0.99)\approx 0.01$ vs $-\log (0.01)\approx 4.6$.

Why the topic needs it: $-\log (\text{prob of the true class})$ is the log-loss. Everything else is bookkeeping.

5. The sigmoid $\sigma (z)$ — where $\hat{y}$ is born

The model first computes a plain number $z$ (a weighted sum, could be any value from $-\infty$ to $+\infty$). We must squash it into a valid probability in $(0,1)$.

The red S-curve is the sigmoid. Notice it lives entirely inside the $(0,1)$ band — that's why its output is a legitimate $\hat{y}$. The dashed line marks $\sigma (0)=0.5$, the point of maximum indecision.

One fact the parent uses without proof — the slope of the sigmoid: ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))=\hat{y}(1-\hat{y})$ Read ${\sigma }^{\prime }(z)$ as "the derivative — how fast $\sigma$ rises as $z$ increases." Its value is largest at $z=0$ (steepest in the middle) and near zero far out on either side (flat tails). This exact expression is the thing that magically cancels inside the gradient in the parent note.

Why the topic needs it: $\hat{y}=\sigma (z)$ links the model's raw score to the probability the loss consumes. See Sigmoid function.

6. Sum $\sum$, product $\prod$, and the average $\frac{1}{N}$

The parent scores a whole dataset of $N$ examples, so we need shorthand for "combine all of them."

• Product $\prod$ appears when we combine independent probabilities — independence means joint chance = multiply.
• Sum $\sum$ appears after we take a log, because $\log (a\times b)=\log a+\log b$ turns products into sums (easier, and no tiny-number underflow).
• The factor $\frac{1}{N}$ simply averages, so the loss doesn't grow just because you have more data — it stays comparable across dataset sizes.

Why the topic needs it: it's the machinery that lifts the one-example loss $\ell$ to the whole-batch loss $J$.

7. Two loss symbols: $\ell$ and $J$

Why the topic needs it: the parent derives $\ell$ first (clean, single-case), then averages into $J$ (what gradient descent actually minimizes — see Gradient Descent).

Sanity check with the parent's numbers

Prerequisite map

Equipment checklist