Foundations — Logistic regression — sigmoid, cross-entropy loss

Before you can read a single line of the parent topic, you need to already own about a dozen small tools. This page builds every one of them from nothing. Read top to bottom — each tool is used by the next.

0. What "classification" even means

The picture: imagine sorting parts on a conveyor belt into two bins. There is no "1.5 bin". Contrast this with predicting altitude (a continuous number) — that would be regression, the job of Linear Regression. Logistic regression is the classification cousin.

Why the topic needs it: the entire machine is built to output a number we can compare against a label that is only ever 0 or 1.

1. Numbers, variables, and subscripts

Let us fix the alphabet the parent uses, because it hides real meaning in tiny letters.

The picture: think of your dataset as a spreadsheet. Rows are examples (superscript $i$). Columns are features (subscript $j$). One cell is $x_j^{(i)}$.

2. Vectors and the dot product

The parent jumps straight to ${\mathbf{w}}^T\mathbf{x}$. Let us earn it.

The picture below shows a 2-feature example as an arrow in a plane, and shows a weight vector $\mathbf{w}$ living in the same space.

Why this tool and not another? We need one single number that summarizes all features into a "confidence score." The dot product is the simplest way to blend many inputs into one, with a tunable knob ($w_j$) per input. That single number gets its own name:

3. The exponential $e^{-z}$ — the bending tool

Why does the topic need $e^{-z}$ specifically? We want a knob that:

• is never negative (probabilities can't come from negative denominators),
• smoothly connects "very small" to "very large" as $z$ sweeps the number line.

$e^{-z}$ does exactly this: when $z$ is large, $e^{-z}\to 0$; when $z$ is very negative, $e^{-z}\to \infty$. Look at the amber curve — that behaviour is what the sigmoid rides on.

4. Putting it together: the sigmoid shape

Now the parent's headline formula $\sigma (z)=\frac{1}{1+e^{-z}}$ is readable. You already know every piece: $z$ is the raw score, $e^{-z}$ is the always-positive bending tool, and dividing 1 by "1 + something positive" forces the answer into $(0,1)$.

• When $z$ huge positive: $e^{-z}\approx 0$, so $\sigma \approx \frac{1}{1}=1$.
• When $z=0$: $e^0=1$, so $\sigma =\frac{1}{2}=0.5$ — the decision boundary (the cyan cross in the figure).
• When $z$ huge negative: $e^{-z}$ huge, so $\sigma \approx 0$.

Why we care about the smooth S-shape and not a hard step at 0: Gradient Descent needs a slope everywhere to know which way to nudge the weights. A vertical cliff has no usable slope. The gentle S has one at every point.

5. Odds, probability, and the logarithm

The parent derives sigmoid starting from "odds." Two more tools.

The picture: probability is a slider from 0 to 1; odds stretch that same slider out to run from 0 to $+\infty$.

That last property is exactly why the parent takes a $\log$ of the likelihood (a big product) — it becomes a friendly sum. The "log-odds" $\log \frac{p}{1-p}$ stretches probability across the entire number line, which is why it can equal the unbounded score $z$.

6. Derivatives — the "which way is downhill" tool

The picture: stand on a hillside (the loss surface). The derivative is the arrow pointing straight downhill under your feet. Gradient Descent just repeatedly steps in the opposite direction of that arrow to reach the valley floor (the minimum loss).

Why the topic needs derivatives at all: without a slope there is no way to know how to change $\mathbf{w}$ and $b$ to make the loss smaller. Training is slope-following.

7. The cost/loss symbols $L$ and $J$

The picture: $J$ is the height of the landscape Gradient Descent walks down; every training step lowers it. This connects to Maximum Likelihood Estimation — minimizing cross-entropy is exactly the same as choosing weights that make the observed labels most likely.

8. The learning rate $\alpha$

The picture: $\alpha$ is your stride length walking downhill. Too big and you leap across the valley and bounce; too small and you crawl forever. The parent chooses a tiny $\alpha =10^{-6}$ precisely because unnormalized features make the slope huge.

How it all feeds the topic

Read it as a loop: features and weights make $z$; the sigmoid turns $z$ into $\hat{y}$; comparing $\hat{y}$ with $y$ gives loss $J$; derivatives give the downhill direction; gradient descent updates the weights; repeat.

Equipment checklist

Cover the right side and answer out loud. If any one stumps you, re-read its section above.