Worked examples — Logistic regression — sigmoid, cross-entropy loss

This is a worked-example deep dive for Logistic Regression. The parent note built the machinery — sigmoid, cross-entropy, the gradient $(\hat{y}-y)x_j$. Here we stress-test it against every case the topic can throw at you: every sign of $z$, the degenerate $\hat{y}=0.5$ boundary, the saturated tails where gradients vanish, a real aerospace word problem, and an exam-style twist.

Nothing new is assumed. If a symbol appears, it was earned in the parent. Let us first name the cases.

---

The scenario matrix

Every logistic-regression computation lives in one of these cells. We will hit all of them.

Cell	What makes it distinct	Covered by
A. $z>0$, correct	model confident and right	Ex 1
B. $z<0$, wrong	model confident and wrong — big loss	Ex 2
C. $z=0$ boundary	$\hat{y}=0.5$ exactly — the decision knife-edge	Ex 3
D. Saturated tail	$|z|$ huge — sigmoid flat, gradient $\to 0$	Ex 4
E. Multi-feature vector	full ${\mathbf{w}}^{\top }\mathbf{x}+b$ with mixed signs	Ex 5
F. Full descent step	forward → loss → gradient → weight update	Ex 6
G. Real-world word problem	translate English → features → prediction	Ex 7
H. Exam twist	odds/log-odds reasoning, no calculator sigmoid	Ex 8

Look at the S-curve above. The four coloured dots are the characters of this whole page: green (confident-right, cell A), red (confident-wrong, cell B), yellow (the boundary, cell C), and the faded dots way out on the tails (saturation, cell D). Keep this picture in your head.

---

Example 1 — Cell A: $z>0$, prediction correct

Forecast: guess before computing — $z$ looks positive, $y=1$, so we expect $\hat{y}>0.5$ and a small loss. Write your guess down.

1. Compute the logit $z$. $z={\mathbf{w}}^{\top }\mathbf{x}+b=0.5(2)+0.5(3)-1=1+1.5-1=1.5$. Why this step? Everything downstream needs $z$; it is the single number that carries all feature information into the sigmoid.
2. Squash. $\hat{y}=\sigma (1.5)=\frac{1}{1+e^{-1.5}}=\frac{1}{1+0.2231}=0.8176$. Why this step? The sigmoid turns the raw score $z=1.5$ into a probability. $z>0\Rightarrow \hat{y}>0.5$, exactly as forecast.
3. Loss. Since $y=1$, only the first term survives: $L=-\log (0.8176)=0.2014$. Why this step? Cross-entropy with $y=1$ is $-\log \hat{y}$; a probability near 1 gives a loss near 0.

Verify: $\hat{y}=0.8176>0.5$ ✓ (predicts class 1, matches truth). Loss $0.20$ is small ✓. Sanity: $-\log (0.82)\approx 0.20$, and $\log$ of a number just under 1 is a small negative, so $-\log$ is small positive. Forecast confirmed.

---

Example 2 — Cell B: $z<0$, prediction WRONG

Forecast: truth is cloudy ($y=1$) but if $z$ comes out negative the model votes "not cloudy" — a confident wrong answer, so expect a large loss.

1. Logit. $z=0.01(180)+(-0.02)(45)-1.5=1.8-0.9-1.5=-0.6$. Why this step? Mixed-sign weights: the bright pixels push up, the variance term pushes down. The net $z=-0.6$ is negative.
2. Squash. $\hat{y}=\sigma (-0.6)=\frac{1}{1+e^{0.6}}=\frac{1}{1+1.8221}=0.3543$. Why this step? $z<0\Rightarrow \hat{y}<0.5$: the model says "probably not cloudy."
3. Loss. $y=1$, so $L=-\log (0.3543)=1.0376$. Why this step? Cross-entropy punishes confident wrongness — being at $0.35$ when the answer is 1 costs about 5× the loss of Example 1.

Verify: $\hat{y}<0.5$ but $y=1$ → misclassified ✓. Compare losses: $1.04$ (wrong) vs $0.20$ (right) — the wrong case hurts far more, exactly what a good loss should do ✓.

---

Example 3 — Cell C: the $z=0$ boundary (degenerate)

Forecast: $z$ will be exactly $0$, so $\hat{y}=0.5$ — the model is maximally uncertain. What does the gradient look like when the model refuses to commit?

1. Logit. $z=2(1)-2=0$. Why this step? This is the decision boundary: $\sigma (0)=0.5$ is where "class 0" and "class 1" tie.
2. Squash. $\hat{y}=\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$. Why this step? $e^0=1$, so the sigmoid gives exactly one half — the yellow dot in the figure above.
3. Loss. $y=0$, so $L=-\log (1-\hat{y})=-\log (0.5)=0.6931$. Why this step? At the boundary the loss is $\log 2\approx 0.693$ regardless of the true label — total uncertainty costs the same either way.
4. Gradient. $\partial L/\partial w=(\hat{y}-y)x=(0.5-0)(1)=0.5$. Why this step? Positive gradient → descent will decrease $w$, pushing $z$ below 0 so $\hat{y}$ drops toward the correct class 0.

Verify: $\hat{y}=0.5$ exactly ✓. Loss $=\log 2=0.693$ ✓. The gradient is non-zero at the boundary — this is why logistic regression keeps learning even from ambiguous points, unlike a hard threshold.

---

Example 4 — Cell D: saturation, where gradients vanish

Forecast: $z=8$ is huge, so $\hat{y}$ is glued to $\approx 1$. When the model is right, expect a tiny gradient. When it is wrong at $z=8$ (a confident blunder), the gradient can only be as large as the feature allows — watch.

1. Logit and squash. $z=8$, $\hat{y}=\sigma (8)=\frac{1}{1+e^{-8}}=\frac{1}{1+0.000335}=0.99966$. Why this step? The sigmoid is essentially flat here (see the near-horizontal curve at the right of the figure).
2. Gradient, correct label $y=1$. $\partial L/\partial w=(\hat{y}-1)x=(0.99966-1)(1)=-0.000335$. Why this step? Almost zero. The model is right and confident, so there is nothing to learn — descent barely moves. This is the vanishing-gradient tail.
3. Gradient, wrong label $y=0$. $\partial L/\partial w=(\hat{y}-0)x=0.99966(1)=0.99966$. Why this step? A confident wrong prediction gives the maximum possible gradient magnitude (near 1× the feature) — the model is dragged back hard.

Verify: $\sigma (8)\approx 0.9997$ ✓. Correct-label gradient $\approx -3.35\times 10^{-4}$ (tiny) ✓; wrong-label gradient $\approx 1.0$ (large) ✓. The gradient magnitude $|\hat{y}-y|$ is bounded by 1 — saturation limits learning speed when right, but never when catastrophically wrong.

---

Example 5 — Cell E: multi-feature vector, mixed signs

Forecast: three features tug in different directions; add the bias. Guess the sign of $z$ before crunching.

1. Dot product term by term. ${\mathbf{w}}^{\top }\mathbf{x}=(1.0)(1.2)+(2.0)(-0.5)+(-0.5)(3.0)=1.2-1.0-1.5=-1.3$. Why this step? The dot product is a weighted vote: each feature times its weight, summed. Negative weights let a feature vote for class 0.
2. Add bias. $z=-1.3+0.4=-0.9$. Why this step? The bias $b$ shifts the whole decision boundary; here it softens the negative vote slightly.
3. Squash. $\hat{y}=\sigma (-0.9)=\frac{1}{1+e^{0.9}}=\frac{1}{1+2.4596}=0.2891$.
4. Loss. $y=0$, so $L=-\log (1-\hat{y})=-\log (0.7109)=0.3412$. Why this step? $y=0$ means we want $\hat{y}$ small; $0.29$ is reasonably small, so the loss is modest.

Verify: $z=-0.9<0\Rightarrow \hat{y}<0.5$ ✓, and truth is class 0, so the model is right → small loss $0.34$ ✓.

---

Example 6 — Cell F: one complete gradient-descent step

Forecast: $z=0.6$ is positive but not confident; $y=1$, so we expect $w$ to increase after the step.

1. Forward. $z=0.2(3)+0=0.6$; $\hat{y}=\sigma (0.6)=\frac{1}{1+e^{-0.6}}=0.6457$. Why this step? Establishes the current prediction before we can measure how wrong it is.
2. Loss. $L=-\log (0.6457)=0.4375$. Why this step? Records the "before" cost so we can later confirm the update helped.
3. Gradient. $\partial L/\partial w=(\hat{y}-y)x=(0.6457-1)(3)=-1.0629$. Why this step? Negative → prediction too low for $y=1$; descent must add to $w$.
4. Update. $w:=w-\alpha \partial L/\partial w=0.2-0.5(-1.0629)=0.2+0.5314=0.7314$. Why this step? Gradient descent moves against the gradient; the double negative raises $w$.

Verify: New $z=0.7314(3)=2.194$, new $\hat{y}=\sigma (2.194)=0.8997$ — up from $0.6457$, closer to the target $1$ ✓. New loss $-\log (0.8997)=0.1057<0.4375$ ✓. The step reduced the loss, so it moved in the right direction.

---

Example 7 — Cell G: aerospace word problem

Forecast: both features are high with positive weights → high risk expected. Guess: flagged.

1. Translate English to $z$. $z=1.2(1.5)+0.9(0.8)-1.0=1.8+0.72-1.0=1.52$. Why this step? Each standardized feature times its weight gives its contribution to risk; the bias sets the baseline threshold.
2. Probability. $\hat{y}=\sigma (1.52)=\frac{1}{1+e^{-1.52}}=\frac{1}{1+0.2187}=0.8206$. Why this step? Converts the risk score into an interpretable failure probability — $82\%$.
3. Decision. $\hat{y}=0.8206\ge 0.5$ → flag for inspection. Why this step? Applies the operational threshold; here safety wins.

Verify: $z>0\Rightarrow \hat{y}>0.5$ ✓, decision consistent with "high hours + high vibration = risky" ✓. Failure probability $\approx 82\%$ — plug $z=1.52$: $e^{-1.52}\approx 0.219$, $1/1.219\approx 0.820$ ✓.

---

Example 8 — Cell H: exam twist (odds & log-odds, no sigmoid table)

Forecast: the parent note showed $\log (\text{odds})=z$. So odds $=e^z$, and adding to $z$ multiplies the odds. Guess: doubling-ish.

1. Odds from log-odds. $\text{odds}=e^z=e^{0.7}=2.0138$. Why this step? By definition $\log (p/(1-p))=z$, so exponentiating undoes the log. Odds $\approx 2:1$ in favour of cloudy.
2. Probability from odds. $\hat{y}=\frac{\text{odds}}{1+\text{odds}}=\frac{2.0138}{3.0138}=0.6682$. Why this step? Rearranging $p/(1-p)=\text{odds}$ gives $p=\text{odds}/(1+\text{odds})$ — this equals $\sigma (0.7)$, but we never needed a sigmoid table.
3. Effect of $+0.7$ on odds. New odds $=e^{z+0.7}=e^z\cdot e^{0.7}=\text{old odds}\times 2.0138$. Why this step? This is the whole point of log-odds being additive: a fixed bump in $z$ multiplies the odds by a constant factor $e^{0.7}\approx 2.01$, no matter where you started.

Verify: $\sigma (0.7)=1/(1+e^{-0.7})=1/(1+0.4966)=0.6682$ ✓ — matches step 2 exactly. The odds-multiplier $e^{0.7}=2.0138$ ✓. This is the odds interpretation that makes coefficients readable: weight $w_j$ means "each unit of $x_j$ multiplies the odds by $e^{w_j}$."

---

Recall Quick self-test across all cells

Which cell has $\hat{y}=0.5$ exactly? ::: Cell C, the boundary $z=0$. When is the gradient vanishingly small? ::: Cell D — saturated tail AND already correct. What does adding $\Delta$ to $z$ do to the odds? ::: Multiplies them by $e^{\Delta }$ (Cell H). Why does a confident wrong prediction still learn fast? ::: Its gradient magnitude $|\hat{y}-y|$ is near 1, the maximum. The one-line gradient formula? ::: $\partial J/\partial w_j=(\hat{y}-y)x_j$ averaged over examples.