Exercises — Logistic regression and the sigmoid function

The sigmoid range is the open interval $(0,1)$: it can get arbitrarily close to $0$ or $1$ but never reaches or exceeds either. Since $1.4>1$, it cannot be a sigmoid output. The mistake is almost certainly forgetting the squash and reporting the raw score $z$ instead of $\sigma (z)$.

The pivot is $z=0\Rightarrow \sigma (0)=0.5$.

• $z=-2$: negative score $\Rightarrow \sigma <0.5$ ($=0.119$) $\Rightarrow$ predict class 0.
• $z=0$: exactly $0.5$ $\Rightarrow$ the boundary, a tie.
• $z=+2$: positive score $\Rightarrow \sigma >0.5$ ($=0.881$) $\Rightarrow$ predict class 1. Rule: sign of $z$ decides the class, magnitude decides confidence.

Step 1 (score): $z=3(1)-3=0$. Why first? The linear part is computed before squashing. Step 2 (squash): $\sigma (0)=0.5$. Step 3 (classify): $p=0.5$ is exactly on the boundary — a tie. By the convention $\hat{y}=1⟺p\ge 0.5$, we predict class 1 (but with zero confidence). This is precisely the decision boundary $x=1$.

The boundary is $z=0$: $2x_1+x_2-6=0$, i.e. $x_2=6-2x_1$. Why $z=0$? Because $\sigma (z)=0.5$ exactly at $z=0$, and $0.5$ is the classification threshold.

• Cross $x_1$-axis ($x_2=0$): $2x_1=6\Rightarrow x_1=3$.
• Cross $x_2$-axis ($x_1=0$): $x_2=6$. The line through $(3,0)$ and $(0,6)$ is the linear boundary.

Step 1: $z=0\cdot 2+0=0\Rightarrow p=\sigma (0)=0.5$. Step 2 (error): $p-y=0.5-0=0.5$ (we over-predicted a negative example). Step 3 (gradients, single point so $m=1$): $\frac{\partial J}{\partial w}=(p-y)x=0.5\cdot 2=1,\frac{\partial J}{\partial b}=(p-y)\cdot 1=\mathrm{0.5.}$ Step 4 (update, subtract $\alpha \cdot$gradient): $w\leftarrow 0-1\cdot 1=-1,b\leftarrow 0-1\cdot 0.5=-\mathrm{0.5.}$ Sanity: both went negative, which lowers $z$ for this positive-$x$ point, pushing $p$ toward $0=y$. Correct direction.

Start from the definition $\sigma (-z)=\frac{1}{1+e^{-(-z)}}=\frac{1}{1+e^z}$. Now multiply numerator and denominator by $e^{-z}$: $\sigma (-z)=\frac{1}{1+e^z}=\frac{1\cdot e^{-z}}{(1+e^z)e^{-z}}=\frac{e^{-z}}{e^{-z}+1}=\frac{e^{-z}}{1+e^{-z}}.$ And $1-\sigma (z)=1-\frac{1}{1+e^{-z}}=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}.$ The two expressions are identical, so $\sigma (-z)=1-\sigma (z)$. ∎ Interpretation: flipping the sign of the score swaps the two class probabilities, since $P(y=0)=1-P(y=1)$. The model treats the classes symmetrically about $z=0$.

With $y=1$: error $=|p-1|=1-\sigma (z)$. As $z\to -\infty$, $\sigma (z)\to 0$, so $|p-y|\to 1$ — the largest possible error. Cross-entropy keeps a strong, non-vanishing gradient exactly when the model is confidently wrong. Good. Squared-error loss $(p-y{)}^2$ instead has gradient $\propto (p-y){\sigma }^{\prime }(z)$, and ${\sigma }^{\prime }(z)=\sigma (1-\sigma )\to 0$ in the tails. So a confidently-wrong point produces a near-zero gradient — learning stalls. This is the analytic reason we use log loss (see Cross-Entropy Loss).

Odds of class 1 $=e^z=e^{-0.7}\approx 0.497$. Probability $p=\frac{e^z}{1+e^z}=\frac{0.497}{1.497}\approx 0.332$. Since odds $\approx 0.5$, class 1 is about half as likely as class 0 — equivalently class 0 is about twice as likely as class 1 (odds of class 0 $=1/0.497\approx 2.01$). Log-odds negative $\Rightarrow$ leans class 0, consistent with $p<0.5$.

Convert each probability to its logit $z=\ln \frac{p}{1-p}$ (undo the sigmoid):

• $p=0.5\Rightarrow z=\ln 1=0$, at $x=4$: $4w+b=0$.
• $p=0.731\Rightarrow z=\ln \frac{0.731}{0.269}=\ln (2.718)=1$, at $x=5$: $5w+b=1$. Subtract the equations: $(5w+b)-(4w+b)=1-0\Rightarrow w=1$. Then $b=-4w=-4$. Model: $z=x-4$. Check: boundary at $x=4$ (where $p=0.5$) — matches the first point. ∎

Step 0: $z=0,p=0.5,\text{err}=p-y=-0.5$. Update 1: $w\leftarrow 0-1(-0.5)(1)=0.5$, $b\leftarrow 0-1(-0.5)=0.5$. Recompute: $z=0.5(1)+0.5=1\Rightarrow p=\sigma (1)=0.731$. Moved from $0.5$ up toward $1$. ✓ Step 1 error: $p-y=0.731-1=-0.269$. Update 2: $w\leftarrow 0.5-1(-0.269)(1)=0.769$, $b\leftarrow 0.5-1(-0.269)=0.769$. Recompute: $z=0.769+0.769=1.538\Rightarrow p=\sigma (1.538)=\frac{1}{1+e^{-1.538}}\approx 0.823$. Each step raises $p$ (0.5 → 0.731 → 0.823) toward the target $1$, and the error shrinks ($-0.5\to -0.269$). This is Gradient Descent working on convex log loss.

Rewrite the desired line as $x_2-x_1-1=0$. We need $z=w_1{x}_1+w_2{x}_2+b$ to equal this (up to a positive scale) and be positive above the line. Take $w_1=-1,w_2=1,b=-1$, giving $z=-x_1+x_2-1$.

• On the line, e.g. $(0,1)$: $z=0+1-1=0$. ✓
• A point above, e.g. $(0,2)$: $z=0+2-1=1>0\Rightarrow p>0.5\Rightarrow$ class 1. ✓ Any positive multiple $(-c,c,-c)$ works too — scaling $z$ leaves the boundary fixed but sharpens confidence. Compare to Decision Boundaries.

Chain rule: $\frac{\partial J}{\partial w}=\frac{\partial J}{\partial p}\cdot \frac{\partial p}{\partial z}\cdot \frac{\partial z}{\partial w}.$ Piece 1 — $\partial J/\partial p$: $\frac{\partial J}{\partial p}=-\left[\frac{y}{p}-\frac{1-y}{1-p}\right]=-\frac{y(1-p)-p(1-y)}{p(1-p)}=-\frac{y-p}{p(1-p)}=\frac{p-y}{p(1-p)}.$ Piece 2 — $\partial p/\partial z$: the sigmoid identity ${\sigma }^{\prime }(z)=p(1-p)$. Piece 3 — $\partial z/\partial w=x$. Multiply: $\frac{\partial J}{\partial w}=\frac{p-y}{p(1-p)}\cdot p(1-p)\cdot x=(p-y)x.$ The $p(1-p)$ cancels exactly — that cancellation is why logistic regression has the same clean update as Linear Regression. ∎

$J(w)=-\ln \sigma (w)$.

• $w=-2$: $\sigma (-2)=0.119\Rightarrow J=-\ln 0.119=2.127$.
• $w=0$: $\sigma (0)=0.5\Rightarrow J=-\ln 0.5=0.693$.
• $w=2$: $\sigma (2)=0.881\Rightarrow J=-\ln 0.881=0.127$. Values fall $2.127\to 0.693\to 0.127$ as $w$ increases: the loss slopes steadily downhill toward the correct classification, with no bump — a picture of convexity. (As $w\to \infty$, $J\to 0$; the minimum is approached, never a spurious dip.)

Solve $\sigma (z)=0.99$: take the logit, $z=\ln \frac{0.99}{0.01}=\ln 99\approx \mathrm{4.595.}$ So any $z\ge 4.595$ already gives $\ge 99\%$ confidence. Push $z$ to $10$ and $p=0.99995$ — a tiny gain for more than double the score. Interpretation: the sigmoid saturates; beyond the tails, extra score buys negligible probability. That flatness is exactly why ${\sigma }^{\prime }\to 0$ in the tails (L3.2), and why we care about the log-odds scale rather than raw probability for very confident points.

Every exercise here was the same three moves in disguise: score → squash → compare. Compute $z$, run it through the S-slide to get a confidence $p$, then measure how wrong you were with $(p-y)$ and nudge the dials. Going backward (probability to score) is the log-odds. Everything else is bookkeeping.