Exercises — Logistic regression — sigmoid, cross-entropy loss

Solution 1.1 What we do: plug $z=0$ into $\sigma (z)=\frac{1}{1+e^{-z}}$. $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{1+1}=\frac{1}{2}=0.5$ What it looks like: the exact middle of the S-curve — the point marked in the figure above where the curve crosses the dotted 0.5 line. Decision: $\hat{y}=0.5\ge 0.5$, so we predict class 1 (the boundary is inclusive by our convention).

Solution 1.2

• $N$ = the number of training examples we average over. Averaging (not just summing) means the loss size does not blow up just because we collected more data.
• $y^{(i)}$ = the true label of example $i$, which is either $0$ or $1$ (never in between).
• ${\hat{y}}^{(i)}$ = the model's predicted probability that example $i$ is class 1, a number in $(0,1)$.
• The minus sign flips "log-likelihood we want to maximise" into a "loss we want to minimise". Optimisers by convention go downhill, so we hand them something that gets smaller as the fit improves.

Fill-in check $\sigma (z)\to 1$ as $z\to$ ::: $+\infty$ $\sigma (z)\to 0$ as $z\to$ ::: $-\infty$ The value of $z$ where $\sigma$ equals exactly $0.5$ is ::: $z=0$

Solution 2.1 Step 1 — logit (the dot product plus bias). Why the dot product? Because ${\mathbf{w}}^T\mathbf{x}$ is the single number that says "how much do the features, weighted by importance, push toward class 1." $z={\mathbf{w}}^T\mathbf{x}+b=(0.5)(2)+(-1)(3)+1=1-3+1=-1$ Step 2 — squash it. $\hat{y}=\sigma (-1)=\frac{1}{1+e^1}=\frac{1}{1+2.71828}=\frac{1}{3.71828}=0.2689$ What it looks like: $z=-1$ sits to the left of centre in the sigmoid figure, so we land below 0.5. Decision: $0.2689<0.5\Rightarrow$ class 0.

Solution 2.2 What we do: use $L=-[y\log \hat{y}+(1-y)\log (1-\hat{y})]$. With $y=1$ the second term dies ($1-y=0$), leaving only $-\log \hat{y}$. $L=-\log (0.2689)=1.3133$ Why so large? Truth was class 1 but the model only gave it probability $0.27$ — a confident wrong lean. Cross-entropy punishes that. (Logarithms here are natural logs, base $e$, matching the $e$ inside the sigmoid.)

Solution 2.3 Step 1 — the error term. $\hat{y}-y=0.2689-1=-0.7311$. Negative because the prediction was too low. Step 2 — per-weight gradients. $\frac{\partial L}{\partial w_1}=(-0.7311)(2)=-1.4622,\frac{\partial L}{\partial w_2}=(-0.7311)(3)=-2.1933$ Step 3 — descend ($w_j:=w_j-\alpha \partial L/\partial w_j$): $w_1:=0.5-0.1(-1.4622)=0.6462,w_2:=-1-0.1(-2.1933)=-0.7807$ What it looks like: both weights moved so as to raise the next prediction — exactly what we want since the truth is class 1. This is one step of Gradient Descent.

Solution 3.1 Compute: $(\hat{y}-y)x_j=(1-1)x_j=0$ for every feature $j$. Why it matters: the update $w_j:=w_j-\alpha \cdot 0=w_j$ leaves the weight unchanged. A perfectly-classified point contributes nothing to further learning — the algorithm has already "used up" what that point can teach. This is the geometric meaning of a minimum: no slope, no motion. The subtle bit: $\hat{y}=1$ requires $z\to +\infty$, which is unreachable with finite weights. So in practice the gradient shrinks toward zero but never truly hits it — the model keeps inching weights larger. That endless drift is exactly the problem Regularization (L1, L2) exists to stop.

Solution 3.2 Losses: $L_a=-\log (0.5)=0.6931,L_b=-\log (0.01)=4.6052$ Error terms: $(\hat{y}-y{)}_a=0.5-1=-0.5,(\hat{y}-y{)}_b=0.01-1=-0.99$ Reading it: the confidently-wrong case (b) has both a far larger loss and a larger-magnitude error term, so it produces a stronger corrective push. This is the whole point of cross-entropy over squared error: being confident and wrong hurts a lot, which drags the model out of that bad state fast. A learner that is merely unsure gets a gentle nudge.

Solution 4.1 Forward pass, point 1 ($z^{(1)}=1\cdot 0.5+0=0.5$): ${\hat{y}}^{(1)}=\sigma (0.5)=\frac{1}{1+e^{-0.5}}=0.6225$ Forward pass, point 2 ($z^{(2)}=1\cdot 2.0+0=2.0$): ${\hat{y}}^{(2)}=\sigma (2.0)=\frac{1}{1+e^{-2.0}}=0.8808$ Error terms: $(0.6225-0)=0.6225$; $(0.8808-1)=-0.1192$. Averaged gradients ($N=2$): $\frac{\partial J}{\partial w}=\frac{1}{2}[(0.6225)(0.5)+(-0.1192)(2.0)]=\frac{1}{2}[0.3112-0.2384]=0.0364$ $\frac{\partial J}{\partial b}=\frac{1}{2}[0.6225+(-0.1192)]=\frac{1}{2}(0.5034)=0.2517$ Update: $w:=1.0-0.5(0.0364)=0.9818,b:=0.0-0.5(0.2517)=-0.1258$ What it looks like / sanity: point 1 (a non-failure the model over-predicted) pushes $w$ down; point 2 (a failure it slightly under-predicted) pushes $w$ up. They nearly cancel on $w$, but both agree on lowering $b$ — shifting the whole S-curve rightward so scores of $0.5$ read as "less likely to fail." A coherent, honest step.

Solution 4.2 Key link: $\hat{y}=0.5⟺\sigma (z)=0.5⟺z=0$. So the boundary is wherever the logit is exactly zero — this is why the boundary of logistic regression is always a straight line (or hyperplane), inherited straight from Linear Regression. Set $z=0$: $2x_1-1\cdot x_2-1=0⟹x_2=2x_1-1$ Description: a straight line of slope $2$ and intercept $-1$. On one side $z>0$ (predict class 1), on the other $z<0$ (predict class 0). The sigmoid only decides how sharply the probability turns as you cross that line, not where the line is.

Solution 5.1 Start from the definition and substitute $-z$: $\sigma (-z)=\frac{1}{1+e^{-(-z)}}=\frac{1}{1+e^z}$ Multiply top and bottom by $e^{-z}$: $=\frac{e^{-z}}{e^{-z}+1}=\frac{e^{-z}+1-1}{1+e^{-z}}=\frac{1+e^{-z}}{1+e^{-z}}-\frac{1}{1+e^{-z}}=1-\sigma (z)$ Interpretation: the probability of "not class 1" at score $z$ equals the probability of "class 1" at the flipped score $-z$. The curve has point symmetry about $(0,0.5)$. This is exactly why a two-class problem needs only one sigmoid: modelling $P(\text{class }1)$ automatically pins down $P(\text{class }0)=1-\hat{y}$. When you have more than two classes you lose this freebie and must move to Softmax Regression.

Solution 5.2 First derivative — chain the two known facts $\frac{dL}{d\hat{y}}=-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}$ and $\frac{d\hat{y}}{dz}=\hat{y}(1-\hat{y})$ (with $\hat{y}=\sigma (z)$): $\frac{dL}{dz}=[-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}}]\hat{y}(1-\hat{y})=-y(1-\hat{y})+(1-y)\hat{y}=\hat{y}-y=\sigma (z)-y$ Second derivative — differentiate $\sigma (z)-y$; the constant $y$ dies and ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$: $\frac{d^{2}L}{d{z}^2}=\sigma (z)(1-\sigma (z))$ Sign: $\sigma (z)\in (0,1)$, so both factors are strictly positive, so the second derivative is strictly positive everywhere. Conclusion: a function with everywhere-positive second derivative is convex — it curves upward like a bowl, guaranteeing a single global minimum with no false valleys to trap Gradient Descent. This is the rigorous reason we abandoned squared error, and it is the payoff promised by the Maximum Likelihood Estimation derivation in the parent note.

Solution 5.3 $\sigma (0)=0.5$. First derivative: $\frac{dL}{dz}=\sigma (0)-y=0.5-1=-0.5$ (negative: at $z=0$ the loss still decreases as we push $z$ up toward the correct class 1 — good). Second derivative: $\sigma (0)(1-\sigma (0))=0.5\times 0.5=0.25>0$ (positive: the bowl curves up, confirming convexity at this point).

Mnemonic for the whole page The one identity that unlocks every exercise here ::: the gradient collapses to $(\hat{y}-y)x_j$ — "prediction minus truth, scaled by the feature."