5.6.2 · D5Machine Learning (Aerospace Applications)
Question bank — Logistic regression — sigmoid, cross-entropy loss
This bank hunts the misconceptions that logistic regression invites — the confusions about what the sigmoid means, why cross-entropy and not squared error, what "linear" refers to, and what happens at the boundaries (zero probability, perfect separation, saturated logits). Read the question, answer out loud in a full sentence, then reveal.
True or false — justify
Logistic regression is a regression algorithm, not a classification one.
False in practice — it outputs a continuous probability via the sigmoid (so "regression" in the name refers to modelling the log-odds), but we threshold that probability to classify, so it is used as a classifier.
The decision boundary of logistic regression is curved because the sigmoid is curved.
False — the boundary is where , i.e. where , which is a straight line (hyperplane). The sigmoid only bends the probability above that flat boundary, not the boundary itself.
Cross-entropy loss can be exactly zero for a real trained model.
False — zero loss needs for a example, but only as . You can approach zero loss but never reach it with finite weights.
If you double every weight and the bias, the classifications stay the same.
True for the hard labels — scaling by a positive constant does not move the boundary, so which side each point falls on is unchanged; but the probabilities become more extreme (more confident).
Mean squared error would also work fine to train logistic regression.
False in a meaningful sense — MSE with a sigmoid is non-convex (local minima) and its gradient vanishes when the sigmoid saturates, so training stalls on confident-wrong points; cross-entropy is convex and keeps a strong gradient there.
The sigmoid output of means the model is confident the class is exactly halfway.
False — means maximal uncertainty (the model has no information favouring either class), not a "half class." It sits exactly on the decision boundary .
Cross-entropy is convex in the predictions and convex in the raw weights .
True for logistic regression specifically — because composes the convex loss with the sigmoid in a way that keeps the overall objective convex in , guaranteeing a single global minimum.
Spot the error
A student writes the loss for as . What is wrong?
They used the branch. For the loss is ; using rewards , which is backwards.
Someone claims the gradient is . Fix it.
The sign is flipped — the correct gradient is . Their version points uphill, so gradient descent would climb the loss instead of descending it.
A note states "the log-odds equal ." Where's the slip?
The log-odds equal directly (linear); the probability equals . Sigmoid maps log-odds → probability, it does not produce log-odds.
Code computes loss as -y*log(y_hat) - (1-y)*log(1-y_hat) and sometimes returns nan. Why?
When hits exactly or , produces NaN. Real code clips into to avoid this.
A student says "since , the gradient w.r.t. weights also has a factor." True?
False — that factor cancels against the cross-entropy derivative , leaving the clean . The cancellation is the whole reason we pair sigmoid with cross-entropy.
Someone normalizes probabilities across the two classes by dividing by . Pointless?
Yes — already, so the division does nothing. Binary logistic regression is already normalized because is the class-0 probability by construction.
Why questions
Why model the log-odds as linear rather than the probability directly?
Because probability lives in but a linear function ranges over all reals — the log-odds also range over all reals, so linearity there is consistent; the sigmoid is exactly the function that undoes the log-odds.
Why does the negative gradient point toward increasing when ?
Because the gradient is ; if (and ) this is negative, and descent subtracts a negative, so grows — pushing up and toward the true label.
Why do we maximize likelihood but call it minimizing loss?
Cross-entropy is the negative log-likelihood averaged over data; maximizing likelihood and minimizing its negative log are the same optimization, we just flip the sign so gradient descent (a minimizer) applies. See Maximum Likelihood Estimation.
Why does cross-entropy penalize a confident wrong prediction so much more than a hesitant one?
Because as ; being 99% sure of the wrong class incurs near-infinite "surprise," whereas being 55% wrong costs only a little.
Why can't we just use a step function instead of the smooth sigmoid?
A step function has zero derivative almost everywhere and an undefined jump at the boundary, so gradient descent has no slope to follow; the sigmoid gives a smooth, non-zero gradient everywhere.
Why is L2 regularization sometimes essential when classes are perfectly separable?
With perfect separation the weights can grow without bound (pushing toward exactly 0/1 to shrink loss forever), so training never converges; Regularization (L1, L2) adds a penalty that keeps weights finite.
Why does adding more features usually keep the boundary a hyperplane, not a curve?
Because stays linear in the features you feed it; a curved boundary only appears if you engineer nonlinear features (e.g. ) or stack layers as in Neural Networks.
Edge cases
What is when , and what does that mean?
exactly — the model is on the decision boundary with maximal uncertainty, favouring neither class.
As and , what happens to and to the sigmoid's slope?
and respectively, while the slope in both limits — the sigmoid saturates and learning slows for extreme logits.
If all features for an example are zero, what is ?
, so — the prediction depends only on the bias, which encodes the base rate of class 1.
What happens to the loss gradient when ?
It is zero for that weight — a feature that is zero for this example provides no signal to update its own weight, regardless of the prediction error.
A dataset has one class occurring 99% of the time. What does an untrained/lazy logistic model tend to do?
With features uninformative, it learns a large so , predicting the majority class always — high accuracy but useless recall, which is why ROC Curves and AUC matters over raw accuracy.
If two identical feature vectors have opposite labels, can training drive loss to zero?
No — no single can satisfy both a and example, so the model settles at the probability that balances their two losses (near ), and residual loss is irreducible.
What does logistic regression reduce to if there is exactly one class in the training data?
The likelihood is maximized by pushing toward that class for every point, so weights/bias diverge — the problem is degenerate and needs regularization or the setup is ill-posed.
How does binary logistic regression relate to the case of Softmax Regression?
They are equivalent — softmax with two classes collapses to the sigmoid because the two class scores can always be shifted so one is zero, leaving a single logit.
Recall Fast self-test
The gradient's magic simplification comes from pairing which two functions? ::: Sigmoid and cross-entropy — their derivatives cancel to leave . The decision boundary is defined by which equation? ::: , equivalently . Why does perfect separation break plain (unregularized) logistic regression? ::: Weights grow without bound chasing ever-smaller loss, so training never converges.