Noesis
2.2.9 · D5Linear & Logistic Regression

Question bank — Logistic regression and the sigmoid function

1,528 words7 min read↑ Back to topic

Before we start, three words we will lean on constantly, each anchored to a picture in your head:

Two more symbols appear on every line below, so we pin them down now:

True or false — justify

The value can equal exactly or exactly .
False. for every finite , so and lands strictly inside ; it only approaches the ends as .
"Logistic regression" is a regression method that outputs a real number to fit.
False. The name is historical. It fits a probability and then classifies; the only "regression" flavour is that the log-odds are modelled as a linear function.
The decision boundary of logistic regression is an S-shaped curve.
False. The S-curve is plotted against . The boundary lives in feature space at , which is a straight line/hyperplane .
, so the sigmoid is symmetric about the point .
True. Flipping the sign of the score swaps "probability of class 1" with "probability of class 0", and both meet at .
Cross-entropy loss is convex in , so gradient descent always reaches a finite global minimizer.
Mostly true, with one caveat. The log-loss is convex, so there are no bad local minima; but when the data are perfectly linearly separable no finite minimizer exists — the loss keeps decreasing as , so descent diverges rather than settling (see the edge-case item). Regularisation restores a finite optimum.
Squared error would also work fine as a logistic loss, just slower.
False. With inside, MSE becomes non-convex and its gradient vanishes in the tails (where ), so a confidently-wrong prediction barely learns.
Doubling all weights (and ) leaves the decision boundary unchanged.
True. The boundary is the set where ; scaling both sides by 2 gives , the same set. Only the confidence (steepness) changes.
If two examples have the same score , they must have the same features .
False. Many different feature vectors can give the same dot product ; collapses a whole vector to one number.

Spot the error

" means 70% of the features fired."
Wrong. is , the probability of the class, produced by squashing the whole score — it has nothing to do with counting features.
"The gradient is , and is the squared error."
Wrong. is the signed residual (predicted probability minus true label), not a squared quantity; the clean form comes from the cancellation inside cross-entropy.
"To get a curvy boundary, use a steeper sigmoid."
Wrong. Steepness only sharpens confidence around ; the boundary stays linear. Curvy boundaries require nonlinear features like or .
"Since never outputs 0 or 1, cross-entropy is always finite."
Half right, dangerous. In exact arithmetic yes, but in practice can round to , making — hence numerical clipping/log-sum-exp tricks.
"Log-odds means the example is 1.1 times more likely to be class 1."
Wrong. The odds are , so it is more likely. The is the logarithm of the odds, which adds, not multiplies.
", so the boundary is always at ."
Wrong. The boundary is at , i.e. . Because of , that is generally not (e.g. gives boundary ).

Why questions

Why take the log of the odds before setting them equal to a linear score?
Odds live in (one-sided), but a linear score can be anything in ; the log stretches onto the whole line so the two ranges match.
Why is the sigmoid "not arbitrary"?
It is forced: if you assume the log-odds are linear in , then algebraically solving for gives exactly . See Maximum Likelihood Estimation for the training side.
Why does the sigmoid derivative cause a "beautiful cancellation" in the gradient?
The term produced by the chain rule cancels the terms from differentiating and , leaving the tidy residual form — the same shape as Linear Regression.
Why do we minimise the negative log-likelihood instead of maximising the likelihood?
Optimisers (like Gradient Descent) are built to descend toward a minimum; flipping the sign turns "climb the likelihood" into "roll down a cost", and this cost is the Cross-Entropy Loss.
Why can we treat one Bernoulli formula for both labels?
Plugging leaves ; plugging leaves . The exponents act as switches so one line covers both cases of the label.
Why does the sigmoid saturate (flatten) for large , and is that a problem?
In the tails dwarfs or vanishes, so . For confidently-correct points that's fine (little to learn); the danger is only if a wrong prediction saturates — which cross-entropy prevents but MSE does not.

Edge cases

What does logistic regression predict when is exactly ?
— a genuine tie. By convention we assign class 1 when , but the model is truly undecided right on the boundary.
What happens to as and as ?
and respectively, but never reaching them; the model gets arbitrarily confident yet stays inside .
If the two classes are perfectly linearly separable, what happens to the weights during training?
They grow without bound — pushing to make hit or — so the loss keeps shrinking and never reaches a finite minimizer. This is exactly the convexity caveat above, and why regularisation is often needed.
With all weights and bias zero, what is the prediction for any input?
always, so for every point — a neutral "no idea" model, the natural starting state before training.
For a multi-class problem, why can't a single sigmoid handle three or more classes?
A sigmoid outputs one probability for a two-way (yes/no) split. For mutually exclusive classes you need Softmax Regression, which normalises scores so they sum to 1.
If a feature is constant across all examples, what is its trained weight's effect on the boundary?
A constant feature just shifts by a fixed amount for everyone — indistinguishable from the bias — so its weight is unidentifiable and adds no discriminating power.
Is stacking many sigmoids (as in Neural Networks) still a linear-boundary model?
No. A single sigmoid keeps the boundary linear, but composing sigmoid layers builds nonlinear feature transformations, so the effective boundary in the original feature space can bend arbitrarily.
Recall One-line self-test

Cover every answer above and re-derive the reason, not the verdict. If you can say why each is true/false in one breath, you own the concept, not just the fact.

Connections