Multinomial - softmax regression: Question bank

Before we start, a short glossary so every symbol below is already earned:

Definition Symbols used on this page

Number of classes $K$ — how many categories we are choosing between (e.g. $K = 10$ for digits $0$ – $9$ ). Class labels are indexed by $k$ , $j$ , or $m$ , each running from $1$ to $K$ ; they are just different letters for "some class", used when we need to talk about two class positions at once.
Input vector $x$ — the list of feature numbers describing one example (e.g. the pixels of an image). It is what we feed in; the model never changes it.
Logit $z_{k} = w_{k}^{⊤} x$ — the raw, unbounded score the model gives class $k$ . It is the dot product of class $k$ 's weight vector $w_{k}$ with the input $x$ , and can be any real number, positive or negative.
Softmax $p_{k} = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$ — turns the $K$ logits into $K$ probabilities that are positive and add to $1$ .
One-hot label $t$ — a vector that is $1$ at the true class and $0$ everywhere else (see One-Hot Encoding).
Cross-entropy loss $J = - \sum_{k = 1}^{K} t_{k} lo g p_{k}$ — the training loss for one example; because $t$ is one-hot, only the true class's term survives, so $J = - lo g p_{true}$ . See Cross-Entropy Loss.
Sigmoid $σ (u) = \frac{1}{1 + e ^{- u}}$ — the classic "S-curve" from Logistic Regression; it squashes any real number $u$ into a single probability between $0$ and $1$ .
$η$ (eta) — the learning rate, a small positive step size in Gradient Descent.

The single figure below fixes two pictures in your head — the softmax "pie" and the sigmoid S-curve — so the traps below have something visual to lean on.

Figure — Multinomial - softmax regression

Look at the left panel: three logits become three positive bars (amber), and softmax just re-slices them into a pie summing to $1$ (cyan). Shifting all three logits up shifts all bars equally — the slices never change. The right panel is the sigmoid $σ$ : notice it is exactly the $K = 2$ softmax plotted against the logit difference $z_{1} - z_{2}$ .

True or false — justify

Adding the same constant

c

to every logit changes the predicted probabilities.

False. Every

e^{z_{k}}

gets multiplied by

e^{c}

, which cancels in the ratio; softmax is shift-invariant, so only differences of logits matter.

Multiplying every logit by a constant

c > 1

leaves the probabilities unchanged.

False. Scaling is not shifting — it stretches the differences

z_{i} - z_{j}

, sharpening the distribution toward the top class (this is the "temperature" effect).

For

K = 2

, softmax is a genuinely different model from binary logistic regression.

False.

p_{1} = σ (z_{1} - z_{2})

, exactly the sigmoid (

σ

, the S-curve above) of the logit difference, so it is the same model — see Logistic Regression.

Each of the

K

weight vectors

w_{k}

is individually identifiable from data.

False. Because of shift-invariance you can add any constant vector to all of them without changing predictions; only

K - 1

vectors' worth of parameters are identifiable.

If two classes have equal logits, they receive equal softmax probabilities.

True. Equal

z

gives equal

e^{z}

, so their normalized shares are identical — softmax is symmetric in classes with equal scores.

The softmax output can equal exactly

0

for some class.

False.

e^{z_{k}} > 0

for every finite

z_{k}

, so every probability is strictly positive; a class only approaches

0

as its logit goes to

- \infty

The cross-entropy loss

J

reads the predicted probability of the wrong classes directly from its formula.

False. With a one-hot label every wrong-class term has

t_{k} = 0

, so

J = - lo g p_{true}

uses only the true class's probability; the wrong classes influence

J

only indirectly, through the shared normalizer inside

p_{true}

The gradient component

p_{m} - t_{m}

can only be negative for the true class.

True. For a wrong class

t_{m} = 0

, so

p_{m} - t_{m} = p_{m} \geq 0

(never negative); only the true class has

t_{m} = 1

, making

p_{m} - t_{m} = p_{true} - 1 < 0

whenever the true probability is below

1

At the perfect prediction

p = t

, the gradient with respect to every logit is zero.

True.

p_{m} - t_{m} = 0

for all

m

, so no weight moves — the loss is at its minimum for that example.

Spot the error

"To avoid overflow, subtract

min_{k} z_{k}

before exponentiating."

Error. You subtract the max, not the min; subtracting the max makes the largest exponent

e^{0} = 1

and all others

\leq 1

, which is what prevents overflow (see Log-Sum-Exp Trick).

"Cross-entropy loss

J = - \sum_{k} t_{k} lo g p_{k}

needs me to compute

lo g p_{k}

for all

K

classes."

Error. All terms with

t_{k} = 0

vanish, so only

- lo g p_{true}

survives; computing the rest is wasted work.

"Since probabilities are positive, I can safely take

lo g p_{k}

without worry."

Error. Naively computed

p_{k}

can underflow to exactly

0.0

, making

lo g 0 = - \infty

; you must compute

lo g p_{k} = z_{k} - logsumexp (z)

in log-space.

"The gradient is

(t_{m} - p_{m}) x

, so we step

w_{m} \leftarrow w_{m} - η (t_{m} - p_{m}) x

Error (sign). The loss gradient is

(p_{m} - t_{m}) x

; the update subtracts that, giving

w_{m} \leftarrow w_{m} - η (p_{m} - t_{m}) x

. Flipping the sign does gradient ascent and diverges.

"Softmax with

K

classes has

K

free scalar outputs, so it needs

K

degrees of freedom."

Error. The sum-to-one constraint removes one degree, leaving

K - 1

genuinely free outputs — the same count binary logistic regression uses (

1

for

K = 2

"For numerical stability I subtract

max z

and renormalize by a hand-picked constant to be safe."

Error. Subtracting the max is exact (shift-invariance guarantees identical answers); an extra hand-picked constant is either redundant or, if multiplicative, corrupts the probabilities.

Why questions

Why does exponentiation, rather than "shift up and clip to positive", get used to make logits positive?

Because

e^{(\cdot)}

is smooth, strictly increasing, and never zero, giving well-defined gradients everywhere and preserving the logit ordering; clipping introduces kinks and dead zones with zero gradient.

Why does dividing by

\sum_{j} e^{z_{j}}

(and nothing else) guarantee the outputs sum to

1

Because summing the outputs gives

\frac{\sum _{k} e ^{z_{k}}}{\sum _{j} e ^{z_{j}}} = 1

identically — the numerator sum equals the denominator by construction.

Why is the softmax+cross-entropy gradient so clean at

p_{m} - t_{m}

The

1/ p_{k}

from differentiating

lo g

cancels the

p_{k}

\partial p_{k} / \partial z_{m}

, and

\sum_{k} t_{k} = 1

collapses the rest — the exp/normalizer mess disappears exactly (see the pocket derivation above).

Why do we derive the loss from Maximum Likelihood instead of just guessing a loss?

Maximum Likelihood Estimation says pick weights that make the observed labels most probable; taking

- lo g

of that product turns it into a sum we can minimize, which is precisely Cross-Entropy Loss.

Why can we set

w_{K} = 0

without losing any expressive power?

Shift-invariance means only logit differences matter, so we can measure all scores relative to class

K

; this is exactly what binary logistic regression does with its single weight vector.

Why does a bigger error on a wrong-but-confident prediction push the input

x

harder into the weights?

The gradient is

(p_{m} - t_{m}) x

, so the error

p_{m} - t_{m}

is largest exactly when the prediction is most wrong, and it multiplies

x

— big mistakes move the weights more, correct ones (

p_{m} = t_{m}

) not at all.

Edge cases

What does softmax output when all

K

logits are equal (including all zero)?

A perfectly uniform distribution

p_{k} = 1/ K

for every class — no logit stands out, so the pie is split evenly.

What happens to

p_{k}

as one logit

z_{k} \to + \infty

while others stay fixed?

p_{k} \to 1

and all others

\to 0

; softmax becomes a hard "argmax" in this limit (which is why it's a soft max).

What is the cross-entropy loss when the model is perfectly right (

p_{true} = 1

J = - lo g 1 = 0

; a perfectly confident correct prediction contributes zero loss.

What is the loss when the model gives the true class probability

\to 0

J = - lo g p_{true} \to + \infty

; confidently wrong is punished without bound, which is why log-space stability matters.

Is softmax defined for

K = 1

(a single class)?

Yes but trivially:

p_{1} = e^{z_{1}} / e^{z_{1}} = 1

always, so a one-class problem carries no information and no learnable signal.

If two logits are both enormous and nearly equal (say

1000

and

1000.1

), what breaks naively and what fixes it?

Direct

e^{1000}

overflows to

\infty

; subtracting the max first (logsumexp) gives

e^{0}

and

e^{0.1}

, recovering a correct near-even split — used by every Neural Network Output Layer.

If the training label is not one-hot but a soft distribution (e.g.

(0.9, 0.1, 0)

), does cross-entropy still work?

Yes; the general form

- \sum_{k} t_{k} lo g p_{k}

handles any target distribution — one-hot is just the special crisp case where one

t_{k} = 1

Connections

Logistic Regression — the $K = 2$ special case that these traps repeatedly reference.
Cross-Entropy Loss — the loss whose "only the true term survives" trap lives here.
Maximum Likelihood Estimation — the justification behind several "why" answers.
Gradient Descent — the sign-flip trap targets its update rule.
Log-Sum-Exp Trick — the max-vs-min and overflow edge cases.
Neural Network Output Layer — where these stability edge cases bite in practice.
One-Hot Encoding — the label form assumed in the loss traps.