Foundations — Multinomial - softmax regression

Before you can read the parent note, you need to see every symbol it throws at you. This page builds each one from nothing — plain words, then a picture, then why the topic can't live without it. Read top to bottom: each idea rests on the one above it.

This is the foundations page for Multinomial / Softmax Regression.

1. The input vector $\mathbf{x}$ and its dimension $d$

The picture. Think of $d=2$: then $\mathbf{x}=(x_1,x_2)$ is a single dot on a flat sheet of paper, at horizontal position $x_1$ and vertical position $x_2$. For $d=3$ it's a dot in a room. For bigger $d$ we can't draw it, but it's still "one point in a $d$-dimensional space."

Why the topic needs it. Everything downstream — the scores, the probabilities — is a function of this one point. No input, nothing to classify.

2. The class label $y$ and the count $K$

The picture. Imagine $K=3$ labelled bins standing in a row: bin 1, bin 2, bin 3. Every example belongs in exactly one bin. $y$ is the bin's number.

Why the topic needs it. Softmax's whole job is to guess which of the $K$ bins the point $\mathbf{x}$ belongs to, and to grade itself against the true $y$.

3. Weight vectors ${\mathbf{w}}_k$ — one ruler per class

The picture. ${\mathbf{w}}_k$ is an arrow starting at the origin, pointing in the direction that class $k$ considers "very much like me." A point $\mathbf{x}$ lying far along that arrow's direction scores high for class $k$.

Why the topic needs it. These arrows are the only thing the model learns. Training = adjusting the ${\mathbf{w}}_k$ arrows so each points toward its class's examples.

4. The dot product ${\mathbf{w}}_k^{\top }\mathbf{x}$ — measuring alignment

The picture. Lay both arrows down. The dot product measures how much of $\mathbf{x}$ "shadows onto" the direction of $\mathbf{w}$ (its projection), scaled by how long $\mathbf{w}$ is. Same direction → large positive shadow. Right angle → zero. Opposite → negative.

5. The logit $z_k$ — a raw, unbounded score

The picture. Picture $K$ contestants, each holding up a scorecard with a number on it. Some numbers are negative (like $-4$), some big (like $12$). These are the $z_k$. They're not yet fair, comparable slices of a pie — just raw enthusiasm.

Why the topic needs it. The logits are the bridge: the learned arrows produce them, and softmax will convert them into honest probabilities. The word "logit" will appear everywhere in the parent note.

6. Euler's number $e$ and the exponential $e^z$

The picture. A curve that hugs the floor on the left (near $0$ but positive), passes through height $1$ at $z=0$, and rockets upward on the right.

7. Summation $\sum$ — adding a list up

The picture. A conveyor belt: it visits each item once, dropping it into a running total. The $j$ is the belt's counter; it's a dummy — you could rename it $m$ and nothing changes.

Why the topic needs it. The softmax denominator ${\sum }_{j=1}^K{e}^{z_j}$ is "the total size of all boosted scores" — the whole pie we're about to divide.

8. Probability $p_k$ and normalization — sharing the pie

The picture. A pie chart. Each contestant's boosted score $e^{z_k}$ claims a wedge; dividing by the total is just measuring each wedge as a fraction of the full circle. All wedges together = one whole pie = $100\%$.

9. Numerical safety — subtracting ${\max }_k{z}_k$ before exponentiating

Why it works. Adding the same constant $c$ to every logit multiplies every $e^{z_k}$ by $e^c$, which cancels in the ratio — softmax is shift-invariant. Choosing $c=-m$ pins the largest term at $1$. The parent note calls this the Log-Sum-Exp Trick.

10. The one-hot vector $\mathbf{t}$

The picture. $K$ light switches, all OFF except the one for the true class, which is ON.

Why the topic needs it. It lets the loss (next section) automatically "pick out" only the true class — every wrong-class term is multiplied by $0$ and vanishes.

11. The natural logarithm $\log (\cdot )$

The picture. A mirror image of the exponential curve flipped across the diagonal line $y=x$. On the interval we care about, $0<a\le 1$: the curve sits at or below zero. It equals exactly $0$ when $a=1$, and dives down toward $-\infty$ as $a$ approaches $0$.

12. The one-hot vector $\mathbf{t}$ and cross-entropy loss $L(\mathbf{p},\mathbf{t})$ — grading the guess

The one-hot shortcut. Because $\mathbf{t}$ is zero everywhere except the true slot, the whole sum collapses to a single term: $L=-\log p_{\text{true}}$ Why? Every wrong-class term is $t_k\cdot (\dots )=0\cdot (\dots )=0$. So cross-entropy is literally "the negative log of the slice the true class received." Full treatment: Cross-Entropy Loss, derived from Maximum Likelihood Estimation.

13. The partial derivative $\partial$ and the gradient $(p_m-t_m)$

The picture. Imagine $L$ as a hilly landscape whose height depends on all the logits. $\partial L/\partial z_m$ is the steepness you'd feel if you walked one step purely along the $z_m$ axis, freezing all others. Downhill in that direction lowers the loss.

Building the gradient, one step at a time. We want $\partial L/\partial z_m$ where $L=-{\sum }_k{t}_k\log p_k$.

Step A — slope of softmax itself. WHAT: first find how each probability $p_k$ moves when we wiggle logit $z_m$. WHY: the loss is built from the $p_k$, so by the chain rule we must know their slopes first. Result (a standard exercise you can accept for now): $\frac{\partial p_k}{\partial z_m}=p_k({\delta }_{km}-p_m)$ Read it: if $k=m$ the term ${\delta }_{km}=1$ makes it $p_m(1-p_m)>0$ (raising a logit raises its own probability); if $k\ne m$ it is $-p_k{p}_m<0$ (raising one logit steals probability from the others). That sign pattern is exactly the "pie must still sum to 1" behaviour.

Step B — slope of $\log p_k$. WHAT: the loss uses $\log p_k$, not $p_k$. WHY: we need the derivative of the log so we can chain it onto Step A. The derivative of $\log a$ is $1/a$, so $\frac{\partial \log p_k}{\partial z_m}=\frac{1}{p_k}\frac{\partial p_k}{\partial z_m}=\frac{1}{p_k}{p}_k({\delta }_{km}-p_m)={\delta }_{km}-p_m.$ WHAT IT LOOKS LIKE: the $1/p_k$ and the $p_k$ cancel — the messy division disappears. This cancellation is the whole reason softmax and log are paired.

Step C — assemble the loss slope. WHAT: put Step B into $L=-{\sum }_k{t}_k\log p_k$. WHY: that is the quantity we actually want to descend. $\frac{\partial L}{\partial z_m}=-{\sum }_k{t}_k({\delta }_{km}-p_m)=-(\underset{=t_m}{\underbrace{\sum _k{t}_k{\delta }_{km}}}-p_m\underset{=1}{\underbrace{\sum _k{t}_k}})=p_m-t_m.$ WHY each collapse: ${\sum }_k{t}_k{\delta }_{km}=t_m$ because ${\delta }_{km}$ keeps only the $k=m$ term; and ${\sum }_k{t}_k=1$ because a one-hot vector has exactly one $1$. What remains is astonishingly clean: $\boxed{\frac{\partial L}{\partial z_m}=p_m-t_m}$

Step D — from logits to weights. WHAT: we ultimately steer the weights ${\mathbf{w}}_m$, not the logits. WHY: since $z_m={\mathbf{w}}_m^{\top }\mathbf{x}$, wiggling ${\mathbf{w}}_m$ changes $z_m$ at a rate of $\mathbf{x}$, so we multiply: $\frac{\partial L}{\partial {\mathbf{w}}_m}=(p_m-t_m)\mathbf{x}$

The picture in words: "predicted minus true, times input." If the true class's slice is too small, $p_m-t_m$ is negative there, and the update pushes ${\mathbf{w}}_m$ to raise that class's logit next time.

14. The prerequisite map

Read it as flow: the input and per-class weights make logits; exp plus the sum (made safe by subtracting the max) make probabilities; probabilities plus the natural log and the one-hot truth make the loss; and the loss's partial derivative — helped by the Kronecker delta — is the clean $(p-t)\mathbf{x}$ that training uses.

Equipment checklist

Test yourself — hide the right side and answer aloud.