Exercises — Multinomial - softmax regression

The denominator is the total of every exponentiated logit — one shared number for all classes (the size of the whole pie in the third box of figure s01). Because every $p_k$ is divided by that same total, ${\sum }_k{p}_k=\frac{{\sum }_k{e}^{z_k}}{{\sum }_j{e}^{z_j}}=1.$ The numerator sum and denominator sum are identical, so they cancel to exactly $1$. This is why softmax outputs form a valid probability distribution: they always add to the whole pie.

Exponentiate: $e^1=e^1=e^1$, three identical numbers. Divide each by their sum: each class gets $\frac{e^1}{3e^1}=\frac{1}{3}$. $p=\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)\approx (0.333,0.333,0.333).$ Meaning: when all scores tie, softmax says "I have no idea" — perfectly uniform. This is the maximum-uncertainty case, matching the flat, equal-height panel of figure s02.

Exponentiate using $e^{\ln a}=a$ (box 2): $e^0=1,e^{\ln 3}=3,e^{\ln 6}=6.$ Sum $=1+3+6=10$ (box 3 normalizer). Divide: $p=\left(\frac{1}{10},\frac{3}{10},\frac{6}{10}\right)=(0.1,0.3,0.6).$ Notice the logs were chosen so the pie splits into clean tenths. Highest logit ($\ln 6$) → highest probability ($0.6$). ✅

One-hot zeroes every term except the true class $k=3$ (only $t_3=1$): $J=-(0\cdot \log 0.1+0\cdot \log 0.3+1\cdot \log 0.6)=-\log \mathrm{0.6.}$ $J=-\ln (0.6)\approx \mathrm{0.5108.}$ Meaning: loss depends only on the probability you gave the correct class. Give it $0.6$ → pay $0.51$. Had the model given class 3 a probability near $1$, the loss would fall toward $0$.

For $m=3$: $p_3-t_3=0.6-1=-0.4$. ${\nabla }_{{\mathbf{w}}_3}J=(-0.4)(2,-1)=(-0.8,0.4).$ The gradient is negative-scaled because the true class was under-predicted ($p_3<1$). Gradient descent subtracts this, so ${\mathbf{w}}_3$ moves in the $+(0.8,-0.4)$ direction — raising $z_3$ next time. The model learns to trust class 3 more for this input. ✅

Start from the definition with shifted logits $z_k+c$: $\frac{e^{z_k+c}}{{\sum }_j{e}^{z_j+c}}=\frac{e^{z_k}{e}^c}{{\sum }_j{e}^{z_j}{e}^c}=\frac{e^c{e}^{z_k}}{e^c{\sum }_j{e}^{z_j}}.$ The factor $e^c$ appears in every numerator and in the denominator, so it factors out and cancels: $=\frac{e^{z_k}}{{\sum }_j{e}^{z_j}}=\text{softmax}(z_k).■$ This is the whole justification for the log-sum-exp trick: choose $c=-{\max }_k{z}_k$ so the largest exponent becomes $0$ (and $e^0=1$, never overflowing) — the probabilities are provably identical.

Divide numerator and denominator by $e^{z_1}$ (legal since $e^{z_1}\ne 0$): $p_1=\frac{e^{z_1}/e^{z_1}}{e^{z_1}/e^{z_1}+e^{z_2}/e^{z_1}}=\frac{1}{1+e^{z_2-z_1}}=\frac{1}{1+e^{-(z_1-z_2)}}=\sigma (z_1-z_2).■$ Meaning: with two classes only the difference $z_1-z_2$ matters — one extra degree of freedom (the common part) is invisible. That is exactly the shift-invariance of L3.1 showing up as Logistic Regression's single logit.

Part (a) — deriving the softmax Jacobian. Write $p_k=\frac{e^{z_k}}{S}$ with normalizer $S={\sum }_j{e}^{z_j}$. We differentiate with respect to one logit $z_m$ using the quotient rule; note $\frac{\partial S}{\partial z_m}=e^{z_m}$ (only the $j=m$ term of $S$ depends on $z_m$), and $\frac{\partial e^{z_k}}{\partial z_m}=e^{z_k}{\delta }_{km}$ (the top moves only when $k=m$ — this is exactly what ${\delta }_{km}$ encodes). $\frac{\partial p_k}{\partial z_m}=\frac{({\partial }_{z_m}{e}^{z_k})S-e^{z_k}({\partial }_{z_m}S)}{S^2}=\frac{e^{z_k}{\delta }_{km}S-e^{z_k}{e}^{z_m}}{S^2}.$ Split into two fractions and recognize $\frac{e^{z_k}}{S}=p_k$ and $\frac{e^{z_m}}{S}=p_m$: $=\frac{e^{z_k}}{S}{\delta }_{km}-\frac{e^{z_k}}{S}\cdot \frac{e^{z_m}}{S}=p_k{\delta }_{km}-p_k{p}_m=p_k({\delta }_{km}-p_m).✓$ Intuition: raising $z_m$ grows its own probability (the ${\delta }_{km}$ "self" term, present only when $k=m$) but simultaneously steals from every class by inflating the shared normalizer $S$ (the $-p_k{p}_m$ "competition" term). Softmax outputs are coupled — one bar rising pushes the others down (exactly what figure s02 shows).

Part (b) — the cancellation. Differentiate $J$; the chain rule on $\log p_k$ gives $\frac{1}{p_k}\frac{\partial p_k}{\partial z_m}$: $\frac{\partial J}{\partial z_m}=-{\sum }_k{t}_k\frac{1}{p_k}\frac{\partial p_k}{\partial z_m}=-{\sum }_k{t}_k\frac{1}{p_k}{p}_k({\delta }_{km}-p_m).$ The $\frac{1}{p_k}$ cancels the $p_k$ — the beautiful cancellation: $=-{\sum }_k{t}_k({\delta }_{km}-p_m)=-{\sum }_k{t}_k{\delta }_{km}+p_m{\sum }_k{t}_k.$ Now ${\sum }_k{t}_k{\delta }_{km}=t_m$ (the delta picks out only the $k=m$ term) and ${\sum }_k{t}_k=1$ (one-hot): $=-t_m+p_m\cdot 1=p_m-t_m.■$ Every messy exp/normalization term vanished, leaving prediction minus truth.

Part (c) — chain rule from logit to weights. The loss depends on ${\mathbf{w}}_m$ only through the logit $z_m={\mathbf{w}}_m^{\top }\mathbf{x}$. So by the chain rule, the derivative with respect to one weight component $w_{m,i}$ is $\frac{\partial J}{\partial w_{m,i}}=\frac{\partial J}{\partial z_m}\cdot \frac{\partial z_m}{\partial w_{m,i}}.$ The first factor is $p_m-t_m$ from Part (b). For the second, since $z_m={\sum }_i{w}_{m,i}{x}_i$, differentiating w.r.t. one weight $w_{m,i}$ leaves only its own input: $\frac{\partial z_m}{\partial w_{m,i}}=x_i$. Therefore $\frac{\partial J}{\partial w_{m,i}}=(p_m-t_m)x_i.$ Stacking over all components $i$ into a vector is exactly ${\nabla }_{{\mathbf{w}}_m}J=(p_m-t_m)\mathbf{x}.■$ This is the rule we simply used in L2.3 — now derived: the scalar error $(p_m-t_m)$ scales the input direction $\mathbf{x}$.

${\max }_k{z}_k=12$. Subtract it from each (shift-invariance guarantees the answer is unchanged): ${\mathbf{z}}^{\prime }=(0,-2,-1).$ Exponentiate the safe, small numbers: $e^0=1,e^{-2}=0.13534,e^{-1}=\mathrm{0.36788.}$ Sum $=1+0.13534+0.36788=1.50322$. Divide: $p=\left(\frac{1}{1.50322},\frac{0.13534}{1.50322},\frac{0.36788}{1.50322}\right)\approx (0.6652,0.0900,0.2447).$ Without the trick you would have computed $e^{12}\approx 162754$ — fine here, but the pattern is identical, and for logits like $1000$ it prevents inf. Highest logit → highest prob. ✅

(a) One-hot picks class 2: $J=-\log p_2=-\ln (0.0900)\approx \mathrm{2.4079.}$ High loss — the model badly under-rated the true class ($0.09$). (b) Error for $m=2$: $p_2-t_2=0.0900-1=-0.9100$. ${\nabla }_{{\mathbf{w}}_2}J=(-0.9100)(1,0)=(-0.9100,0).$ Large negative error → a big corrective step: descent subtracts this, pushing ${\mathbf{w}}_2$ up strongly to raise $z_2$. The second component is $0$ because $x_2=0$ carries no information to update. ✅

Compute the new logits after subtracting $\mathbf{a}$ from every weight vector: $z_k^{\prime }=({\mathbf{w}}_k-\mathbf{a}{)}^{\top }\mathbf{x}={\mathbf{w}}_k^{\top }\mathbf{x}-{\mathbf{a}}^{\top }\mathbf{x}=z_k-c,$ where $c={\mathbf{a}}^{\top }\mathbf{x}$ is the same scalar for every class (it depends on $\mathbf{a}$ and $\mathbf{x}$, not on $k$). So all logits drop by the identical amount $c$. By shift-invariance (Exercise L3.1), subtracting a common constant $c$ from all logits leaves softmax unchanged: $\text{softmax}(z_k^{\prime })=\text{softmax}(z_k-c)=\text{softmax}(z_k).$ Conclusion: the model is over-parameterized — an entire direction of weight-space (subtract any $\mathbf{a}$) produces zero change in predictions. You can therefore pin ${\mathbf{w}}_K=0$ without losing any expressive power, giving effectively $K-1$ independent weight vectors. This matches Logistic Regression ($K=2$), which needs just one logit/weight vector, not two. ✅

(a) Softmax: $e^3=20.0855$, $e^1=2.71828$, sum $=22.8038$. $p_1=\frac{20.0855}{22.8038}\approx \mathrm{0.8808.}$ (b) Sigmoid of difference: $z_1-z_2=3-1=2$. $\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.13534}=\frac{1}{1.13534}\approx \mathrm{0.8808.}$ They agree to four decimals. ✅ This is the Neural Network Output Layer fact: a 2-way softmax head is a sigmoid unit on the logit gap — the Logistic Regression special case, using Cross-Entropy Loss as its objective, itself derived by Maximum Likelihood Estimation.