Multinomial - softmax regression — Hinglish notes

2.2.12 · AI-ML › Linear & Logistic Regression

2 classes se $K$ classes tak logistic regression ko generalize karna.

The Big Picture

Figure — Multinomial - softmax regression

HOW: Softmax ko first principles se derive karna

Step 1 — cheezein positive banao. Exponential $e^{z_{k}}$ kisi bhi real $z_{k}$ ke liye positive hota hai. Yeh step kyun? Logits negative ho sakte hain; probabilities nahi ho sakti. $e^{(\cdot)}$ natural positive, monotone map hai.

Step 2 — unhe sum to 1 banao. Har ek ko total se divide karo: $p_{k} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$ Yeh step kyun? Sum se divide karne par $\sum_{k} p_{k} = 1$ automatically force ho jaata hai — yeh bas normalization hai.

The Loss: Cross-Entropy (Maximum Likelihood se derive ki gayi)

Derivation. True class $y$ wale ek example ke liye, likelihood $p_{y}$ hai. $N$ independent examples par: $L = \prod_{i = 1}^{N} p_{y^{(i)}}^{(i)}$ Negative log-likelihood (NLL) lo: $J = - \frac{1}{N} \sum_{i = 1}^{N} lo g p_{y^{(i)}}^{(i)}$ One-hot label vector $t^{(i)}$ use karte hue (jahan $t_{k} = 1$ agar $k = y^{(i)}$ else 0): $J = - \frac{1}{N} i = 1 \sum N k = 1 \sum K t_{k}^{(i)} lo g p_{k}^{(i)}$ Yeh form kyun? Inner sum sirf true class ka $lo g p$ pick karta hai, kyunki baaki saare $t_{k} = 0$ hain.

The Gradient (woh beautiful result)

Derivation sketch. Key fact: $\frac{\partial p _{k}}{\partial z _{m}} = p_{k} (δ_{k m} - p_{m})$ jahan $δ_{k m} = 1$ agar $k = m$ else 0. Phir ek single example ke liye,

= -\sum_k t_k(\delta_{km} - p_m) = p_m - t_m$$ ($\sum_k t_k = 1$ use karte hue). Kyunki $z_m = \mathbf w_m^\top \mathbf x$: $$\boxed{\;\frac{\partial J}{\partial \mathbf w_m} = (p_m - t_m)\,\mathbf x\;}$$ > [!formula] Gradient descent update > $$\mathbf w_m \leftarrow \mathbf w_m - \eta\,(p_m - t_m)\,\mathbf x$$ --- ## Worked Examples > [!example] Example 1 — softmax compute karna > Logits $\mathbf z = (2,\,1,\,0)$. > - $e^2=7.389,\;e^1=2.718,\;e^0=1$. *Kyun?* Step 1: exponentiate karo. > - Sum $=11.107$. *Kyun?* Step 2: normalizer. > - $p = (0.665,\;0.245,\;0.090)$. *Kyun?* Har ek ko sum se divide karo → probabilities jo 1 mein sum hoti hain. ✅ Sabse bada logit → sabse badi prob. > [!example] Example 2 — shift-invariance check > $\mathbf z = (2,1,0)$ vs $\mathbf z' = (5,4,3)$ lo (saare mein $+3$ add kiya). > - $e^5,e^4,e^3 = 148.4,54.6,20.1$, sum $=223.1$. > - $p' = (0.665,0.245,0.090)$ — Example 1 se **bilkul same**. > - *Yeh step kyun?* Constant $c$ add karne se har $e^{z_k}$ mein $e^c$ multiply hota hai, jo ratio mein cancel ho jaata hai. Isliye hum numerical stability ke liye $\max(z)$ subtract kar sakte hain. > [!example] Example 3 — ek gradient step > Maano true class $y=1$ (one-hot $\mathbf t=(1,0,0)$), predicted $p=(0.665,0.245,0.090)$. > - Error $= p - t = (-0.335,\;0.245,\;0.090)$. *Kyun?* Class 1 ki prob bahut kam hai → negative error $\mathbf w_1$ ko push karta hai taaki $z_1$ badhao. > - $\mathbf x=(1,2)$ ke saath, $\nabla_{\mathbf w_1}J = (-0.335)(1,2) = (-0.335,-0.670)$. > - Update $\mathbf w_1 \leftarrow \mathbf w_1 - \eta(-0.335,-0.670)$ se $\mathbf w_1$ badhta hai → agali baar $z_1$ raise hoga. *Kyun?* Model is input ke liye class 1 par zyada trust karna seekhta hai. ✅ --- ## Common Mistakes (Steel-manned) > [!mistake] "Mujhe $K$ poore independent parameter sets chahiye, saare identifiable." > *Yeh sahi kyun lagta hai:* ek weight vector per class symmetric aur complete lagta hai. > *Fix:* Softmax over-parameterized hai — saare logits mein constant add karne se kuch nahi hota. Effectively $(K-1)$ independent sets hote hain. Aap $\mathbf w_K=0$ fix kar sakte ho (binary logistic ki tarah), lekin practically saare $K$ + regularization rakhna theek aur cleaner hai. > [!mistake] "$e^{z_k}$ seedha compute karo, phir divide karo." > *Yeh sahi kyun lagta hai:* yeh literal formula hai. > *Fix:* Bade logits overflow karte hain ($e^{1000}=\infty$). Hamesha **log-sum-exp trick** use karo: pehle $\max_k z_k$ subtract karo. Shift-invariance se answer same rehta hai lekin numerically safe hota hai. > [!mistake] "Cross-entropy saari classes par sum karti hai, to mujhe har class ka $\log p$ chahiye." > *Yeh sahi kyun lagta hai:* formula mein $\sum_k$ dikha hai. > *Fix:* One-hot $t_k$ true class ke siwa har term ko zero kar deta hai, to effectively $J=-\log p_{\text{true}}$. Compute waste mat karo — seedha index karo. > [!mistake] "Softmax aur sigmoid related nahi hain." > *Fix:* Sigmoid = softmax at $K=2$, logit *difference* use karke. Yeh same family hain. --- ## Active Recall > [!recall]- Softmax outputs ko kaun si teen properties satisfy karni chahiye, aur formula har ek ko kaise guarantee karta hai? > Positive ($e^{z_k}$ se), sum to 1 ($\sum_j e^{z_j}$ se divide karke), logits mein monotone (exp increasing hai). > [!recall]- Softmax + cross-entropy ka gradient simply $(p-t)\mathbf x$ kyun hota hai? > $\log$ derivative ka $\frac{1}{p_k}$, $\partial p_k/\partial z_m$ mein $p_k$ ko cancel kar deta hai, aur $\sum_k t_k = 1$ baaki collapse kar deta hai, sirf $p_m - t_m$ bachta hai. > [!recall]- Exponentiate karne se pehle $\max(z)$ kyun subtract karte hain? > Shift-invariance: probabilities unchanged rehti hain lekin numerical overflow prevent hota hai. > [!recall]- (Feynman, ek 12-saal ke bacche ko explain karo) > Socho 3 contestants ko judges score kar rahe hain (raw scores kuch bhi ho sakte hain, negative bhi). Pehle har score ko "boost" karo — use magic number $e$ ki power mein dalo, taaki saare positive ho jayein aur bade scores bahut bade ho jayein. Phir 100% ki puri pie un sab mein unke boosted score ke hisaab se baanto. Har contestant ko ek slice milti hai, saari slices puri pie mein add hoti hain. Machine ko sikhane ke liye dekho kaun actually jeeta: agar true winner ko bahut chhoti slice mili, to us contestant ke rules nudge karo taaki agali baar zyada score kare. > [!mnemonic] **"EXP-onentiate, then SHARE the pie."** > E-S: **E**xp everything (positive), **S**hare — total se divide karke (sums to 1). Aur gradient bas **"predicted minus true, times input"** hai. --- ## Connections - [[Logistic Regression]] — softmax iska $K$-class generalization hai ($K=2$ ⇒ sigmoid). - [[Cross-Entropy Loss]] — yahan maximum likelihood se derive ki gayi. - [[Maximum Likelihood Estimation]] — loss ke peeche ka principle. - [[Gradient Descent]] — $(p-t)\mathbf x$ update use karta hai. - [[Log-Sum-Exp Trick]] — softmax ke liye numerical stability. - [[Neural Network Output Layer]] — softmax standard multiclass head hai. - [[One-Hot Encoding]] — label representation $\mathbf t$. #flashcards/ai-ml Softmax formula for class k ::: $p_k = e^{z_k} / \sum_j e^{z_j}$ Why exponentiate the logits ::: Outputs ko positive (aur monotone) force karne ke liye, kyunki logits negative ho sakte hain. Why divide by the sum ::: Normalize karne ke liye taaki probabilities sum to 1 hon. Softmax with K=2 reduces to ::: Logit difference ka sigmoid, $\sigma(z_1 - z_2)$. Softmax + cross-entropy gradient wrt logit m ::: $p_m - t_m$. Gradient wrt weight vector w_m ::: $(p_m - t_m)\,\mathbf{x}$. Cross-entropy loss (one-hot) ::: $J = -\frac1N\sum_i \sum_k t_k^{(i)}\log p_k^{(i)}$. Log-sum-exp / stability trick ::: Exponentiating se pehle saare logits se $\max_k z_k$ subtract karo; softmax shift-invariant hai. Why softmax is over-parameterized ::: Har logit mein constant add karne se probabilities nahi badlti, isliye sirf K−1 sets identifiable hain. Derivative identity for softmax ::: $\partial p_k/\partial z_m = p_k(\delta_{km}-p_m)$. ## 🖼️ Concept Map ```mermaid flowchart TD BIN[Binary logistic regression] NEED[Need K probabilities positive and sum to 1] LOGITS[Logits zk = wk dot x] EXP[Exponential makes positive] NORM[Divide by sum normalizes] SOFTMAX[Softmax distribution] SIGMOID[Sigmoid] SHIFT[Shift-invariance] MLE[Maximum Likelihood] CE[Cross-entropy loss] GRAD[Clean gradient p minus t] BIN -->|generalize to K classes| NEED NEED -->|requires| LOGITS LOGITS -->|Step 1| EXP EXP -->|Step 2| NORM NORM -->|yields| SOFTMAX SOFTMAX -->|K=2 reduces to| SIGMOID SOFTMAX -->|adding c changes nothing| SHIFT MLE -->|neg log-likelihood| CE SOFTMAX -->|plugged into| CE CE -->|differentiate| GRAD ```