Softmax for output layers — Hinglish notes

3.1.6 · AI-ML › Neural Network Fundamentals

Softmax KYA hai?

YEH TEEN design choices KYUN hain?

Exponentiate KYUN karte hain? Humein har output positive chahiye. Raw logits negative ho sakte hain; $e^{z_{i}}$ hamesha positive hota hai, isliye koi bhi probability kabhi negative nahi hogi. Exponentiation gaps ko amplify bhi karta hai — thoda bada logit bahut badi probability ban jaata hai, jo ek confident-lekin-differentiable "winner" deta hai.

Sum se divide KYUN karte hain? Normalization. Exponentiate karne ke baad hamare paas positive numbers hote hain, lekin unka sum 1 nahi hota. Unke total se divide karna ek valid distribution force karta hai — jaisa ki ek pie ko is tarah share karna ki saare slices poori pie jitne ho jaayein.

Raw logits ko seedha normalize KYUN nahi karte (yaani $z_{i} / \sum z_{j}$ )? Kyunki raw logits negative ho sakte hain, jisse negative "probabilities" milti hain, aur sum zero ho sakta hai (division blow up ho jaata hai). Exponential dono problems fix kar deta hai.

First principles se HOW derive karein

Hum ek function $z \to p$ chahte hain in demands ke saath:

$p_{i} \geq 0$
$\sum_{i} p_{i} = 1$
Bada $z_{i} \Rightarrow$ bada $p_{i}$ (monotonic)
Har jagah differentiable (taaki hum gradient descent se train kar sakein)

Step 1 — positivity force karo. Ek positive, increasing function $f$ lagaao. Exponential $f (z) = e^{z}$ canonical choice hai (1, 3, 4 satisfy karta hai).

Specifically exp KYUN? Yeh naturally maximum-entropy / Gibbs distribution se aata hai: agar aap assume karo ki log-probability score mein linear hai, yaani $lo g p_{i} \propto z_{i}$ , toh $p_{i} \propto e^{z_{i}}$ . Yeh given expected scores ke saath consistent least-biased distribution hai.

Step 2 — sum-to-1 enforce karo. Haare paas unnormalized weights $e^{z_{i}}$ hain. Unke total se divide karo: $p_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$ Ab $\sum_{i} p_{i} = \frac{\sum _{i} e ^{z_{i}}}{\sum _{j} e ^{z_{j}}} = 1$ . ✔️ Ho gaya — yahi softmax hai.

Numerical stability trick (safely compute karne ka HOW)

Yeh safe KYUN hai: ratio unchanged rehta hai, isliye predictions identical hain — humne sirf floating-point overflow fix kiya.

Gradient (KYUN softmax + cross-entropy magic hai)

Softmax ko cross-entropy loss ke saath lagbhag hamesha pair kiya jaata hai. True class $y$ (one-hot $t_{i}$ ) ke liye, loss $L = - \sum_{i} t_{i} lo g p_{i}$ .

Yeh beautiful result hai — derive karo:

$\frac{\partial p _{i}}{\partial z _{k}} = p_{i} (δ_{ik} - p_{k})$ jahan $δ_{ik} = 1$ agar $i = k$ ho warna $0$ . Cross-entropy ke saath combine karne par, logits ke w.r.t. gradient reduce ho jaata hai: $\frac{\partial L}{\partial z _{k}} = p_{k} - t_{k}$

Yeh matter KYUN karta hai: gradient sirf (prediction − target) hai. Clean, cheap, softmax aur log ke cancel hone se koi vanishing pieces nahi. Yahi woh reason hai ki inhe saath use kiya jaata hai.

Common mistake Steel-manned common errors

Mistake 1: "Softmax outputs confidences/certainties hain." Kyun sahi lagta hai: ye 1 tak sum hote hain aur probabilities jaisa dikhta hai. Fix: ye relative scores hain, easily overconfident aur uncalibrated. Ek galat prediction tab bhi $p = 0.99$ dikha sakta hai. Softmax ek distribution deta hai, calibrated truth nahi.

Mistake 2: "Multi-label problems ke liye softmax use karo." Kyun sahi lagta hai: yeh go-to output layer hai. Fix: softmax classes ko compete karne par force karta hai (sum to 1). Agar ek image "beach" AUR "sunny" dono ho sakti hai, toh har class ke liye independent sigmoids use karo.

Mistake 3: "Softmax phir se softmax," ya already-normalized values par do baar lagaana. Kyun sahi lagta hai: harmless lagta hai. Fix: softmax do baar lagaane se distribution uniform ki taraf flatten ho jaati hai aur information destroy ho jaati hai. Ise bilkul ek baar, raw logits par lagaao.

Mistake 4: max-subtraction bhool jaana aur NaN/inf paana. Kyun sahi lagta hai: math formula mein koi subtraction nahi hai. Fix: code mein hamesha $max z$ subtract karo — mathematically identical, numerically safe.

Recall Feynman: ek 12-saal ke bachche ko samjhao

Tumhare teen dost vote kar rahe hain ki kahan khaana khaayein, aur har ek apna "loudness score" chilla ke bolta hai. Softmax un loudness scores ko ek pizza ke slices mein convert karta hai. Zyada zor se cheekha ⇒ bada slice, lekin har slice positive hai aur saare slices milke ek poora pizza banate hain. Sabse zyada cheekhaane wale ko sabse zyada milta hai, lekin sabko kuch na kuch milta hai. Isi tarah computer kehta hai "mujhe 66% lagta hai yeh cat hai, 24% dog, 10% bird" — saare milake ek poore "sure" jitne hain.

Active Recall

Flashcards

Softmax logits ko kya convert karta hai?

Ek valid probability distribution mein — values

(0, 1)

mein hoti hain jo 1 tak sum hoti hain.

Class

i

ke liye softmax formula likho.

p_{i} = e^{z_{i}} / \sum_{j} e^{z_{j}}

Logits ko exponentiate KYUN karte hain?

e^{z}

hamesha positive hota hai (koi negative probabilities nahi) aur scores ke beech gaps ko amplify karta hai, saath hi differentiable bhi rehta hai.

Exponentials ke sum se divide KYUN karte hain?

Normalize karne ke liye taaki saare outputs 1 tak sum ho sakein, ek proper distribution banate hue.

Softmax ki shift-invariance property kya hai?

Saare logits mein ek constant

c

add karne se outputs unchanged rehte hain:

softmax (z + c) = softmax (z)

Overflow rokne ka numerical trick kya hai?

Exponentiate karne se pehle har logit se

max_{k} z_{k}

subtract karo.

Cross-entropy+softmax ka logits ke w.r.t. gradient kya hai?

\partial L / \partial z_{k} = p_{k} - t_{k}

(prediction minus target).

Softmax vs sigmoid output layer — kab kaunsa?

Mutually-exclusive single-label classification ke liye softmax; multi-label ke liye independent sigmoids (classes compete nahi karte).

Derivative

\partial p_{i} / \partial z_{k}

kya hai?

p_{i} (δ_{ik} - p_{k})

Softmax do baar lagaana galat KYUN hai?

Yeh distribution ko uniform ki taraf flatten kar deta hai aur information discard kar deta hai; ek baar raw logits par lagaao.

Connections

Cross-Entropy Loss — softmax ka natural partner; saath milke gradient $p - t$ dete hain.
Sigmoid Activation — softmax ka 2-class / multi-label sibling.
Logits and Log-Odds — woh raw inputs jo softmax consume karta hai.
Maximum Entropy Distributions — exponential form ka theoretical justification.
Backpropagation — jahan clean $p - t$ gradient flow karta hai.
Temperature Scaling — logits ko $T$ se divide karna softmax ko sharpen/soften karne ke liye (calibration).