Weight initialization (Xavier, He) — Hinglish notes

3.1.12 · AI-ML › Neural Network Fundamentals

Problem KYA hai?

KYU care karein? Ek deep net layer by layer compute karta hai: $z^{[l]} = W^{[l]} a^{[l - 1]}, a^{[l]} = ϕ (z^{[l]})$ Agar $W$ mein numbers average par bahut bade hain, toh har layer signal ko upar multiply karti hai → activations blow up ho jaate hain (exploding). Bahut chhote hain → activations $0$ par shrink ho jaate hain (vanishing). Dono cases mein gradients useless ho jaate hain aur learning ruk jaati hai.

Sahi scale derive kaise karein (first principles)

Hum ek single pre-activation $z = \sum_{i = 1}^{n_{in}} w_{i} x_{i}$ ki variance track karte hain, jahaan $n_{in}$ = inputs ki sankhya (fan-in) hai.

Assume karo: weights $w_{i}$ aur inputs $x_{i}$ independent, zero-mean, i.i.d. hain.

Independent zero-mean variables ke liye, product aur sum ki variance deti hai: $Var (z) = \sum_{i = 1}^{n_{in}} Var (w_{i} x_{i}) = n_{in} Var (w) Var (x)$

Yeh step kyun? Zero-mean independent $w, x$ ke liye: $Var (w x) = Var (w) Var (x)$ , aur independents ke sum ki variance add hoti hai.

Forward goal: hum chahte hain $Var (z) = Var (x)$ taaki signal size preserve ho. Isse yeh force hota hai: $n_{in} Var (w) = 1 \Rightarrow Var (w) = \frac{1}{n _{in}}$

Backward goal: gradients ke backward flow par wahi argument $n_{o u t} Var (w) = 1$ deta hai, yaani $Var (w) = 1/ n_{o u t}$ .

Dono ko exactly satisfy nahi kar sakte, isliye Xavier average se compromise karta hai:

ReLU ko alag constant (He) kyun chahiye

ReLU aadhe inputs ko zero kar deta hai (saare negatives). Toh average par woh aadhi variance khatam kar deta hai: $Var (a) = \frac{1}{2} Var (z)$

Yeh step kyun? Zero-mean symmetric $z$ ke liye, $ϕ (z) = max (0, z)$ sirf positive half pass karta hai → second moment half ho jaata hai.

$Var (z^{[l]}) = Var (z^{[l - 1]})$ rakhhne ke liye hume compensate karne ke liye weight variance double karni padti hai: $n_{in} Var (w) \cdot \frac{1}{2} = 1 \Rightarrow Var (w) = \frac{2}{n _{in}}$

Figure — Weight initialization (Xavier, He)

Worked examples

Recall Feynman: 12-saal ke bachche ko samjhao

Socho 50 bachcho ki line mein ek whisper pass kar rahe ho. Agar har bachcha thoda loud bolta hai, toh end tak woh scream ban jaata hai (exploding). Agar har bachcha thoda soft bolta hai, toh aakhri bachcha kuch nahi sunta (vanishing). Weight initialization yeh choose karna hai ki pehli whisper kitni loud ho, aur har bachche ke liye ek "volume knob" set karna hai, taaki message same loudness par poori line mein jaaye. ReLU bachche sirf aadhi baar message pass karte hain (woh "sad" numbers ignore karte hain), isliye hum unhe double loud bolne ko kehte hain — yahi He initialization mein "2" hai.

Flashcards

Saare weights zero kyun initialize nahi kar sakte?

Yeh symmetry break nahi karta — ek layer ke saare neurons identical outputs compute karte hain aur identical gradients lete hain, isliye woh kabhi differentiate nahi hote.

Achhi weight initialization ka goal kya hai?

Activations ki variance (forward) aur gradients ki variance (backward) ko layers ke across roughly constant rakho, vanishing/exploding se bacho.

Xavier/Glorot weight variance formula?

Var (w) = \frac{2}{n _{in} + n _{o u t}}

He/Kaiming weight variance formula?

Var (w) = \frac{2}{n _{in}}

He,

1/ n_{in}

variance-preserving rule ke comparison mein 2 ka factor kyun use karta hai?

ReLU negative inputs ko zero karta hai, variance half kar deta hai; weight variance double karna compensate karta hai.

ReLU vs tanh/sigmoid ke liye kaunsa init?

ReLU (aur variants) ke liye He; tanh/sigmoid/linear ke liye Xavier.

z = \sum w_{i} x_{i}

ke liye zero-mean i.i.d. terms ke saath

Var (z)

derive karo.

Var (z) = n_{in} Var (w) Var (x)

U [- a, a]

mein Uniform Xavier bound

a

a = 6/ (n_{in} + n_{o u t})

a^{2} /3 = Var (w)

se.

n_{in}

(fan-in) kya hai?

Ek neuron/layer ke inputs ki sankhya =

W

ki input dimension.

Per-layer variance gain aur depth ka relationship?

Total scaling = (per-layer gain)

^{d e pt h}

, isliye koi bhi gain

\neq = 1

depth ke saath exponentially explode/vanish karta hai.

Connections

Vanishing and Exploding Gradients — woh failure modes jinhe init prevent karta hai.
Activation Functions (ReLU, tanh, sigmoid) — yeh determine karte hain ki kaunsa init constant use karna hai.
Batch Normalization — ek alternative/complement jo har step mein activations ko re-normalize karta hai.
Backpropagation — backward variance argument gradient flow se aata hai.
Variance and Expectation — derivation mein use hone waale statistical tools.
Deep Network Training Stability