Intuition The one-sentence ideaIf every layer multiplies or shrinks the signal even a little, then stacking 50 layers turns that tiny factor into a giant or vanishing number. Good initialization picks the starting weights so that the variance of activations (going forward) and gradients (going backward) stays roughly constant across layers — no explosion, no vanishing.
Definition Weight initializationThe rule for choosing the initial random values of the weight matrices W [ l ] W^{[l]} W [ l ] 0 0 0 W W W
WHY do we care? A deep net computes, layer by layer:
z [ l ] = W [ l ] a [ l − 1 ] , a [ l ] = ϕ ( z [ l ] ) z^{[l]} = W^{[l]} a^{[l-1]}, \qquad a^{[l]} = \phi(z^{[l]}) z [ l ] = W [ l ] a [ l − 1 ] , a [ l ] = ϕ ( z [ l ] ) W W W exploding ). Too small → activations shrink to 0 0 0 vanishing ). Either way gradients become useless and learning stalls.
Common mistake "Just set all weights to zero — simple and symmetric!"Why it feels right: zero is neutral, it can't explode or vanish.
Why it's wrong: if all weights in a layer are identical, every neuron in that layer computes the same output and receives the same gradient , so they update identically forever. This is the symmetry breaking failure — the neurons are permanently clones.
Fix: initialize with random values (breaks symmetry) but with a controlled variance (prevents explode/vanish).
We track the variance of a single pre-activation z = ∑ i = 1 n i n w i x i z = \sum_{i=1}^{n_{in}} w_i x_i z = ∑ i = 1 n in w i x i n i n n_{in} n in
Assume: weights w i w_i w i x i x_i x i
For independent zero-mean variables, variance of a product and sum gives:
V a r ( z ) = ∑ i = 1 n i n V a r ( w i x i ) = n i n V a r ( w ) V a r ( x ) \mathrm{Var}(z) = \sum_{i=1}^{n_{in}} \mathrm{Var}(w_i x_i) = n_{in}\,\mathrm{Var}(w)\,\mathrm{Var}(x) Var ( z ) = ∑ i = 1 n in Var ( w i x i ) = n in Var ( w ) Var ( x )
Why this step? For zero-mean independent w , x w,x w , x V a r ( w x ) = V a r ( w ) V a r ( x ) \mathrm{Var}(wx)=\mathrm{Var}(w)\mathrm{Var}(x) Var ( w x ) = Var ( w ) Var ( x )
Forward goal: we want V a r ( z ) = V a r ( x ) \mathrm{Var}(z)=\mathrm{Var}(x) Var ( z ) = Var ( x ) n i n V a r ( w ) = 1 ⇒ V a r ( w ) = 1 n i n \boxed{\,n_{in}\,\mathrm{Var}(w) = 1 \;\Rightarrow\; \mathrm{Var}(w) = \frac{1}{n_{in}}\,} n in Var ( w ) = 1 ⇒ Var ( w ) = n in 1
Backward goal: the same argument on gradients flowing back gives n o u t V a r ( w ) = 1 n_{out}\,\mathrm{Var}(w)=1 n o u t Var ( w ) = 1 V a r ( w ) = 1 / n o u t \mathrm{Var}(w)=1/n_{out} Var ( w ) = 1/ n o u t
You can't satisfy both exactly, so Xavier compromises with the average:
ReLU zeroes out half the inputs (all negatives). So on average it kills half the variance:
V a r ( a ) = 1 2 V a r ( z ) \mathrm{Var}(a) = \tfrac{1}{2}\,\mathrm{Var}(z) Var ( a ) = 2 1 Var ( z )
Why this step? For zero-mean symmetric z z z ϕ ( z ) = max ( 0 , z ) \phi(z)=\max(0,z) ϕ ( z ) = max ( 0 , z )
To keep V a r ( z [ l ] ) = V a r ( z [ l − 1 ] ) \mathrm{Var}(z^{[l]})=\mathrm{Var}(z^{[l-1]}) Var ( z [ l ] ) = Var ( z [ l − 1 ] ) double the weight variance to compensate:
n i n V a r ( w ) ⋅ 1 2 = 1 ⇒ V a r ( w ) = 2 n i n n_{in}\,\mathrm{Var}(w)\cdot\tfrac12 = 1 \;\Rightarrow\; \mathrm{Var}(w) = \frac{2}{n_{in}} n in Var ( w ) ⋅ 2 1 = 1 ⇒ Var ( w ) = n in 2
Worked example Example 1 — He init for a ReLU layerLayer maps 512 → 256 512 \to 256 512 → 256 n i n = 512 n_{in}=512 n in = 512 V a r ( w ) = 2 512 = 0.00390625 , σ = 0.0039 ≈ 0.0625 \mathrm{Var}(w)=\frac{2}{512}=0.00390625,\quad \sigma=\sqrt{0.0039}\approx 0.0625 Var ( w ) = 512 2 = 0.00390625 , σ = 0.0039 ≈ 0.0625 N ( 0 , 0.0625 2 ) \mathcal{N}(0, 0.0625^2) N ( 0 , 0.062 5 2 ) Why this step? ReLU → use He → variance 2 / n i n 2/n_{in} 2/ n in n i n = 512 n_{in}=512 n in = 512
Worked example Example 2 — Xavier uniform for a tanh layerLayer maps 100 → 100 100 \to 100 100 → 100 n i n = n o u t = 100 n_{in}=n_{out}=100 n in = n o u t = 100 a = 6 100 + 100 = 0.03 ≈ 0.173 a=\sqrt{\frac{6}{100+100}}=\sqrt{0.03}\approx 0.173 a = 100 + 100 6 = 0.03 ≈ 0.173 U [ − 0.173 , 0.173 ] \mathcal{U}[-0.173,\ 0.173] U [ − 0.173 , 0.173 ] Why this step? tanh is symmetric & near-linear at 0 0 0
Worked example Example 3 — Feel the explosionSuppose you naively use V a r ( w ) = 1 \mathrm{Var}(w)=1 Var ( w ) = 1 n i n = 256 n_{in}=256 n in = 256 = n i n ⋅ V a r ( w ) ⋅ 1 2 = 256 ⋅ 1 ⋅ 0.5 = 128 = n_{in}\cdot\mathrm{Var}(w)\cdot\tfrac12 = 256\cdot1\cdot0.5=128 = n in ⋅ Var ( w ) ⋅ 2 1 = 256 ⋅ 1 ⋅ 0.5 = 128 128 5 ≈ 3.4 × 10 10 128^5\approx 3.4\times10^{10} 12 8 5 ≈ 3.4 × 1 0 10 explosion .
With He (V a r ( w ) = 2 / 256 \mathrm{Var}(w)=2/256 Var ( w ) = 2/256 = 256 ⋅ 2 256 ⋅ 1 2 = 1 =256\cdot\frac{2}{256}\cdot\frac12=1 = 256 ⋅ 256 2 ⋅ 2 1 = 1 Why this step? The per-layer gain gets raised to the power = depth, so a gain of even 2 2 2 0.5 0.5 0.5
Recall Feynman: explain to a 12-year-oldImagine passing a whisper down a line of 50 kids. If each kid speaks a little louder , by the end it's a scream (exploding). If each speaks a little softer , the last kid hears nothing (vanishing). Weight initialization is choosing how loud the first whisper is, and a "volume knob" for each kid, so the message stays the same loudness all the way down the line. ReLU kids only pass the message half the time (they ignore "sad" numbers), so we tell them to speak twice as loud — that's the "2" in He initialization.
Mnemonic Remember the constants"He is Two, Xavier is Sum."
He (ReLU) → 2 / n i n 2/n_{in} 2/ n in 2 because ReLU throws away half.Xavier (tanh/sigmoid) → 2 / ( n i n + n o u t ) 2/(n_{in}+n_{out}) 2/ ( n in + n o u t ) Sums both fans.
Why can't we initialize all weights to zero? It fails to break symmetry — all neurons in a layer compute identical outputs and get identical gradients, so they never differentiate.
What is the goal of good weight initialization? Keep the variance of activations (forward) and gradients (backward) roughly constant across layers, avoiding vanishing/exploding.
Xavier/Glorot weight variance formula? V a r ( w ) = 2 n i n + n o u t \mathrm{Var}(w)=\dfrac{2}{n_{in}+n_{out}} Var ( w ) = n in + n o u t 2 .
He/Kaiming weight variance formula? V a r ( w ) = 2 n i n \mathrm{Var}(w)=\dfrac{2}{n_{in}} Var ( w ) = n in 2 .
Why does He use a factor of 2 vs the 1 / n i n 1/n_{in} 1/ n in ReLU zeroes negative inputs, halving the variance; doubling the weight variance compensates.
Which init for ReLU vs tanh/sigmoid? He for ReLU (and variants); Xavier for tanh/sigmoid/linear.
Derive V a r ( z ) \mathrm{Var}(z) Var ( z ) z = ∑ w i x i z=\sum w_i x_i z = ∑ w i x i V a r ( z ) = n i n V a r ( w ) V a r ( x ) \mathrm{Var}(z)=n_{in}\mathrm{Var}(w)\mathrm{Var}(x) Var ( z ) = n in Var ( w ) Var ( x ) .
Uniform Xavier bound a a a U [ − a , a ] \mathcal{U}[-a,a] U [ − a , a ] a = 6 / ( n i n + n o u t ) a=\sqrt{6/(n_{in}+n_{out})} a = 6/ ( n in + n o u t ) , from
a 2 / 3 = V a r ( w ) a^2/3=\mathrm{Var}(w) a 2 /3 = Var ( w ) .
What is n i n n_{in} n in The number of inputs to a neuron/layer = input dimension of
W W W .
Per-layer variance gain and depth relationship? Total scaling = (per-layer gain)
d e p t h ^{depth} d e pt h , so any gain
≠ 1 \ne 1 = 1 explodes/vanishes exponentially with depth.
each layer multiplies signal
Random weights + controlled variance
Xavier / Glorot: 2 / n_in+n_out
Intuition Hinglish mein samjhoSocho aap ek 50-layer ki deep network bana rahe ho. Har layer input ko matrix W W W W W W scale sahi chunna zaroori hai.
Idea simple hai: hum chahte hain ki har layer ke baad activation ki variance same rahe . Maths se: z = ∑ w i x i z=\sum w_i x_i z = ∑ w i x i = n i n ⋅ V a r ( w ) ⋅ V a r ( x ) = n_{in}\cdot \mathrm{Var}(w)\cdot \mathrm{Var}(x) = n in ⋅ Var ( w ) ⋅ Var ( x ) V a r ( w ) = 1 / n i n \mathrm{Var}(w)=1/n_{in} Var ( w ) = 1/ n in Xavier ka logic hai — tanh/sigmoid ke liye, jahan hum fan-in aur fan-out dono ka average lete hain: V a r ( w ) = 2 / ( n i n + n o u t ) \mathrm{Var}(w)=2/(n_{in}+n_{out}) Var ( w ) = 2/ ( n in + n o u t )
He initialization ReLU ke liye hai. ReLU saare negative numbers ko zero kar deta hai — yaani aadha signal maar deta hai, variance aadhi ho jaati hai. Isko compensate karne ke liye hum weight ki variance double kar dete hain: V a r ( w ) = 2 / n i n \mathrm{Var}(w)=2/n_{in} Var ( w ) = 2/ n in
Yaad rakhne ka tarika: "He is Two, Xavier is Sum." ReLU use kar rahe ho to He (2/n_in), tanh/sigmoid ho to Xavier (fan-in + fan-out ka sum). Aur kabhi bhi saare weights zero mat karna — warna saare neurons clone ban jaate hain aur kuch seekhte hi nahi (symmetry break nahi hota).
