Weight initialization (Xavier, He): Question bank

This page assumes you already met the two rules:

Xavier / Glorot (for tanh, sigmoid, linear): $Var (w) = \frac{2}{n _{in} + n _{o u t}}$
He / Kaiming (for ReLU and its variants): $Var (w) = \frac{2}{n _{in}}$

where $n_{in}$ = fan-in (number of inputs feeding one neuron) and $n_{o u t}$ = fan-out (number of neurons the signal feeds forward into next). If either term is fuzzy, revisit Variance and Expectation and 3.1.12 Weight initialization (Xavier, He) (Hinglish) first.

True or false — justify

Good init guarantees the network will train successfully.

False. It only keeps the starting variance sane so gradients are usable at step 0; a bad learning rate, bad architecture, or bad data can still stall training. Init removes one failure mode, not all of them.

He initialization gives a larger weight variance than Xavier for the same layer.

Generally true. He uses

2/ n_{in}

while Xavier uses

2/ (n_{in} + n_{o u t})

; since

n_{in} < n_{in} + n_{o u t}

, He's denominator is smaller, so its variance is larger — the extra size compensates for ReLU discarding half the signal.

Setting all weights to the same nonzero constant (say

0.1

) is fine because it's not zero.

False. The problem was never the value zero specifically — it is that identical weights make every neuron in a layer compute the same output and receive the same gradient. Any constant fails symmetry breaking; only randomness breaks it.

Biases must also be randomized to break symmetry.

False. Randomizing the weights already gives each neuron a different function, so biases can safely start at

0

. Symmetry breaking is a property of the weight matrix rows being different, not the biases.

If we use Batch Normalization, weight initialization no longer matters at all.

Mostly false. Batch Normalization re-centers and re-scales activations each step, so it makes the network far more forgiving of init — but a wildly scaled init can still hurt the very first forward/backward passes before BN's statistics settle. Good init and BN are complements, not substitutes.

Multiplying every weight by

2

after He init is harmless since it's "still random."

False. Scaling weights by

2

multiplies the variance by

4

, so each layer's gain becomes

g = 4

instead of

1

; over

L

layers that is

4^{L}

— an exponential explosion. The scale is the whole point, not just the randomness.

Xavier init works fine for a deep ReLU network, just slightly less optimal.

False in a dangerous way. Xavier assumes the activation passes ~all the variance; ReLU passes only half, so each layer loses a factor of

2

in variance, giving gain

g \approx 1/2

per layer and

2^{- L}

vanishing across depth. The mismatch compounds exponentially — it is not a mild inefficiency.

The variance-preserving derivation needs weights and inputs to be independent and zero-mean.

True. The clean rule

Var (w x) = Var (w) Var (x)

and "variance of a sum adds" both rely on independence and zero mean; without them cross terms and covariances appear and the tidy

n_{in} Var (w)

result breaks.

Glorot-normal and Glorot-uniform give different variances, so they train differently.

False. Both are tuned to the same target variance

2/ (n_{in} + n_{o u t})

— the normal form sets that as

σ^{2}

directly, the uniform form picks its bound

a

so that

a^{2} /3

equals it. They differ only in the shape of the random draws, not the variance the theory cares about.

Spot the error

"Fan-in for a layer mapping

512 \to 256

256

, because that's the layer's output size."

Error. Fan-in is the input dimension, so

n_{in} = 512

, not

256

. The

256

is fan-out. He init would use

2/512

, not

2/256

"He variance is

2/ n_{in}

, so the standard deviation to sample from is also

2/ n_{in}

Error. Standard deviation is the square root of variance:

σ = 2/ n_{in}

. Sampling with std

= 2/ n_{in}

would give the wrong (much smaller) spread for large

n_{in}

"Xavier uniform samples from

U [- 2/ (n_{in} + n_{o u t}), + 2/ (n_{in} + n_{o u t})]

to match the variance."

Error. A uniform

U [- a, a]

has variance

a^{2} /3

, not

a

; setting

a

equal to the target variance ignores that factor. The correct bound is

a = 6/ (n_{in} + n_{o u t})

"ReLU halves the variance, so He fixes it by halving the weight variance."

Error. ReLU halving the activation variance means we must double the weight variance to compensate (

2/ n_{in}

), not halve it. The factor

2

pushes the signal back up to full strength.

"Since forward wants

Var (w) = 1/ n_{in}

and backward wants

1/ n_{o u t}

, Xavier just picks whichever is smaller to be safe."

Error. Xavier takes the harmonic-style average by using

2/ (n_{in} + n_{o u t})

, a compromise that lands between the two targets. Picking the smaller would over-shrink and risk vanishing in one direction.

"With He init the per-layer variance gain is

g = 2

, which is why it prevents explosion."

Error. He is designed so the gain equals $g = 1$ :

n_{in} \cdot \frac{2}{n _{in}} \cdot \frac{1}{2} = 1

. The three factors (fan-in, weight variance, ReLU's

\frac{1}{2}

) multiply to exactly

1

— that unit gain is what keeps variance stable across depth.

"For a conv layer with

64

input channels and

3 \times 3

kernels mapping to

128

output channels, fan-in is just

64

Error. For convolutions, fan-in counts every weight feeding one output pixel:

n_{in} = channels_{in} \times k_{h} \times k_{w} = 64 \times 3 \times 3 = 576

. Fan-out is

channels_{o u t} \times k_{h} \times k_{w} = 128 \times 9 = 1152

. Ignoring the kernel area badly under-scales the variance.

Why questions

Why does the per-layer variance factor get raised to the depth rather than added?

Each layer multiplies the running variance by its gain

g

, and repeated multiplication is exponentiation:

L

layers of gain

g

give

g^{L}

. That is why even a mild gain like

g = 1.5

g = 0.7

becomes catastrophic in a deep net.

Why do we track the variance of the pre-activation

z

specifically, and not its mean?

With zero-mean weights and inputs the mean of

z

stays

0

automatically, so the mean carries no information about signal size; variance is what measures how big

z

is, so keeping it constant is the real goal.

Why does the same weight-variance argument apply to gradients flowing backward?

Backpropagation multiplies gradients by (transposed) weight matrices layer by layer, so gradients obey the same "sum of products of independents" structure as the forward pass — just with

n_{o u t}

replacing

n_{in}

, giving the backward target

1/ n_{o u t}

Why can't a single choice of

Var (w)

satisfy both the forward and backward goals exactly?

Forward preservation needs

n_{in} Var (w) = 1

and backward needs

n_{o u t} Var (w) = 1

; unless

n_{in} = n_{o u t}

these demand two different variances, so we compromise (Xavier's average) instead of picking one.

Why is symmetry breaking a permanent failure and not something training fixes later?

If two neurons start identical, they get identical gradients, so they update identically and stay identical forever — gradient descent can never separate points it moves in lockstep. Only different starting weights can break the tie.

Why does sigmoid combined with a variance-preserving init still risk vanishing gradients?

Even if activation variance is preserved, sigmoid's derivative is tiny (near

0

) whenever inputs drift into its saturated flat regions, so gradients shrink through the derivative term regardless of init — see Vanishing and Exploding Gradients.

Why does Xavier suit tanh but He suit ReLU, given both are "activations"?

tanh is symmetric and nearly linear near

0

, so it passes almost all the variance — Xavier's variance-preserving assumption holds. ReLU discards the negative half, breaking that assumption, which is exactly what He's factor

2

repairs.

Why do frameworks offer truncated normal init instead of plain normal?

A plain normal occasionally draws extreme weights far out in the tails, which can seed a few outsized activations; truncated normal re-samples anything beyond about

2 σ

, keeping the same target variance while removing rare huge values that could destabilize the first steps.

Edge cases

For a layer where

n_{in} = n_{o u t}

, Xavier reduces to which simple rule?

It becomes

Var (w) = 2/ (2 n) = 1/ n

, i.e. exactly the plain forward variance-preserving rule

1/ n_{in}

— because the forward and backward targets coincide when the fans are equal.

What happens to He variance

2/ n_{in}

as a layer gets extremely wide (

n_{in} \to \infty

The variance

\to 0

, so individual weights become tiny; this is correct, because summing many terms of small variance still yields a

z

with the intended finite variance — the shrink exactly offsets the growing number of inputs.

The very first layer's inputs are raw data (not zero-mean, not unit-variance). Does the derivation still apply?

Not cleanly — the zero-mean/i.i.d. assumptions can fail on raw features, which is why we usually standardize inputs first so the first layer sees roughly zero-mean, unit-variance data and the init math holds.

A layer with fan-in

1

(a single input) — is init even meaningful?

Yes, but degenerate: with

n_{in} = 1

, He gives

Var (w) = 2

and Xavier depends on

n_{o u t}

; there's no averaging over many inputs, so the variance-of-a-sum argument is trivial and the practical benefit of careful scaling is minimal.

For the output layer feeding a loss (no further layer), what should

n_{o u t}

mean?

Conceptually the backward target loses its usual meaning since nothing follows, so people often fall back to the forward-only rule or use the number of output units as

n_{o u t}

; either way the exact choice matters least here because it's the last layer.

Leaky ReLU passes a small slope

α

for negatives instead of zeroing them. Should we still use plain He?

Not exactly — since leaky ReLU keeps some negative-side variance, less than half is discarded, so the ideal constant is slightly less than

2

; the corrected He formula uses

2/ ((1 + α^{2}) n_{in})

If a network is shallow (2–3 layers), does init choice still matter much?

Much less — the exponential term

g^{L}

barely moves for tiny

L

, so a gain of

g = 1.5

over

2

layers is only

2.25 \times

, easily recoverable by training. Init matters most precisely when depth

L

is large.

For a convolutional layer, which distribution do frameworks default to — uniform or normal?

It varies: PyTorch's default kaiming_uniform_ uses the uniform form, while many He-init recipes in papers use the normal form; both target the same variance

2/ n_{in}

(with

n_{in} = channels_{in} \times k_{h} \times k_{w}

), so results are equivalent up to draw shape.

Recall One-line self-test before you leave

Can you, from memory, state (1) why the gain $g$ is exponentiated by depth, (2) why He doubles Xavier for ReLU, and (3) why zero init fails even though it can't explode? Answers ::: (1) Layers multiply variance, and repeated multiplication is a power, so $L$ layers give $g^{L}$ ; (2) ReLU kills half the variance so we double the weight variance to restore gain $g = 1$ ; (3) zero (or any constant) makes neurons identical, so symmetry never breaks.

Connections

Vanishing and Exploding Gradients — every trap here is ultimately about avoiding these two.
Activation Functions (ReLU, tanh, sigmoid) — decides which constant is correct.
Batch Normalization — why it makes init less critical but not irrelevant.
Backpropagation — source of the backward variance target $1/ n_{o u t}$ .
Variance and Expectation — the assumptions the derivation leans on.
Deep Network Training Stability — the practical payoff of getting init right.