Weight initialization (Xavier, He): Worked examples

Before anything, two reminders of the vocabulary (built in the parent, repeated so you never guess):

How to read the scenario matrix

Every weight-init question is one of the nine cells below. First skim the flowchart — it is the "decision tree" you walk to land on the right cell; then the table names each cell and its example.

Cell	Situation	What makes it tricky	Example
A	ReLU layer, $n_{in} \neq = n_{o u t}$	He uses only fan-in → output size is a decoy	Ex 1
B	tanh/sigmoid layer, general	Xavier uses both fans	Ex 2
C	Square layer $n_{in} = n_{o u t}$	He vs Xavier give different answers even here	Ex 3
D	Uniform sampling (need bound $a$ )	must convert variance → half-width via $a^{2} /3$	Ex 4
E	Degenerate: $n_{in} = 1$	limiting/edge case, formula still holds	Ex 5
F	Wrong constant → exploding	per-layer gain $> 1$ , raised to depth	Ex 6
G	Wrong constant → vanishing	per-layer gain $< 1$ , raised to depth	Ex 7
H	Real-world word problem	pick activation → pick init → size it	Ex 8
I	Exam twist: mixed activations	which formula per layer, and why	Ex 9

Example 1 — ReLU layer, fan-in ≠ fan-out (cell A)

Worked example Statement

A hidden layer maps a vector of size $784 \to 128$ with ReLU. Give $Var (w)$ and the standard deviation $σ$ to sample $w \sim N (0, σ^{2})$ .

Forecast: which number does $σ$ depend on — $784$ , $128$ , or both? Guess before reading.

Activation is ReLU → use He. He variance is $Var (w) = 2/ n_{in}$ . Why this step? ReLU zeroes the negative half of its inputs, halving variance; He's factor-2 compensates (parent derivation). Xavier's constant would leave the signal too small in a ReLU net.
Identify fan-in. Inputs come in with dimension $784$ , so $n_{in} = 784$ . The $128$ is fan-out and He ignores it. Why this step? He balances the forward pass only, which depends on how many terms are summed into each $z = \sum_{i = 1}^{n_{in}} w_{i} x_{i}$ — that count is $n_{in}$ .
Plug in. $Var (w) = \frac{2}{784} \approx 0.002551$ .
Standard deviation. $σ = 0.002551 \approx 0.05051$ .

Verify: $σ^{2} = 0.0505 1^{2} \approx 0.002551 = 2/784$ . ✓ The answer used only fan-in, exactly as He prescribes. Answer to the forecast: only 784.

Example 2 — tanh layer, general fan-in ≠ fan-out (cell B)

Worked example Statement

A layer maps $256 \to 64$ with tanh. Give $Var (w)$ (normal form).

Forecast: will the variance be bigger or smaller than He on the same fan-in?

Activation is tanh → use Xavier. $Var (w) = \frac{2}{n _{in} + n _{o u t}}$ . Why this step? tanh is symmetric and near-linear around $0$ , so it does not kill half the signal — no factor-2 boost needed. Xavier instead compromises between the forward rule $1/ n_{in}$ and the backward rule $1/ n_{o u t}$ by averaging the two fans.
Sum the fans. $n_{in} + n_{o u t} = 256 + 64 = 320$ . Why this step? Xavier's denominator is the sum, so both the forward and backward variance constraints are approximately met at once.
Plug in. $Var (w) = \frac{2}{320} = 0.00625$ .

Verify: compare to He on $n_{in} = 256$ : He gives $2/256 = 0.0078125$ . Xavier here ( $0.00625$ ) is smaller — as forecast, because the denominator $320 > 256$ . ✓

Example 3 — Square layer: He and Xavier disagree (cell C)

Example 4 — Uniform sampling: convert variance to a bound (cell D)

Worked example Statement

Use Xavier uniform for a $400 \to 200$ tanh layer: find the half-width $a$ so that $w \sim U [- a, a]$ .

Forecast: will $a$ be larger or smaller than $Var (w)$ ?

Target variance (Xavier). $Var (w) = \frac{2}{400 + 200} = \frac{2}{600} = 0.003333$ .
Relate uniform variance to $a$ . A uniform on $[- a, a]$ has variance $a^{2} /3$ . Why this step? Sampling uniformly, not normally, changes the spread–variance link. We must invert it so the uniform delivers the same variance the theory demands.
Solve for $a$ algebraically. Set the uniform's variance equal to the target: $\frac{a ^{2}}{3} = \frac{2}{n _{in} + n _{o u t}} .$ Multiply both sides by $3$ to isolate $a^{2}$ : $a^{2} = 3 \cdot \frac{2}{n _{in} + n _{o u t}} = \frac{6}{n _{in} + n _{o u t}} .$ Take the square root: $a = \frac{6}{n _{in} + n _{o u t}} .$ Why this step? This is exactly where the mysterious "6" in Xavier-uniform comes from — it is $3 \times 2$ , the $3$ from the uniform's $a^{2} /3$ and the $2$ from Xavier's numerator. Now plug numbers: $a = 6/600 = 0.01 = 0.1$ .

Verify: $a^{2} /3 = 0. 1^{2} /3 = 0.01/3 = 0.003333 = Var (w)$ . ✓ Note $a = 0.1$ is larger than $Var = 0.003333 \approx 0.0577$ , because uniform spreads mass to the edges.

Example 5 — Degenerate case: a single input, $n_{in} = 1$ (cell E)

Worked example Statement

A neuron has exactly one input ( $n_{in} = 1$ ) and uses ReLU. What He variance does the formula give, and does it make sense?

Forecast: with one input, is variance-preservation even meaningful?

Apply He. $Var (w) = \frac{2}{1} = 2$ , so $σ = 2 \approx 1.4142$ .
Sanity check the forward variance. For one term, $Var (z) = n_{in} Var (w) Var (x) = 1 \cdot 2 \cdot Var (x) = 2 Var (x)$ . Why this step? The forward-variance law $Var (z) = n_{in} Var (w) Var (x)$ still holds with $n_{in} = 1$ . He deliberately doubles it so that ReLU's halving brings the activation variance back to $Var (x)$ .
After ReLU. $Var (a) = \frac{1}{2} Var (z) = \frac{1}{2} \cdot 2 Var (x) = Var (x)$ . Preserved. ✓ Why this step? This shows the formula is not a fluke of large $n$ — it works at the tiny edge $n_{in} = 1$ too.

Verify: $2/1 = 2$ , and $\frac{1}{2} (1 \cdot 2) = 1$ , so the activation variance equals the input variance exactly. ✓ Degenerate case is consistent.

Example 6 — Wrong constant → explosion (cell F)

Worked example Statement

A 6-layer ReLU net with $n_{in} = 256$ per layer is initialized naively with $Var (w) = 1$ (standard normal). By what factor does the activation variance scale from input to output?

Forecast: guess the order of magnitude — tens? millions? billions?

Per-layer gain. Forward variance factor $g = n_{in} \cdot Var (w) \cdot \frac{1}{2} = 256 \cdot 1 \cdot 0.5 = 128$ . Why this step? Using the gain definition above with the naive $Var (w) = 1$ : each ReLU layer multiplies variance by $128$ .
Raise to depth. Over $6$ layers the total scaling is $g^{6} = 12 8^{6}$ . Why this step? The gain compounds — layer after layer multiplies, so exponents, not sums, govern the outcome.
Evaluate. $12 8^{6} = 4, 398, 046, 511, 104 \approx 4.4 \times 1 0^{12}$ .

Verify: $128 = 2^{7}$ , so $12 8^{6} = 2^{42} = 4398046511104$ . ✓ Activation variance blows up by ~ $4.4$ trillion → explosion (see Vanishing and Exploding Gradients).

How to read the figure below: the horizontal axis is layer depth $0 \to 6$ ; the vertical axis is the activation-variance scaling on a log scale (each gridline is $\times 10$ ). The orange curve is this naive gain- $128$ case — it rockets almost vertically, passing $1 0^{12}$ by layer 6. The flat magenta line is He ( $g = 1$ ): read straight across at height $1$ , meaning variance is unchanged. The gap between the two curves at any depth is exactly how badly the naive init has distorted the signal.

Figure — Weight initialization (Xavier, He)

Example 7 — Wrong constant → vanishing (cell G)

Worked example Statement

Same 6-layer ReLU net, $n_{in} = 256$ , but now over-cautious: $Var (w) = \frac{1}{1024}$ . What is the total variance scaling across 6 layers?

Forecast: will the signal survive to layer 6, or die?

Per-layer gain. $g = 256 \cdot \frac{1}{1024} \cdot \frac{1}{2} = \frac{256}{2048} = 0.125$ . Why this step? Same gain recipe (fan-in × weight-variance × ReLU's ½), but now $g < 1$ , so each layer shrinks the signal.
Raise to depth. Total $= g^{6} = 0.12 5^{6}$ .
Evaluate. $0.12 5^{6} = (1/8)^{6} = 1/262144 \approx 3.81 \times 1 0^{- 6}$ .

Verify: $0.125 = 2^{- 3}$ , so $0.12 5^{6} = 2^{- 18} = 1/262144 \approx 3.815 \times 1 0^{- 6}$ . ✓ The signal is $\sim$ 260,000× smaller at the output → vanishing.

How to read the figure below: same axes as before — depth on the horizontal, variance scaling on a log vertical axis. The violet curve is this gain- $0.125$ case; it plunges below $1 0^{- 5}$ by layer 6, i.e. the signal has all but disappeared. The magenta flat line at height $1$ is He ( $g = 1$ ) — the only choice that neither climbs nor collapses. Reading the two figures together: explosion and vanishing are mirror images across the magenta He line.

Example 8 — Real-world word problem (cell H)

Worked example Statement

You are building an image classifier. The convolutional backbone flattens to $2048$ features, then a fully-connected head maps $2048 \to 512 \to 512 \to 10$ . Layers 1–2 use ReLU, the final $512 \to 10$ uses a softmax (linear pre-activation). Initialize every weight matrix.

Forecast: how many different init formulas will you use across these three matrices?

Layer $2048 \to 512$ (ReLU) → He. $Var = 2/2048 = 0.0009766$ , $σ \approx 0.03125$ . Why this step? ReLU → He, fan-in $= 2048$ .
Layer $512 \to 512$ (ReLU) → He. $Var = 2/512 = 0.003906$ , $σ \approx 0.0625$ . Why this step? ReLU again, fan-in $= 512$ .
Layer $512 \to 10$ (linear/softmax) → Xavier. $Var = \frac{2}{512 + 10} = \frac{2}{522} = 0.003831$ . Why this step? The softmax's pre-activation is a plain linear combination $z = \sum w_{i} x_{i}$ — softmax normalises the outputs but does not rectify or clip the pre-activation, so unlike ReLU it does not zero out half the signal. Its second moment is therefore not halved, and He's factor-2 boost would over-inflate the variance. On the backward pass, gradients flow through this narrow head symmetrically (both the $512$ inputs and the $10$ outputs matter), which is exactly the situation Xavier's symmetric $2/ (n_{in} + n_{o u t})$ was derived for. Note the fan-out ( $10$ ) genuinely matters here because the head is so narrow.

Verify: $2/2048 = 0.0009766$ (√=0.03125); $2/512 = 0.003906$ (√=0.0625); $2/522 = 0.003831$ . Two He, one Xavier → two formulas across three matrices, as forecast. ✓

Example 9 — Exam twist: sigmoid then ReLU, same shape (cell I)

Worked example Statement

Two layers, both $300 \to 300$ . Layer 1 uses sigmoid, layer 2 uses ReLU. A student initializes both with He ( $2/300$ ). Which layer is mis-initialized, by what factor, and what's correct?

Forecast: which of the two is wrong?

Layer 2 (ReLU) with He is correct. $Var = 2/300 = 0.006667$ . ✓ Why this step? ReLU → He is exactly right.
Layer 1 (sigmoid) should be Xavier. Correct value $= \frac{2}{300 + 300} = \frac{2}{600} = 0.003333$ . Why this step? Sigmoid (like tanh) is not a half-killer, so it needs Xavier, not He.
Quantify the error. Student used $2/300 = 0.006667$ ; correct is $0.003333$ . Ratio $= 0.006667/0.003333 = 2$ → the sigmoid layer's weights are twice the correct variance. Why this step? Over-large variance in a sigmoid layer pushes pre-activations into the flat tails where the gradient $\approx 0$ — the vanishing-gradient trap from Activation Functions (ReLU, tanh, sigmoid).

Verify: $(2/300) / (2/600) = 600/300 = 2$ . ✓ Layer 1 (sigmoid) is over-scaled by exactly $2 \times$ ; fix it with Xavier $= 2/600 = 0.003333$ variance.

Recall Quick self-test

ReLU $512 \to 128$ : which fan and which constant? ::: He, $2/ n_{in} = 2/512$ ; fan-out $128$ is ignored. tanh $256 \to 64$ variance? ::: Xavier $2/ (256 + 64) = 2/320 = 0.00625$ . Xavier uniform half-width for $400 \to 200$ ? ::: $a = 6/600 = 0.1$ (from $a^{2} /3 = 2/600 \Rightarrow a^{2} = 6/600$ ). He variance for $n_{in} = 1$ ? ::: $2/1 = 2$ (still preserves variance after ReLU). Naive $Var (w) = 1$ on $n_{in} = 256$ ReLU, 6 layers — total gain? ::: $12 8^{6} = 2^{42} \approx 4.4 \times 1 0^{12}$ (explodes). Why is a square layer's He twice its Xavier? ::: Xavier denominator is $2 n_{in}$ , so it is half of He's $2/ n_{in}$ — ReLU's compensation factor. What value do we initialize biases to, and why? ::: Zero — biases have no fan-in to amplify and keeping $b = 0$ preserves the zero-mean assumption the variance derivation needs.

Connections

Vanishing and Exploding Gradients — Examples 6 & 7 are these failure modes, quantified.
Activation Functions (ReLU, tanh, sigmoid) — the activation picks the formula (Examples 8, 9).
Variance and Expectation — every "Verify" uses the variance laws.
Backpropagation — the fan-out term in Xavier comes from the backward pass.
Batch Normalization — re-centres activations, mitigating ReLU's mean-shift caveat.
Deep Network Training Stability — the depth-exponent effect (gain $^{depth}$ ).
Weight initialization (Xavier, He) — parent note with the full derivation.