Visual walkthrough — Weight initialization (Xavier, He)

---

Step 1 — What "spread" means (variance, from zero)

WHAT. Before touching neural nets, we need one measuring stick: how spread out is a bunch of random numbers? We call this the variance, written $\mathrm{Var}(x)$.

WHY this tool and not the average? The average (mean) tells you the center of a cloud of numbers — but a whisper and a scream can have the same center (zero). We care about loudness, i.e. how far numbers stray from the center. That is exactly what variance measures. See Variance and Expectation for the full toolkit; here is the one idea we need.

PICTURE. Two clouds of dots with the same center but different spread. The narrow cloud has small variance (quiet); the wide cloud has large variance (loud). The horizontal bars show one standard deviation $\sigma =\sqrt{\mathrm{Var}}$ — the typical distance from center.

Recall Why zero-mean matters

Which is easier: average of $(x-\mu {)}^2$, or average of $x^2$? ::: Average of $x^2$ — and they're equal exactly when $\mu =0$, which is why we always center weights and inputs at zero.

---

Step 2 — One neuron is a sum of little pushes

WHAT. A single neuron computes one number $z$ before its activation: $z=\underset{\text{push 1}}{\underbrace{w_1{x}_1}}+\underset{\text{push 2}}{\underbrace{w_2{x}_2}}+\cdots +\underset{\text{push }{n}_{in}}{\underbrace{w_{n_{in}}{x}_{n_{in}}}}$

• $x_i$ = the $i$-th input coming in,
• $w_i$ = the weight (volume knob) on that input,
• $n_{in}$ = fan-in, the number of inputs feeding this neuron (see Backpropagation for how these connect layer to layer).

WHY. Each product $w_i{x}_i$ is one small "push" up or down. The neuron just adds them all. If all pushes are zero-mean and independent, they point randomly — some up, some down.

PICTURE. $n_{in}$ little arrows (the products $w_i{x}_i$), pointing randomly up/down, being summed head-to-tail into one net arrow $z$. Because they cancel partially, $z$ does not grow like $n_{in}$ — it grows like $\sqrt{n_{in}}$. That  ​ is the whole ballgame.

---

Step 3 — How variance flows through the sum

WHAT. We compute the spread of $z$. Two rules of variance (proven in Variance and Expectation) do all the work:

Put them together. All the $w_i$ share one spread $\mathrm{Var}(w)$, all $x_i$ share $\mathrm{Var}(x)$, so all $n_{in}$ terms are identical:

$\mathrm{Var}(z)=\underset{\text{how many pushes}}{\underbrace{n_{in}}}\times \underset{\text{knob spread}}{\underbrace{\mathrm{Var}(w)}}\times \underset{\text{input spread}}{\underbrace{\mathrm{Var}(x)}}$

WHY this matters. This single equation is the amplifier gain of one layer. Read it as: output loudness = (fan-in) × (weight loudness) × (input loudness).

PICTURE. A funnel/amplifier: $n_{in}$ inputs each of variance $\mathrm{Var}(x)$ enter; the box multiplies loudness by the factor $n_{in}\mathrm{Var}(w)$; out comes $z$ with variance $\mathrm{Var}(z)$. The multiplier is highlighted.

---

Step 4 — Stack many layers: the gain gets a power

WHAT. Call the per-layer multiplier $g=n_{in}\mathrm{Var}(w)$. If every layer has the same gain $g$, then after $L$ layers the loudness is scaled by $\mathrm{Var}(z^{[L]})=g^L\mathrm{Var}(x)$

• $g$ = per-layer gain,
• $L$ = depth (number of layers),
• $g^L$ = total gain — a gain raised to the depth.

WHY this is the whole problem. An exponent is merciless:

• $g=1.1$, $L=50$ → $g^L\approx 117$ (loud scream),
• $g=0.9$, $L=50$ → $g^L\approx 0.005$ (dead silence),
• $g=1$, $L=50$ → $g^L=1$ (perfect — the whisper stays a whisper).

This is Vanishing and Exploding Gradients in one line. The only safe target is $g=1$.

PICTURE. Three curves of $g^L$ vs depth $L$: orange $g=1.1$ rockets up (exploding), red $g=0.9$ collapses to zero (vanishing), green $g=1$ stays flat. Log scale on the y-axis to show the exponential.

---

Step 5 — Solve for the weight spread (the forward answer)

WHAT. We demand $g=1$ and solve for the one thing we control — $\mathrm{Var}(w)$: $\underset{g}{\underbrace{n_{in}\mathrm{Var}(w)}}=1⟹\boxed{\mathrm{Var}(w)=\frac{1}{n_{in}}}$

• $n_{in}$ = fan-in (given by the architecture, fixed),
• $\mathrm{Var}(w)$ = the knob spread we get to choose.

WHY. More inputs (bigger $n_{in}$) means more pushes piling up, so each push must be quieter — hence weight spread shrinks like $1/n_{in}$. This is the "variance-preserving forward" rule.

PICTURE. Two neurons side by side: a small one ($n_{in}=4$) with fat weight arrows, a big one ($n_{in}=64$) with thin weight arrows. A caption shows both produce the same output loudness because the thin knobs exactly cancel the extra inputs.

---

Step 6 — The backward pass wants the other fan

WHAT. Training also sends gradients backward (see Backpropagation). The identical variance argument, run on the backward signal, replaces $n_{in}$ with $n_{out}$ (the number of outputs the neuron feeds): $\text{forward wants }\mathrm{Var}(w)=\frac{1}{n_{in}},\text{backward wants }\mathrm{Var}(w)=\frac{1}{n_{out}}$

• $n_{in}$ = inputs into the layer (forward fan),
• $n_{out}$ = outputs of the layer (backward fan).

WHY the conflict. Unless $n_{in}=n_{out}$, you cannot satisfy both. Xavier/Glorot splits the difference by averaging the two demands: $\mathrm{Var}(w)=\frac{2}{n_{in}+n_{out}}$ The "2" here is just the "$1+1$" from averaging two $1/n$ targets — not the ReLU "2" of Step 8.

PICTURE. A tug-of-war rope: forward pass (blue) pulls toward $1/n_{in}$, backward pass (orange) pulls toward $1/n_{out}$. The green flag (Xavier) sits at the balance point $2/(n_{in}+n_{out})$.

Recall The uniform version

Xavier is often written as $\mathcal{U}[-a,a]$. What is $a$? ::: $a=\sqrt{6/(n_{in}+n_{out})}$, because a uniform on $[-a,a]$ has variance $a^2/3$; setting $a^2/3=2/(n_{in}+n_{out})$ gives the $6$.

---

Step 7 — ReLU throws away half the signal

WHAT. So far we assumed the activation is (near-)linear, true for tanh/sigmoid near zero (see Activation Functions (ReLU, tanh, sigmoid)). But ReLU is $\varphi (z)=\max (0,z)$: it deletes every negative value.

WHY it changes variance. If $z$ is symmetric around zero, exactly half its values are negative → set to $0$. Killing half the values (the negative half) halves the average squared size: $\mathrm{Var}(a)=\frac{1}{2}\mathrm{Var}(z)$

• $z$ = pre-activation (symmetric, zero-mean),
• $a=\varphi (z)$ = post-ReLU output (only the positive half survives),
• $\frac{1}{2}$ = the fraction of variance that gets through.

PICTURE. A symmetric bell of $z$ values; the left (negative) half is greyed-out/flattened to zero by ReLU; the surviving right half is shaded green. A bar on the right shows the output variance is exactly half the input variance.

---

Step 8 — Compensate the halving: the He constant

WHAT. With ReLU, the real per-layer gain includes the $\frac{1}{2}$: $g=n_{in}\mathrm{Var}(w)\cdot \frac{1}{2}$ Demand $g=1$ again: $n_{in}\mathrm{Var}(w)\cdot \frac{1}{2}=1⟹\boxed{\mathrm{Var}(w)=\frac{2}{n_{in}}}$

• The $\frac{1}{2}$ = ReLU's damage from Step 7,
• The $2$ in the answer = exactly what undoes that $\frac{1}{2}$.

WHY. ReLU speaks only half the time, so we tell each input to speak twice as loud — that is the entire meaning of the "2" in He/Kaiming initialization. It is a different 2 from Xavier's averaging 2.

PICTURE. Side-by-side bar chart of the per-layer gain $g$: with variance-preserving $1/n_{in}$ and ReLU, $g=0.5$ (signal decays — red); with He's $2/n_{in}$ and ReLU, $g=1.0$ (flat — green). A dashed line at $g=1$ marks the target.

---

Step 9 — Every case, at a glance

Activation	Damage factor	Weight variance	Name
linear / tanh / sigmoid (near 0)	$1$	$\frac{1}{n_{in}}$ (fwd) or $\frac{2}{n_{in}+n_{out}}$ (both)	Xavier
ReLU & variants	$\frac{1}{2}$	$\frac{2}{n_{in}}$	He
all weights = 0	—	—	❌ symmetry never breaks

Degenerate case — zero init. If $\mathrm{Var}(w)=0$, every neuron is identical: same output, same gradient, forever clones. Variance analysis assumes randomness; with zero spread there is nothing to preserve. That is why the recipe is random weights with a controlled variance, never a constant. This complements Batch Normalization, which re-fixes variance during training instead of just at the start, and underpins Deep Network Training Stability.

---

The one-picture summary

PICTURE. The full pipeline in one frame: inputs of variance $1$ → layer gain box $g=n_{in}\mathrm{Var}(w)\cdot (\text{½ if ReLU})$ → demand $g=1$ → branch into Xavier ($2/(n_{in}+n_{out})$, blue path, no halving) and He ($2/n_{in}$, orange path, undoes ReLU's ½). Green output arrow of variance $1$, unchanged through depth.

Recall Feynman: the whole walkthrough in plain words

Variance is just loudness — how far random numbers stray from zero (Step 1). A neuron adds up many little pushes, and because they point randomly they don't all pile up (Step 2). The loudness of the sum equals how many pushes × knob loudness × input loudness (Step 3). Stack layers and this loudness factor gets raised to the depth, so anything but exactly $1$ explodes or dies (Step 4). Force the factor to $1$ and the knob loudness must be $1/n_{in}$ — quieter when there are more inputs (Step 5). The backward pass wants $1/n_{out}$ instead, so Xavier averages the two into $2/(n_{in}+n_{out})$ (Step 6). ReLU deletes the negative half, halving the loudness (Step 7), so we speak twice as loud to compensate: $2/n_{in}$, the He rule (Step 8). And setting everything to zero is banned because clones never learn (Step 9).

---

Connections

• Vanishing and Exploding Gradients — Step 4 is this phenomenon.
• Activation Functions (ReLU, tanh, sigmoid) — decides the damage factor in Step 7.
• Backpropagation — where the backward $n_{out}$ demand of Step 6 comes from.
• Variance and Expectation — Rules A and B of Step 3.
• Batch Normalization — fixes variance during training, not just at init.
• Deep Network Training Stability — the goal all of this serves.