Foundations — Weight initialization (Xavier, He)

This page builds every symbol the parent note throws at you, from absolute zero. If you have never seen $\sum$, $\text{Var}$, $\mathcal{N}$, or $\varphi$ before — start here and read top to bottom. Each block gives you: plain words → the picture → why the topic needs it.

0. The picture the whole topic lives inside

Before any symbol, hold this image in your head: a stack of layers, each turning a list of numbers into a new list of numbers.

• The circles are neurons — each one holds a single number.
• A whole column of circles is a layer — a list of numbers.
• Every arrow carries a number from one neuron to the next, and each arrow has its own weight that scales what passes along it.

Everything below is a tool for answering ONE question: as numbers flow left-to-right through many layers, do they stay a healthy size, or do they blow up / die out?

1. A single number and a list of numbers (the vector)

Picture: a column of boxes, one number per box.

Why the topic needs it: each layer of the network is a vector — the parent note writes $a^{[l-1]}$ and $z^{[l]}$, and those are vectors (whole layers of numbers), not single numbers. Before you can ask "does the layer explode?" you need to see a layer as a list you can measure.

2. Superscripts and subscripts — the address labels

Picture: a filing cabinet. The bracket-superscript names the drawer (which layer), the subscript names the folder inside (which item).

Why the topic needs it: the parent note constantly compares layer $l$ to layer $l-1$ ("variance stays constant across layers"). Without the address labels you can't say which layer's numbers you mean.

3. The weight, the input, and the summation $\sum$

Picture: many arrows feeding one circle; each arrow scales its number, and the circle sums them.

• $x_i$ — the $i$-th input number arriving on an arrow.
• $w_i$ — the weight on that arrow (how strongly it counts).
• $z$ — the pre-activation: the total the neuron receives before it decides its output.

Why the topic needs it: the final answers are literally $\text{Var}(w)=\frac{2}{n_{in}}$ and $\frac{2}{n_{in}+n_{out}}$. Those symbols are just "count the arrows in" and "count the arrows out". The whole derivation is asking: if I add up $n_{in}$ scaled numbers, how big does the total get?

4. Expectation — the "average" $\mathbb{E}$

Picture: a dartboard with darts scattered around a bullseye. $\mathbb{E}[X]$ is the centre of mass of the dart cloud.

Why the topic needs it: the parent note assumes weights and inputs are "zero-mean". This is why we set biases to $0$ and centre the random weights on $0$. Many later simplifications (like $\text{Var}(wx)=\text{Var}(w)\text{Var}(x)$) are only true when the mean is zero — so this symbol earns the rest of the algebra.

See Variance and Expectation for the full toolkit; here we only need the two facts above.

5. Variance $\text{Var}$ — the "how spread out" number

This is the star of the whole topic.

Picture: two dart clouds with the same centre — one tight (small variance), one splattered (big variance).

Why "variance" and not just "size"? We can't track the exact value of every signal — it's random. But we can track how spread out the signal is, and spread is exactly what "explodes" or "vanishes". Variance is the right ruler because it obeys clean addition/multiplication rules for independent zero-mean numbers — the two rules the whole derivation leans on:

Why the topic needs it: the sum rule (variance of a sum = sum of variances) is only valid when the terms are independent — that's why the derivation states "assume i.i.d." out loud before using it.

6. The bell curve $\mathcal{N}$ and the flat box $\mathcal{U}$

Picture: a smooth hill $\mathcal{N}$ next to a flat-topped table $\mathcal{U}$, both symmetric about $0$.

Why the topic needs it: the final formulas say "sample $w\sim \mathcal{N}(0,2/n_{in})$". Now you know that's an instruction: pick each weight from a bell curve of that spread. The symbol $\sim$ means "is drawn from".

7. The activation $\varphi$ — the neuron's decision

Picture: the pre-activation $z$ enters a shaped ramp; the output $a$ comes out reshaped.

See Activation Functions (ReLU, tanh, sigmoid) for their shapes. Here you only need: the activation decides how much variance survives to the next layer, and ReLU is the one that halves it.

Why the topic needs it: the choice of $\varphi$ is why there are two different constants. tanh/sigmoid → Xavier; ReLU → He. The activation is the fork in the road.

8. Forward and backward — the two directions

Picture: two arrows on the same tower — a blue one flowing right (activations), an orange one flowing left (gradients).

Why the topic needs it: Xavier's "$2/(n_{in}+n_{out})$" is a compromise between the forward wish and the backward wish. You can't see why it averages two numbers until you know there are two directions to please. This is exactly the Vanishing and Exploding Gradients problem that init prevents.

Prerequisite map

Everything upstream feeds the parent topic Weight initialization. Batch Normalization and Deep Network Training Stability sit downstream — they build on this same variance idea.

Equipment checklist

Read each question, answer in your head, then reveal.