Weight initialization (Xavier, He): Exercises

This page is a self-test ladder. Each problem is followed by a collapsible solution — try it first, then reveal. We climb from recognising the two rules, to applying them, analysing failure, synthesising whole-network reasoning, and finally mastery (multi-step, edge cases).

Prerequisites you should have from the parent parent note: the two variance rules, the fan-in/fan-out idea, and the variance algebra from Variance and Expectation. Everything else is rebuilt below as needed.

The two rules we test against: $Xavier: Var (w) = \frac{2}{n _{in} + n _{o u t}} He: Var (w) = \frac{2}{n _{in}}$

Figure — Weight initialization (Xavier, He)

The figure above shows the whole logic as a decision tree — glance back at it while solving.

Level 1 — Recognition

Exercise 1.1

A layer uses the ReLU activation. Which initialization rule should you pick, and what is its variance formula?

Recall Solution 1.1

WHAT: ReLU zeroes negative inputs. WHY it matters: killing half the signal halves the variance, so we need the rule that pre-compensates with an extra factor of $2$ . Answer: He / Kaiming initialization, $Var (w) = \frac{2}{n _{in}}$ .

Exercise 1.2

A weight matrix $W$ has shape $256 \times 64$ (rows $\times$ columns), following the convention $z = W x$ . What are $n_{in}$ and $n_{o u t}$ ?

Recall Solution 1.2

With $z = W x$ , each row of $W$ dots into the input vector $x$ , so columns = inputs, rows = outputs.

$n_{in} = 64$ (columns, matches the input dimension).
$n_{o u t} = 256$ (rows, matches how many neurons produce outputs).

Exercise 1.3

True or false: "Initializing all weights to zero is safe because zero can neither explode nor vanish."

Recall Solution 1.3

False. Zero avoids explosion but destroys symmetry breaking: every neuron computes the same output and receives the same gradient, so they update identically forever and stay clones. You need random weights with controlled variance.

Level 2 — Application

Exercise 2.1

A dense layer maps $1024 \to 512$ with ReLU. Compute $Var (w)$ and $σ$ for He initialization.

Recall Solution 2.1

WHY He: ReLU activation. Fan-in is the input dimension $= 1024$ . $Var (w) = \frac{2}{1024} = 0.001953125, σ = 0.001953125 \approx 0.04419$ Sample each weight from $N (0, 0.0441 9^{2})$ .

Exercise 2.2

A tanh layer maps $200 \to 300$ . Give the Xavier normal variance and the Xavier uniform bound $a$ .

Recall Solution 2.2

WHY Xavier: tanh → sum both fans. Here $n_{in} + n_{o u t} = 200 + 300 = 500$ . Normal variance: $Var (w) = \frac{2}{500} = 0.004$ Uniform bound: a flat $U [- a, a]$ has variance $a^{2} /3$ . Set $a^{2} /3 = 0.004$ : $a = 3 \times 0.004 = 0.012 \approx 0.10954$ Sample from $U [- 0.10954, 0.10954]$ .

Exercise 2.3

A layer maps $784 \to 128$ with sigmoid. Give the Xavier normal $σ$ .

Recall Solution 2.3

Sigmoid → Xavier. $n_{in} + n_{o u t} = 784 + 128 = 912$ . $Var (w) = \frac{2}{912} \approx 0.0021930, σ = 0.0021930 \approx 0.046829$

Level 3 — Analysis

Exercise 3.1

A ReLU net has 8 layers, each with $n_{in} = 200$ , but the engineer used standard normal weights ( $Var (w) = 1$ ). Compute the per-layer variance gain $g$ and the total scaling after 8 layers.

Recall Solution 3.1

WHAT is the gain? One ReLU layer multiplies activation variance by $g = n_{in} \cdot Var (w) \cdot \frac{1}{2}$ (the $\frac{1}{2}$ is ReLU halving). Plug in: $g = 200 \cdot 1 \cdot \frac{1}{2} = 100$ WHY the power: each layer multiplies by $g$ , so 8 stacked layers give $g^{8}$ : $10 0^{8} = 1 0^{16}$ The activation variance blows up by ten-quadrillion → exploding. This is the depth-power effect from the parent note.

Exercise 3.2

Same 8-layer, $n_{in} = 200$ ReLU net, but now with He init. Show the total scaling is $1$ .

Recall Solution 3.2

He gives $Var (w) = 2/200 = 0.01$ . $g = 200 \cdot 0.01 \cdot \frac{1}{2} = 1, g^{8} = 1^{8} = 1$ Variance is preserved through every layer → stable. That's exactly the design purpose of the factor $2$ .

Exercise 3.3

Suppose an engineer used Xavier ( $Var (w) = 2/ (n_{in} + n_{o u t})$ with $n_{in} = n_{o u t} = 200$ , so $Var (w) = 2/400 = 0.005$ ) on a ReLU net. Compute the per-layer gain and the total after 20 layers. Does it explode or vanish?

Recall Solution 3.3

$g = n_{in} \cdot Var (w) \cdot \frac{1}{2} = 200 \cdot 0.005 \cdot 0.5 = 0.5$ Gain $< 1$ , so signal shrinks each layer. $g^{20} = 0. 5^{20} = \frac{1}{1048576} \approx 9.5367 \times 1 0^{- 7}$ The activations vanish by a factor of about a million → vanishing (Vanishing and Exploding Gradients). Using Xavier on ReLU under-scales by exactly the missing factor of $2$ .

The figure shows how gain raised to depth diverges (amber) or decays (cyan) away from the stable line $g = 1$ .

Level 4 — Synthesis

Exercise 4.1

Derive the He constant from scratch. Starting from $Var (z) = n_{in} Var (w) Var (x)$ and the ReLU fact $Var (a) = \frac{1}{2} Var (z)$ , find the $Var (w)$ that keeps $Var (a^{[l]}) = Var (a^{[l - 1]})$ .

Recall Solution 4.1

WHAT we track: the input to layer $l$ is $a^{[l - 1]}$ with variance $Var (x)$ . After the linear step: $Var (z^{[l]}) = n_{in} Var (w) Var (x)$ After ReLU: $Var (a^{[l]}) = \frac{1}{2} Var (z^{[l]}) = \frac{1}{2} n_{in} Var (w) Var (x)$ WHY set them equal: to keep the layer output the same size as its input we require $Var (a^{[l]}) = Var (x)$ . Cancel $Var (x)$ : $\frac{1}{2} n_{in} Var (w) = 1 \Rightarrow Var (w) = \frac{2}{n _{in}}$ The $2$ is exactly the reciprocal of ReLU's $\frac{1}{2}$ .

Exercise 4.2

An autoencoder has an encoder layer $784 \to 32$ (ReLU) and a decoder layer $32 \to 784$ (tanh output). Choose the initialization variance for each layer.

Recall Solution 4.2

Each layer is decided independently by its own activation and its own fans.

Encoder $784 \to 32$ , ReLU → He, $n_{in} = 784$ : $Var (w) = \frac{2}{784} \approx 0.0025510$
Decoder $32 \to 784$ , tanh → Xavier, $n_{in} + n_{o u t} = 32 + 784 = 816$ : $Var (w) = \frac{2}{816} \approx 0.0024510$ Notice they came out numerically close by coincidence — but the reasons differ (ReLU-halving vs symmetric-sum).

Exercise 4.3

For the forward goal we found $Var (w) = 1/ n_{in}$ ; for the backward goal $Var (w) = 1/ n_{o u t}$ . Show that Xavier's $2/ (n_{in} + n_{o u t})$ is the harmonic-style compromise, and check it reduces to both when $n_{in} = n_{o u t}$ .

Recall Solution 4.3

WHAT Xavier does: instead of choosing forward OR backward, it averages the two requirements. Writing $\frac{1}{Var ( w )}$ as the "constraint number":

forward demands $\frac{1}{Var ( w )} = n_{in}$
backward demands $\frac{1}{Var ( w )} = n_{o u t}$

Average the two demands: $\frac{1}{Var ( w )} = \frac{n _{in} + n _{o u t}}{2}$ , hence $Var (w) = \frac{2}{n _{in} + n _{o u t}} .$ Edge check $n_{in} = n_{o u t} = n$ : $\frac{2}{n + n} = \frac{1}{n}$ , which matches both the forward $1/ n_{in}$ and backward $1/ n_{o u t}$ exactly. So when fans are equal there is no compromise at all — Xavier is optimal for both directions.

Level 5 — Mastery

Exercise 5.1

A network has three ReLU layers with fan-ins $n_{in} = 100, 400, 900$ . He init is used everywhere. Compute each layer's $σ$ , and confirm the total activation-variance gain across all three layers is $1$ .

Recall Solution 5.1

He variance per layer $= 2/ n_{in}$ :

Layer 1: $Var (w) = 2/100 = 0.02$ , $σ = 0.02 \approx 0.14142$
Layer 2: $Var (w) = 2/400 = 0.005$ , $σ = 0.005 \approx 0.070711$
Layer 3: $Var (w) = 2/900 \approx 0.0022222$ , $σ \approx 0.047140$

Per-layer gain $g_{ℓ} = n_{in} \cdot Var (w) \cdot \frac{1}{2} = n_{in} \cdot \frac{2}{n _{in}} \cdot \frac{1}{2} = 1$ for every layer regardless of $n_{in}$ . Total gain $= 1 \cdot 1 \cdot 1 = 1$ → perfectly stable. He self-corrects at each layer independently.

Exercise 5.2

Degenerate/edge case. A "layer" has $n_{in} = 1$ (a single input feature) and uses ReLU. What is the He $σ$ ? Then explain what happens to variance preservation, given that a single input has no averaging.

Recall Solution 5.2

He: $Var (w) = 2/1 = 2$ , so $σ = 2 \approx 1.41421$ . The formula still targets $g = 1$ in expectation: $g = n_{in} \cdot Var (w) \cdot \frac{1}{2} = 1 \cdot 2 \cdot \frac{1}{2} = 1$ . ✔ BUT the variance-preservation argument relied on summing many independent terms so the law-of-large-numbers smooths fluctuations. With $n_{in} = 1$ there is a single term $w_{1} x_{1}$ — no averaging — so the actual $z$ has huge relative fluctuation even though its variance equals $Var (x)$ on average. Lesson: the theory is asymptotic in fan-in; tiny fans are noisy but the mean scaling is still correct.

Exercise 5.3

Mixed net + backward stability. A layer maps $n_{in} = 500 \to n_{o u t} = 1500$ with tanh. Using Xavier, compute the forward gain $g_{f} = n_{in} Var (w)$ and the backward gain $g_{b} = n_{o u t} Var (w)$ . Interpret why neither is exactly $1$ .

Recall Solution 5.3

Xavier: $Var (w) = \frac{2}{500 + 1500} = \frac{2}{2000} = 0.001$ .

Forward gain: $g_{f} = 500 \cdot 0.001 = 0.5$
Backward gain: $g_{b} = 1500 \cdot 0.001 = 1.5$

Interpretation: because $n_{in} \neq = n_{o u t}$ , the single variance cannot satisfy both directions. Xavier lands between them — signal shrinks slightly forward ( $0.5$ ) and grows slightly backward ( $1.5$ ), and their geometric mean $0.5 \cdot 1.5 = 0.75 \approx 0.866$ is close to $1$ . That controlled split is the whole point of the "average the two demands" compromise, connecting to Backpropagation gradient flow.

Exercise 5.4

Synthesis with normalization. A colleague says "we use Batch Normalization after every layer, so weight init doesn't matter." Give a one-line quantitative reason init still helps at the very start of training.

Recall Solution 5.4

Batch norm re-centres/re-scales activations after the linear step, but at step 0 the pre-norm activations still pass through the nonlinearity; a wildly scaled init (e.g. gain $128$ per layer, as in Exercise 3.1) can push ReLU/tanh into saturated or dead regions before BN's running statistics settle, producing dead units and unstable first gradients. Good init keeps the initial forward pass in a sane range so BN starts from healthy statistics — the two are complementary, not redundant. See Deep Network Training Stability.

Connections

Vanishing and Exploding Gradients — the exact failure computed in L3.
Activation Functions (ReLU, tanh, sigmoid) — decides He vs Xavier every time.
Batch Normalization — the complement discussed in Exercise 5.4.
Backpropagation — source of the backward gain in Exercise 5.3.
Variance and Expectation — the algebra behind every derivation.
Deep Network Training Stability — the umbrella goal.
← Back to the parent topic