Exercises — Forward propagation computation

$\mathbf{b}$ (the bias) is added. Adding — not multiplying — means it shifts the pre-activation up or down. Consequence: a neuron can output a nonzero value even when every input $x_i=0$. Geometrically its decision boundary need not pass through the origin. (Multiplying instead would only scale the inputs — see the L1 mistake below.)

• $\mathbf{x}$ is $5\times 1$ (the $n$ inputs).
• $W$ is $m\times n=3\times 5$ — output rows, input columns, so that $W\mathbf{x}$ produces $3$ numbers.
• $\mathbf{b}$ is $3\times 1$ (one bias per neuron).
• $\mathbf{z}=W\mathbf{x}+\mathbf{b}$ is $3\times 1$.

Sanity chain: $(3\times 5)(5\times 1)=(3\times 1)$, and adding $\mathbf{b}$ ($3\times 1$) keeps it $3\times 1$. ✔

• Output layer: softmax. Reason: it turns 3 raw scores into 3 positive numbers that sum to $1$ — a valid probability distribution over the classes.
• Hidden layer: ReLU (or tanh). Reason: it is a cheap per-unit nonlinearity that breaks the linear collapse without coupling units together. See Activation functions.

Weigh + Bias: $z=0.4(1)+(-0.6)(2)+1.0(-1)+0.2=0.4-1.2-1.0+0.2=-1.6$ Bend (ReLU clips negatives to 0): $a=\max (0,-1.6)=0$ The neuron is "dead" for this input — ReLU gated it shut.

Each row of $W$ dotted with the input, plus that row's bias: ${\mathbf{z}}^{[1]}=\left[\begin{array}{c}1(2)+(-1)(1)\\ 0.5(2)+2(1)\\ -3(2)+1(1)\end{array}\right]+\left[\begin{array}{c}0\\ -1\\ 2\end{array}\right]=\left[\begin{array}{c}1\\ 3\\ -5\end{array}\right]+\left[\begin{array}{c}0\\ -1\\ 2\end{array}\right]=\left[\begin{array}{c}1\\ 2\\ -3\end{array}\right]$ Apply ReLU element-wise: ${\mathbf{a}}^{[1]}=\left[\begin{array}{c}\max (0,1)\\ \max (0,2)\\ \max (0,-3)\end{array}\right]=\left[\begin{array}{c}1\\ 2\\ 0\end{array}\right]$

Exponentiate each logit (why: make them positive and monotonic in the score): $e^0=1,e^2=7.389,e^{-1}=0.368,\text{sum}=8.757$ Divide by the sum (why: force them to add to $1$): $\hat{y}=\left[\begin{array}{c}1/8.757\\ 7.389/8.757\\ 0.368/8.757\end{array}\right]=\left[\begin{array}{c}0.114\\ 0.844\\ 0.042\end{array}\right]$ Predicted class = index 1 (the largest logit ⇒ the largest probability).

Substitute the first into the second: ${\mathbf{z}}^{[2]}=W_2(W_1\mathbf{x}+{\mathbf{b}}_1)+{\mathbf{b}}_2=\underset{W_{\text{eff}}}{\underbrace{(W_2{W}_1)}}\mathbf{x}+\underset{{\mathbf{b}}_{\text{eff}}}{\underbrace{(W_2{\mathbf{b}}_1+{\mathbf{b}}_2)}}$ So ${\mathbf{z}}^{[2]}=W_{\text{eff}}\mathbf{x}+{\mathbf{b}}_{\text{eff}}$ — one linear map. Consequence: depth without a nonlinearity buys zero extra expressive power. This is exactly why activations are non-optional, and why the Universal approximation theorem needs a nonlinear $\varphi$. See Matrix multiplication for why $W_2{W}_1$ is itself a single matrix.

Layer 1 pre-activation: ${\mathbf{z}}^{[1]}=\left[\begin{array}{c}0.5(3)+(-0.5)(0)+0\\ 1(3)+1(0)-1\end{array}\right]=\left[\begin{array}{c}1.5\\ 2\end{array}\right]$ ReLU: both positive, unchanged: ${\mathbf{a}}^{[1]}=\left[\begin{array}{c}1.5\\ 2\end{array}\right]$. Layer 2: $z^{[2]}=2(1.5)+(-1)(2)+0.5=3-2+0.5=1.5$, so $\hat{y}=1.5$.

What changed: for $\left[\begin{array}{c}1\\ 2\end{array}\right]$ neuron 1 had $z=-0.5<0$, so ReLU killed it — the network was in a region where only neuron 2 was active (one linear piece). For $\left[\begin{array}{c}3\\ 0\end{array}\right]$ both neurons are active — a different linear piece. ReLU networks are stitched together from such flat pieces, one per on/off pattern of the hidden units. See figure.

Not legal. $AW$ with $A$ being $6\times 8$ and $W$ being $4\times 6$ requires the inner dimensions to match ($8$ vs $4$) — they don't. The correct batch form keeps $W$ on the left: $Z=WA+\mathbf{b},(4\times 6)(6\times 8)=(4\times 8)$ Inner dims $6=6$ match ✔. $Z$ is $4\times 8$. The bias $\mathbf{b}$ ($4\times 1$) is broadcast across all $8$ columns. Rule of thumb: the weight matrix always left-multiplies the data, and its column count equals the input size $n$.

Hidden pre-activation: ${\mathbf{z}}^{[1]}=\left[\begin{array}{c}1(1)+0(-2)\\ 0(1)+1(-2)\end{array}\right]+\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}1\\ -2\end{array}\right]+\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}1\\ -1\end{array}\right]$ ReLU: ${\mathbf{a}}^{[1]}=\left[\begin{array}{c}\max (0,1)\\ \max (0,-1)\end{array}\right]=\left[\begin{array}{c}1\\ 0\end{array}\right]$. Output logits: ${\mathbf{z}}^{[2]}=\left[\begin{array}{c}1(1)+(-1)(0)\\ 0(1)+2(0)\end{array}\right]+\left[\begin{array}{c}0\\ 0\end{array}\right]=\left[\begin{array}{c}1\\ 0\end{array}\right]$ Softmax: $e^1=2.718,e^0=1,\text{sum}=3.718$: $\hat{y}=\left[\begin{array}{c}2.718/3.718\\ 1/3.718\end{array}\right]=\left[\begin{array}{c}0.731\\ 0.269\end{array}\right]$ Predicted class 0 with probability $0.731$.

Failure: sigmoid squashes its output into $(0,1)$. A house price can be $500,000$ — the sigmoid can never reach it; it saturates at $1$. Worse, in the saturated region the gradient $\to 0$, so learning stalls (see Vanishing and exploding gradients). Correct choice: the identity (linear) output activation for regression, so $\hat{y}=z^{[L]}$ can take any real value. Hidden layers can still use ReLU to provide nonlinearity; only the final layer's range must match the target's range.

Each of the $2$ ReLU units is independently on ($z>0$) or off ($z\le 0$): $2^2=4$ possible patterns, hence at most 4 linear pieces.

• For $\left[\begin{array}{c}1\\ 2\end{array}\right]$: ${\mathbf{z}}^{[1]}=\left[\begin{array}{c}-0.5\\ 2\end{array}\right]$ ⇒ pattern (off, on).
• For $\left[\begin{array}{c}3\\ 0\end{array}\right]$: ${\mathbf{z}}^{[1]}=\left[\begin{array}{c}1.5\\ 2\end{array}\right]$ ⇒ pattern (on, on).

Two different patterns ⇒ two different linear pieces, exactly the piecewise-linear behaviour of ReLU nets. More units ⇒ exponentially more possible pieces ⇒ more expressive power, connecting to the Universal approximation theorem.

(a) Kink location: ReLU switches at $w_{1}x+b_1=0\Rightarrow 2x-4=0\Rightarrow x=2$. (b) Two pieces:

• For $x\le 2$ (ReLU off, output $0$): $\hat{y}=w_2\cdot 0+b_2=1$ (flat).
• For $x>2$ (ReLU on): $\hat{y}=3(2x-4)+1=6x-11$.

Evaluate:

• $x=1$: on the flat side ⇒ $\hat{y}=1$.
• $x=5$: on the sloped side ⇒ $\hat{y}=6(5)-11=19$.

The function is a hinge: flat then rising, joined at the kink $x=2$. See figure.

The forward pass computes every ${\mathbf{z}}^{[\ell ]}$ and ${\mathbf{a}}^{[\ell ]}$ and finally the prediction $\hat{y}$, from which a loss is measured. Backpropagation then computes how the loss changes with respect to each weight by applying the chain rule backwards through the very same layers. But the chain rule's local derivatives (e.g. ReLU's slope $1[z>0]$, or the softmax Jacobian) are evaluated at the values ${\mathbf{z}}^{[\ell ]},{\mathbf{a}}^{[\ell ]}$ produced during the forward pass. So those cached values are the raw material backprop consumes — you cannot differentiate at a point you never computed. Forward first, backward second: the same pipeline, run in two directions.

Every exercise was the same three moves — Weigh, Bias, Bend — done carefully: get the shapes right, exponentiate before you normalise, keep $W$ on the left, and remember ReLU turns the network into linear pieces glued at kinks the weights and biases choose.