Exercises — Multi-layer perceptron architecture

• $\mathbf{x}$ ::: the input feature vector, a column of numbers fed into the network.
• $W^{(l)}$ ::: the weight matrix of layer $l$. Row $j$, column $i$ holds $w_{ji}^{(l)}$, the strength of the wire from neuron $i$ (previous layer) to neuron $j$ (this layer).
• ${\mathbf{b}}^{(l)}$ ::: the bias vector of layer $l$, one number per neuron, that shifts the decision.
• ${\mathbf{z}}^{(l)}=W^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$ ::: the pre-activation (weighted sum before the non-linearity).
• ${\mathbf{a}}^{(l)}=\sigma ({\mathbf{z}}^{(l)})$ ::: the activation (after the non-linearity $\sigma$).
• logits ::: the raw output-layer scores ${\mathbf{z}}^{(L)}$ before softmax — real numbers of any size and sign, not yet probabilities.
• $\sigma$ ::: activation function — ReLU, sigmoid, tanh, softmax. See 3.1.04-Activation-Functions.

(a) input layer — it only holds and forwards the raw feature vector $\mathbf{x}$; no weights act inside it. (b) output layer — "sum to 1" is the fingerprint of softmax, used for multi-class predictions. (c) hidden layer — "hidden" means it never touches the outside world; it only reshapes data for the next layer.

Each number is the count of neurons in that layer, read left (input) to right (output).

• $784$ = input neurons (one per pixel)
• $128$ and $64$ = the two hidden layers
• $10$ = output neurons (one per digit class)

So there are 2 hidden layers. The arrows $\to$ mean "fully connected, information flows forward only."

• House price → linear (1 neuron). Prices are unbounded real numbers; squashing them into $[0,1]$ would be wrong.
• Spam/not-spam → sigmoid (1 neuron). Two classes = one probability $p\in [0,1]$; the other class is $1-p$.
• 5 animal types → softmax (5 neurons). Mutually exclusive classes need probabilities that sum to 1.

What we do: the weighted-sum-plus-bias, exactly $z={\sum }_i{w}_i{a}_i+b$. $z=(0.5)(2)+(1.0)(-1)+(-0.5)(3)+0.5=1-1-1.5+0.5=-1.0$ Then the non-linearity: $\text{ReLU}(z)=\max (0,z)=\max (0,-1.0)=0$. Why the answer is 0: ReLU kills all negative pre-activations. This neuron is "off" for this input — a completely normal event, not an error.

$z=(0.5)(2)+(1.0)(1)+(-0.5)(1)+0.5=1+1-0.5+0.5=2.0$ $\text{ReLU}(2.0)=\max (0,2.0)=2.0$. Positive input passes straight through unchanged — that is ReLU's whole personality on the positive side.

Why sigmoid here: it squashes any real number into $(0,1)$ so we can read the output as a probability.

• $z=0$: $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{1+1}=0.5$. The midpoint — maximum uncertainty.
• $z=2$: $\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.1353}\approx 0.8808$.

Step 1 — exponentiate. $e^{z_i}$ makes every value positive (probabilities can never be negative) and stretches the gaps so a slightly bigger logit becomes a noticeably bigger score. $e^1=2.7183,e^2=7.3891,e^0=1.0000,\text{sum}=11.1074$ Step 2 — divide by the sum (normalize). Why normalize? Three positive numbers are not yet a probability distribution — to be one, they must add up to exactly 1 (something is certain to happen). Dividing each $e^{z_i}$ by their total ${\sum }_j{e}^{z_j}$ forces exactly that: the ratios are preserved but the pieces now sum to 1, turning raw scores into shares of a whole. $\text{softmax}=\left[\frac{2.7183}{11.1074},\frac{7.3891}{11.1074},\frac{1.0000}{11.1074}\right]\approx [0.2447,0.6652,0.0900]$ Check: $0.2447+0.6652+0.0900=0.9999\approx 1$. ✓ (exactly 1 before rounding) Highest probability is index 1 (0.6652), so we predict class 1.

The rule: a layer with $n$ inputs and $m$ neurons has $m\times n$ weights (the matrix $W$) plus $m$ biases.

• Layer $10\to 16$: $16\times 10+16=160+16=176$
• Layer $16\to 8$: $8\times 16+8=128+8=136$
• Layer $8\to 1$: $1\times 8+1=8+1=9$ $\text{Total}=176+136+9=321$ This matches Example 1 in the parent note. Notice the biases ($16+8+1=25$) are a small slice — most parameters live in the weight matrices.

Substitute the first equation into the second: ${\mathbf{a}}^{(2)}=W^{(2)}\left(W^{(1)}\mathbf{x}\right)=\left(W^{(2)}{W}^{(1)}\right)\mathbf{x}=W_{\text{combined}}\mathbf{x}$ Why this matters: matrix multiplication is associative, so two matrices multiply into one matrix $W_{\text{combined}}$. The whole two-layer stack is equivalent to a single linear layer with weights $W_{\text{combined}}$. No amount of stacking adds power without a non-linear $\sigma$ in between. This is the entire reason activation functions exist — see 3.1.04-Activation-Functions and the single-perceptron limits.

A single perceptron draws exactly one straight line (a linear decision boundary). XOR wants the two blue corners on one side and the two amber corners on the other — no single line can do that. A hidden layer supplies two lines (two hidden neurons, each drawing its own boundary — the dashed cyan lines in the figure). The output neuron then combines the two half-plane answers with a non-linearity, carving out the bent region that isolates the amber class. Geometrically: hidden neurons draw simple lines; the output neuron glues those lines into a non-linear boundary. That composition of simple pieces into a complex shape is the whole point of "multi-layer."

• $784\to 128$: $128\times 784+128=100,352+128=100,480$
• $128\to 64$: $64\times 128+64=8,192+64=8,256$
• $64\to 10$: $10\times 64+10=640+10=650$ $\text{Total}=100,480+8,256+650=109,386$ Design reasoning: the funnel shape $784\to 128\to 64\to 10$ compresses raw pixels into progressively more abstract features (strokes → loops → digit identity). The first layer dominates the parameter budget because 784 inputs are expensive — a hint at why CNNs replace this dense first layer with shared filters.

Parameters as a function of $h$: $P(h)=\underset{\text{input}\to \text{hidden}}{\underbrace{(h\times 4+h)}}+\underset{\text{hidden}\to \text{output}}{\underbrace{(2\times h+2)}}=5h+2h+2=7h+2$ Solve $7h+2\le 100\Rightarrow h\le 14$. Take the largest fitting value $h=14$: $P(14)=7(14)+2=98+2=100$ Exactly on budget. A neat coincidence that $h=14$ lands on 100; had it not, we'd pick the largest $h$ below budget.

Output: $\text{ReLU}(-3)=\max (0,-3)=0$. The neuron always outputs 0. Gradient: ReLU's slope is $0$ for $z<0$ and $1$ for $z>0$. At $z=-3$ the slope is $0$. Why dead: during backpropagation, the weight update is proportional to this local slope. A slope of $0$ means no gradient flows back, so the weights never change, so $z$ stays negative forever — a self-locking trap. Leaky ReLU rescue: $\sigma (z)=\max (0.01z,z)$ has slope $0.01$ (not $0$) for $z<0$. At $z=-3$ the output is $0.01\times (-3)=-0.03$ and the slope is $0.01$. That tiny non-zero gradient lets the weights slowly crawl back to life. See 3.1.04-Activation-Functions and gradient descent.

Proof: $\text{softmax}(z_i-c)=\frac{e^{z_i-c}}{{\sum }_j{e}^{z_j-c}}=\frac{e^{z_i}{e}^{-c}}{e^{-c}{\sum }_j{e}^{z_j}}=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}=\text{softmax}(z_i)$ The factor $e^{-c}$ cancels top and bottom. Why we care: raw logits like $1000$ make $e^{1000}$ overflow to infinity. Subtracting $c=\max (z)$ makes the largest exponent $e^0=1$, keeping every number safe — same probabilities, no overflow. Numerical check with shifted logits $[1-2,2-2,0-2]=[-1,0,-2]$: $e^{-1}=0.3679,e^0=1.0000,e^{-2}=0.1353,\text{sum}=1.5032$ $[0.2447,0.6652,0.0900]$ Identical to Exercise 2.4's unshifted result. ✓

Every $e^{z_i}=e^c$ is identical, so: $\text{softmax}(z_i)=\frac{e^c}{K{e}^c}=\frac{1}{K}$ For $K=10$ that is $0.1$ for each class. Interpretation: this is maximal uncertainty — the network is guessing uniformly and has learned nothing that distinguishes one class from another. This is exactly the output of a freshly initialized network (before training pushes the logits apart) and it makes a useful sanity baseline: a 10-class model should start near $10\%$ accuracy. Notice too that the constant $c$ vanished entirely — it cancels top and bottom — so only the differences between logits matter to softmax, never their absolute size (the same fact that powered Exercise 5.2).

Which formula gives a single neuron's pre-activation? ::: $z={\sum }_i{w}_i{a}_i+b$ How many parameters does a layer with $n$ inputs and $m$ neurons have? ::: $m\times n+m$ What is a logit? ::: the raw output score before softmax — a real number, not yet a probability. Why must there be a non-linearity between layers? ::: without it, stacked linear layers collapse to one matrix $W_{\text{combined}}$ (Ex 3.2). What does softmax on all-equal logits give? ::: $1/K$ for every class — pure uncertainty.