Multi-layer perceptron architecture: Question bank

Before we start, we lock down the vocabulary this whole page leans on. Read these once — every trap below reuses them.

Definition The symbols we reuse everywhere

pre-activation $z$ — the plain weighted sum $\sum_{i} w_{i} a_{i} + b$ before any bending. Think of it as a raw score on a number line.
activation $a = σ (z)$ — that score after being pushed through a shape-bending function $σ$ (like ReLU or sigmoid). This is what actually leaves the neuron.
weight matrix $W$ — a rectangular grid of numbers holding all the weights of one layer at once. If a layer turns a vector of $n$ inputs into $m$ outputs, then $W$ is an $m \times n$ grid and the whole layer is $z = W a + b$ . When we stack layers we number them $W_{1}, W_{2}, \dots$ (one grid per layer). Look at the picture below.
$K$ — the number of classes (equivalently, the number of output neurons / logits) in a classification task, e.g. $K = 10$ for the digits $0$ – $9$ .
monotonic — a function that never reverses direction: as its input goes up, its output only ever goes up (or only ever goes down). It preserves the order of its inputs.

Figure — Multi-layer perceptron architecture

The figure shows one layer as a $W$ -grid multiplying an input vector — trace the blue arrow: each output neuron reads a whole row of $W$ . Keep pre-activation ( $z$ ) and activation ( $a$ ) apart in your head; half the traps below live in the gap between them.

True or false — justify

Every item is Claim ::: verdict + reason. Read the claim, decide, then reveal.

Stacking two linear layers with no activation is more powerful than one linear layer

False. With

W_{1}, W_{2}

the two layers' weight matrices,

W_{2} (W_{1} x) = (W_{2} W_{1}) x

, and

W_{2} W_{1}

is just one matrix, so the two linear layers collapse into a single linear map — you gain nothing but wasted parameters. See 3.1.04-Activation-Functions.

An MLP with one hidden layer can approximate any continuous function on a bounded region

True, that is the Universal Approximation Theorem — but it says a set of weights exists, not that it is small or that gradient descent will find it.

Because the input layer is called a "layer", it performs a weighted-sum computation like the others

False. The input layer only holds the raw feature values

x

and passes them forward; it has no weight matrix

W

, no bias, and no activation.

A neuron in a fully connected layer receives input from exactly one neuron in the previous layer

False. "Fully connected" means every neuron in layer

l

receives from all neurons in layer

l - 1

; that is the whole point of the weighted sum

\sum_{i} w_{j i} a_{i}

(one weight per incoming neuron).

Softmax and sigmoid give the same result on a single output neuron

Half-true trap. Softmax over one logit always returns

1

(it normalizes over itself), so it is useless there; sigmoid on one logit gives a genuine probability in

(0, 1)

. That is why binary tasks use sigmoid, not softmax.

Adding more hidden layers always lowers test error

False. Extra depth adds capacity, but past a point it overfits (memorizes noise) and can also become harder to train; see 3.3.02-Dropout-Regularization.

The bias term

b

changes the slope of a neuron's decision boundary

False. The weights (the entries of

W

) set the slope/orientation; the bias shifts the boundary along the number line without tilting it. Removing bias forces every boundary through the origin.

Widening a layer (more neurons) is generally more parameter-efficient than deepening (more layers)

False in practice. For the same parameter budget, deeper-but-narrower networks usually generalize better because they reuse features across levels of abstraction.

Spot the error

Each item states a flawed piece of reasoning; the reveal names the bug.

"XOR is unsolvable by neural networks, so we need a different model."

The error is scope: a single-layer perceptron cannot solve XOR (it is not linearly separable), but an MLP with one hidden layer can. See 3.1.01-SinglePerceptron-and-Linear-Separability.

"I'll use ReLU on my final layer for a regression that must output negative temperatures."

ReLU clamps everything below zero to

0

, so it can never output a negative value; a regression output needs linear (identity) activation on the last layer.

"My 10-class classifier outputs

[2.1, 0.3, \dots]

; those are the class probabilities."

Those are logits (raw pre-activations

z

), not probabilities — they are unbounded and don't sum to

1

. You must pass all

K

of them through softmax first.

"Sigmoid saturates, so I'll just make its inputs huge to get sharper decisions."

Backwards. Large

∣ z ∣

is exactly where sigmoid's gradient collapses toward

0

, causing vanishing gradients so the neuron barely learns — see 3.1.03-Backpropagation-Algorithm.

"To count parameters in a 10→16 layer I compute

10 \times 16 = 160

You forgot the biases: the weight matrix

W

holds

10 \times 16

entries, plus one bias per output neuron, so

10 \times 16 + 16 = 176

. Every layer's parameter count is (weights + biases).

"ReLU is non-linear, so a single ReLU neuron can carve any curved boundary."

One ReLU is only piecewise linear (a flat part and a sloped part); rich curves emerge only when many such kinks from many neurons are summed together.

"Softmax outputs must each be large for the network to be confident."

Confidence is about contrast, not magnitude — one output near

1

while the rest are near

0

is confident; softmax always makes all

K

outputs sum to

1

regardless.

"I removed all activation functions to speed up training; the math is simpler."

You destroyed the model's power: with no non-linearity the whole MLP collapses to one linear map

W_{combined} x

and can only fit linearly separable data.

Why questions

Why does an MLP need a non-linear activation and not just any function?

A linear function composed with linear layers stays linear, so it can't add expressive power; the non-linearity is what lets stacked layers represent curved, non-linearly-separable boundaries.

Why is exponentiation the right tool in softmax?

Two jobs at once. Positivity:

e^{z}

is always

> 0

for any real

z

, so every logit becomes a valid unnormalized probability. Amplifying contrast:

e^{z}

grows faster than

z

itself, so a small lead in a logit becomes a large lead after exponentiating — the top class dominates. A plain "divide each logit by the sum" fails both jobs, since logits can be negative (giving negative or ill-defined "probabilities") and their raw gaps are not amplified.

Why do implementations subtract the largest logit before computing

e^{z}

For numerical stability. Big logits make

e^{z}

overflow to

\infty

(and very negative ones underflow to

0

). Subtracting the max logit

z_{m a x}

shifts every exponent to

\leq 0

, so

e^{z - z_{m a x}} \in (0, 1]

— safe to compute. The result is identical because the shared factor

e^{- z_{m a x}}

cancels in numerator and denominator (this is the log-sum-exp trick).

Why does backpropagation care about the derivative of the activation, not just its value?

Learning multiplies gradients layer by layer through each

σ

's slope. Sigmoid's derivative is

σ (z) (1 - σ (z))

, which peaks at just

0.25

and vanishes when saturated — repeatedly multiplying such small numbers shrinks the signal (vanishing gradients). ReLU's derivative is a clean piecewise

1

for

z > 0

and

0

for

z < 0

, which passes the gradient through undamped where active — that is why ReLU trains deep nets better. See 3.1.03-Backpropagation-Algorithm.

Why are hidden layers called "hidden"?

They never touch the outside world directly — we neither feed them raw data nor read their outputs as predictions; they are internal feature extractors between input and output.

Why can Leaky ReLU rescue a "dead" neuron that plain ReLU cannot?

Plain ReLU has zero derivative for

z < 0

, so a neuron stuck negative gets no update signal; Leaky ReLU keeps a small slope (

0.01 z

, derivative

0.01

) there, letting a gradient still flow and revive it.

Why do we prefer the matrix form

z = W a + b

over looping neuron-by-neuron?

It computes an entire layer's neurons in one parallel operation, which hardware and libraries (NumPy/PyTorch) execute far faster than an explicit Python loop.

Why does the output layer's size depend on the task but hidden-layer size is a design choice?

Output size is fixed by the answer shape — 1 for binary,

K

for

K

classes, 1 for scalar regression — while hidden size is a tunable capacity knob you choose.

Why does a deeper network often need fewer total neurons than a shallow one for the same task?

Depth lets later layers reuse and combine features built by earlier ones (edges → shapes → objects), so it composes complexity instead of enumerating every pattern flatly. See 4.1.01-Convolutional-Neural-Networks.

Edge cases

What is the output of a ReLU neuron when its pre-activation

z

is exactly

0

Exactly

0

, since

max (0, 0) = 0

; at that kink the derivative is undefined, so libraries just pick

0

(or sometimes

1

) as a convention.

What happens to a sigmoid output as

z \to + \infty

and as

z \to - \infty

It saturates: approaching

1

from below as

z \to + \infty

and

0

from above as

z \to - \infty

, never quite reaching either — which is exactly where its gradient dies.

What goes numerically wrong if you compute softmax on logits like

z = 1000

directly, and how is it fixed?

e^{1000}

overflows to

\infty

(and very negative logits underflow to

0

), giving

\infty/\infty =

NaN. The fix is to subtract the maximum logit

z_{m a x}

first, so every exponent is

\leq 0

and safe — the answer is unchanged.

If every weight and bias in a layer is initialized to the same value, what breaks?

Symmetry: all neurons in the layer compute identical outputs and receive identical gradients, so they update in lockstep and can never learn different features — this is why we use random initialization.

For a single-neuron regression output with linear activation, what values can

\overset{y}{^}

take?

Any real number in

(- \infty, + \infty)

, because a linear (identity) activation imposes no bound — unlike sigmoid or ReLU.

What does softmax return when all

K

logits are equal?

A perfectly uniform distribution

[\frac{1}{K}, \dots, \frac{1}{K}]

— maximal uncertainty, the network expressing "no preference."

If a hidden layer has

0

neurons, what is the network?

There is no layer at all — the connection collapses and the input must map straight to the next layer, so you have effectively removed a layer (and possibly the model's non-linear power).

What happens to the parameter count if you double a hidden layer's width from

n

2 n

, where

n

is that layer's original number of neurons, sitting between a previous layer of size

a

(neurons feeding in) and a next layer of size

b

(neurons it feeds)?

Both the incoming block (

a \cdot n

weights) and the outgoing block (

n \cdot b

weights), plus its biases, roughly double, so parameters scale about linearly in that layer's width

n

— a real cost worth weighing against overfitting risk.

Recall Quick self-check

The pre-activation is called ::: $z$ , the weighted sum before the activation function. The single tool that keeps stacked layers from collapsing into one linear map is ::: a non-linear activation function $σ$ . Binary classification uses which output activation ::: sigmoid. Multi-class classification uses which output activation ::: softmax over the $K$ output neurons. Sigmoid's derivative peaks at what value ::: $0.25$ , which is why deep sigmoid stacks suffer vanishing gradients.

Related deep dives: 3.1.04-Activation-Functions · 3.1.03-Backpropagation-Algorithm · 3.2.01-Gradient-Descent-Optimization · 2.1.05-Feature-Engineering.