3.1.3 · D5Neural Network Fundamentals
Question bank — Forward propagation computation
Before you start, recall the one recipe every layer runs: weighted sum, add bias, bend through an activation — , then . Almost every trap below is a way of quietly breaking one of those three moves.
True or false — justify
A network made of 50 linear layers with no activations can still learn very complex curves because it is deep.
False. Composing linear maps gives another linear map, , so 50 layers do exactly what 1 layer does — the depth buys nothing without a nonlinearity.
The bias term lets a neuron produce a nonzero output even when every input is zero.
True. When all the weighted sum is , so ; the bias sets the neuron's resting/default value and shifts its firing threshold.
Softmax should be placed after every hidden layer to keep the numbers "normalized."
False. Softmax ties all units into one probability simplex (they must sum to 1), which throttles information between hidden layers; it belongs only at a classification output. See Softmax and cross-entropy loss.
In batch form , the same bias vector is reused for every example in the batch.
True. The bias is broadcast across all columns of — each example gets the identical shift, because the neuron's bias is a property of the neuron, not of the input.
Forward propagation already includes the learning of the weights.
False. Forward propagation only computes the prediction for fixed weights; adjusting the weights is Backpropagation plus an optimizer. Forward is "run the network," not "train it."
ReLU is nonlinear, but it is linear on each side of zero, so a ReLU network is piecewise-linear overall.
True. Each fixed pattern of which ReLUs are on/off gives a linear map on that input region; the network stitches many such linear pieces into a bent surface.
Swapping for (multiplying in the other order) gives the same result as long as shapes fit.
False. Matrix multiplication is not commutative; the row-of--dots-input structure is what makes row equal neuron 's weighted sum, and reversing it computes something else entirely.
For a regression network the output activation is often just the identity.
True. Regression targets can be any real number, so squashing the output through sigmoid/softmax would wrongly restrict the range; identity lets be unbounded.
Spot the error
"A layer has inputs and units, so its weight matrix is ."
The multiply must output an vector, so is ==== (output rows, input columns). times a vector is shape-illegal.
"The bias scales the weighted sum, so I write ."
The bias is added, not multiplied: . It shifts the pre-activation; it never rescales it.
"Since deeper is better, I'll stack ten Dense layers with no activation between them for a strong feature extractor."
Ten activation-free linear layers collapse to a single linear map; you need a nonlinearity like ReLU between them or the extra layers are wasted parameters.
" but I forgot the activation on the hidden layer, that's fine — I only care about the output."
A missing hidden activation turns that block linear, silently reducing expressive power upstream of the output; every layer's nonlinearity matters, not just the last.
"To batch, I stack the examples as extra rows of the input, keeping as ."
Convention here puts examples as columns, so is ; that keeps producing with bias broadcast per column. Row-stacking breaks the matmul dimensions.
"softmax logits are , so the probabilities are just divided by 3."
Softmax exponentiates first: then normalizes, giving roughly — not a plain linear normalization. The exponential is what makes larger logits dominate.
"ReLU has no vanishing-gradient problem, so it never kills a neuron."
ReLU can still go dead: if a unit's pre-activation is always negative it outputs 0 and passes zero gradient, a "dying ReLU." No-vanishing-gradient only holds on the positive side. See Vanishing and exploding gradients.
Why questions
Why do we insert a nonlinear activation between layers rather than a second linear step?
Because two linear steps compose into one linear step; only a nonlinearity can bend the function so a deep stack represents curved, non-collapsible mappings. This is the heart of the Universal approximation theorem.
Why is row of the weight matrix exactly neuron 's weights?
A matrix–vector product computes each output as a dot product of a row with the input, and row is precisely the weighted sum for neuron .
Why does softmax exponentiate the logits instead of using them directly?
Exponentials are always positive and monotonic, so bigger logits map to bigger weights, and dividing by their sum yields a valid probability distribution — impossible with possibly-negative raw logits.
Why do we run forward propagation as one big batched matrix multiply instead of a loop over examples?
One large matmul saturates GPU/parallel hardware, whereas tiny multiplies waste it on overhead; the math per example is identical, only the efficiency differs.
Why must forward propagation happen before backpropagation, not after?
Backprop needs the stored pre-activations and activations ( at each layer) plus the final prediction to compute gradients; forward propagation is what produces those intermediate values.
Why can a single neuron never model an XOR-style decision by itself?
A single neuron applies one weighted sum + threshold, giving one straight-line boundary; XOR needs a bent/piecewise boundary, which requires a hidden layer with a nonlinearity.
Edge cases
If every weight in a layer is initialized to the same value and every bias equal, what breaks?
All neurons compute the identical pre-activation and identical gradient, so they stay identical forever ("symmetry"). The layer effectively has one distinct unit — which is why we random-initialize.
What is the output of a ReLU layer when all its pre-activations are negative?
The layer outputs an all-zeros vector, gating off completely and passing no signal (and, in training, no gradient) to the next layer — the degenerate "dead" case.
What does softmax return when all logits are equal, say ?
A uniform distribution : equal logits give equal exponentials, so the model expresses maximal uncertainty across the classes.
For an identity output activation, is there any bound on the prediction ?
No — identity means can be any real number, positive or negative, which is exactly why it suits regression rather than probability outputs.
If the input is the zero vector , is the network output necessarily zero?
No. The first layer gives , so the biases (and later layers) can produce a nonzero prediction even from an all-zero input.
What happens to a huge positive logit inside softmax numerically, and how is it tamed?
can overflow to infinity; the standard fix subtracts from every logit before exponentiating, which leaves the probabilities unchanged but keeps the exponentials finite.
Recall One-line self-test
Name the three moves of a layer and the one thing that would collapse the whole network if removed. Answer ::: Weighted sum, add bias, bend through activation; removing the activation collapses all layers into a single linear map.