5.6.7 · D5Machine Learning (Aerospace Applications)
Question bank — Feedforward network — forward pass
Before we start, three words must mean exactly the same thing to you as they do in the parent note, so let me pin them down in plain language:
True or false — justify
Nonlinearity claims are where most misconceptions live, so we start there.
Stacking two linear (identity-activation) layers can represent something a single linear layer cannot
False. Two affine maps composed give , which is again one affine map . No new power is gained.
The bias vector is what makes a network nonlinear
False. Bias only shifts the weighted sum up or down; shifting a straight line keeps it straight. The nonlinearity comes entirely from the activation function .
The forward pass changes the weights as it runs
False. The forward pass reads the weights and biases as fixed constants and only produces activations. Weights change during training via backpropagation, which is a separate, backward step.
means layer 0 is just a relabelling of the input with no computation
True. We call the raw input "activation of layer 0" purely so the recursion has a uniform starting point. No weights touch layer 0.
Because information flows only input→output, a feedforward net can still contain a loop as long as data never travels backward
False. "Feedforward" means the computation graph is a directed acyclic graph: no cycles at all. A loop would make "one pass" undefined — you'd never know when to stop.
Applying ReLU element-wise and applying it to the whole vector at once give different results
False. ReLU (and tanh, sigmoid) act independently on each entry, so is exactly the vector of applied to each component. There is no cross-talk between entries inside one activation.
Softmax is an element-wise activation just like ReLU
False. Softmax is the one common exception: each output depends on all the pre-activations (it divides each by the sum ). That coupling is what forces the outputs to sum to 1 and behave like probabilities.
Spot the error
Each item states something a student thinks is right. Name the mistake.
" has shape so the product with works."
Wrong shape. It must be : rows = neurons in the current layer, columns = neurons feeding in. Only then does give an -vector.
"For the 2-3-1 net, is a 2-vector because the input had 2 features."
Wrong dimension. The hidden layer has 3 neurons, so . Activation size is set by the current layer's neuron count, not by the input.
"Neuron 3 output 0 in Example 1, so we can delete that neuron; it contributes nothing."
Case error. It output 0 for that one input because its pre-activation was negative (). For a different input the same neuron may fire strongly. ReLU zeros are input-dependent, not permanent.
"I applied softmax to the hidden layer to get probabilities, then fed those forward."
Wrong layer. Softmax belongs at the output of a classifier so the final scores read as class probabilities. Using it internally squashes and normalises hidden features, destroying the raw signal the next layer expects.
" — matrix times vector, order doesn't matter."
Order matters. It is : the matrix is on the left. Matrix multiplication is not commutative, and generally isn't even a legal shape.
"The output prediction is , the pre-activation of the last layer."
Off by one step. The prediction is . You must still pass through the final activation — even if is the identity, that identity choice is a decision, not a skip.
"I normalised the inputs, so I don't need small weights on the altitude column anymore."
Confused cause. The tiny altitude weights in the raw example were a patch for un-normalised data. Once inputs are normalised the network can learn ordinary weights — the two are alternative fixes for the same scale problem, not both needed.
Why questions
Explain the reasoning; a bare fact is not enough.
Why do we compute activations layer by layer instead of jumping straight to the output?
Because layer 's input is layer 's output — the computation is a chain of dependencies. You physically cannot form until exists.
Why does the parent note call the input "" instead of just ""?
To make the recurrence formula uniform: one rule now covers every layer, including the first, with no special case.
Why is ReLU said to create "sparse" representations?
Because it sends every negative pre-activation to exactly 0, typically silencing many neurons at once. A vector with lots of zeros is sparse, which makes each active neuron's role easier to interpret.
Why must aerospace models be nonlinear — why not trust a big linear net?
Phenomena like stall, shock waves and compressibility are genuinely curved relationships. A linear (affine) map can only tilt and shift straight lines, so no matter how many linear layers you stack it can never bend to fit those curves.
Why does a linear output activation suit thrust regression but not fault classification?
Regression needs an unbounded real number (thrust can be any value), which linear/identity gives. Classification needs numbers that behave like probabilities across classes, which softmax — bounded in and summing to 1 — provides.
Why do we write the weighted sum as a dot product per row of ?
Each row holds one neuron's weights, so multiplying it against is exactly — the weighted aggregation of evidence for that single neuron. Row ⇒ neuron .
Why does normalising inputs matter for the forward pass even though the math "works" either way?
With a feature like altitude () dwarfing Mach (), one column of weighted contributions swamps the others unless weights are absurdly tiny. Normalising puts features on comparable scales so weights of ordinary size stay balanced.
Edge cases
The boundaries the topic quietly assumes you can handle.
What is the forward pass output of a network with (input straight to output, no hidden layer)?
A single affine-then-activation step: . Perfectly valid — it's just logistic/linear regression when is sigmoid/identity.
What does the network output if the input vector is all zeros?
Then (the weight term vanishes), so the biases alone drive layer 1, and the pass proceeds normally. Output is generally nonzero because of those biases.
If ALL pre-activations in a ReLU layer are negative, what is that layer's output?
The zero vector. ReLU clips every entry to 0, so nothing propagates forward from that layer for this input — a genuine dead-layer moment, though only for this particular input.
For tanh, what happens to a very large pre-activation like versus a moderate one?
tanh saturates toward for large positive (hence , nearly pinned), while moderate values map more gently. Very large or very small both flatten out near .
Two different inputs give the same hidden activations . Must they give the same output ?
Yes. The forward pass from onward is a deterministic function of alone. Once the hidden activations match, everything downstream matches.
If is the identity, is the output layer "doing nothing"?
No — it still applies its weights and bias . Only the final bending is skipped; the linear combination of the last hidden layer is very much real work.
Can a hidden neuron's bias alone make it fire when every weight into it is zero?
Yes. With zero weights, , so a positive bias through ReLU gives a constant positive output independent of the input — a constant-feature neuron. Unusual, but the math allows it.
Recall One-sentence self-test
State, without looking, the two-part rule executed at every layer of the forward pass. ::: Compute the pre-activation , then the activation .