Feedforward network — forward pass: Worked examples

Before anything, plain-word reminders (no symbol used below is left undefined):

Definition The machine parts (plain words)

A weight matrix $W$ is a rectangular grid of numbers. Row $i$ , column $j$ holds $w_{ij}$ : how strongly input $j$ pushes on output $i$ . Boldface always means a whole vector or matrix; a plain italic letter like $b^{(2)}$ means a single number.
A pre-activation $z$ is the raw score of each neuron before any squashing: $z = Wa + b$ . Here $a$ is the incoming column of activations, $b$ the bias column (a constant added to each neuron).
An activation vector $a$ is the column that leaves a layer after squashing: $a = σ (z)$ , meaning "apply the curve $σ$ to every entry of $z$ ". The input layer's activation is just the input itself, $a^{(0)} = x$ ; this is the $a$ that feeds the first $z = Wa + b$ .
The output $y$ (or column $y$ ) is the activation of the last layer — the network's actual prediction. Every example below ends by naming this $y$ .

We will use four activation curves. Each answers a different question, so we say why we reach for it — and each is plotted so you can see its shape before we ever compute with it:

The figure below draws all four so their behaviour is visual first, algebra second — notice ReLU's hard elbow at $0$ , sigmoid's smooth climb from $0$ to $1$ , and tanh's S-shape from $- 1$ to $+ 1$ :

Figure — Feedforward network — forward pass

Read the plot before moving on: the yellow ReLU is flat (dead) for $z < 0$ then a $4 5^{\circ}$ ramp; the blue sigmoid hugs $0$ far left and $1$ far right; the pink tanh does the same but between $- 1$ and $+ 1$ . These three shapes explain every "why did this neuron die / saturate / go negative" moment on this page.

Here is $e$ appearing: $e \approx 2.71828$ is the number whose exponential curve $e^{z}$ grows at a rate equal to its own height. We reach for it in sigmoid/tanh/softmax because it is the natural currency of "smooth squashing" — it makes the derivatives clean later during backprop. For hand-work you only need: $e^{0} = 1$ , $e^{1} \approx 2.718$ , $e^{- 1} \approx 0.368$ .

The scenario matrix

Every cell below is a distinct thing a forward pass can be asked to do. The examples that follow cover all of them.

Cell	Scenario	Covered by
A	Positive-only inputs, ReLU hidden, linear output (plain regression)	Ex 1
B	Sign mix in $z$ — some neurons fire, some die (ReLU zeroing)	Ex 2
C	Zero input vector $x = 0$ (degenerate) — bias-only output	Ex 3
D	Huge input → saturation of a squashing curve (limiting behaviour)	Ex 4
E	Sigmoid output read as a probability + a decision threshold	Ex 5
F	tanh hidden layer producing signed activations	Ex 6
G	Softmax multi-class: raw scores → probabilities summing to 1	Ex 7
H	Real-world word problem (stall-warning autopilot) end-to-end	Ex 8
I	Exam twist: "collapse" — network with no nonlinearity is just one affine map	Ex 9

Forecast: Guess whether both hidden neurons survive ReLU, then guess $y$ before reading on.

Pre-activation, layer 1. $z^{(1)} = W^{(1)} a^{(0)} + b^{(1)}$ with $a^{(0)} = x$ , giving $[1 \cdot 2 + 0 \cdot 3 0 \cdot 2 + 2 \cdot 3] + [- 1 - 1] = [15]$ . Why this step? Each neuron is a dot product of its weight row with the incoming activation, then the bias is added — this is the only way scores enter the machine.
Activate. $a^{(1)} = σ (z^{(1)})$ with $σ = ReLU$ . Both scores are positive, so ReLU leaves them: $a^{(1)} = [15]$ . Why this step? ReLU only deletes negatives; positives pass unchanged, so no neuron dies here — look back at the yellow curve in the intro figure, both points sit on its rising ramp.
Output layer. $z^{(2)} = 1 \cdot 1 + 1 \cdot 5 + 0 = 6$ . Linear output ⇒ the last activation is $y = a^{(2)} = z^{(2)} = 6$ . Why this step? Linear output keeps the prediction unbounded — right for a regression number.

What the figure shows: the two blue input nodes feed both yellow hidden nodes (both alive, values $1$ and $5$ ), which feed the single pink output node holding $y = 6$ — a picture of "dot, add, squash" happening twice.

Verify: Recompute by full sum: $y = (2 - 1) + (6 - 1) = 1 + 5 = 6$ . ✓ Units: whatever the target is (e.g. N of thrust) — a single real number, as a regression head should give.

Forecast: One neuron will be negative before ReLU. Which one, and does its weight into the output still matter?

Pre-activation. $z_{1}^{(1)} = 1 \cdot 2 + 0 \cdot 7 = 2$ , $z_{2}^{(1)} = - 1 \cdot 2 + 0 \cdot 7 = - 2$ . So $z^{(1)} = [2 - 2]$ . Why this step? The second row has a negative weight on feature 1, so it produces a negative score — the classic setup for ReLU to zero it.
Activate. $a^{(1)} = ReLU (z^{(1)})$ : $ReLU (2) = 2$ , $ReLU (- 2) = 0$ , so $a^{(1)} = [20]$ . Neuron 2 is dead for this input. Why this step? ReLU sets every negative to exactly $0$ — a hard cliff, visible as the flat left part of the yellow curve.
Output. $y = 3 \cdot 2 + 3 \cdot 0 + 1 = 7$ . The dead neuron's weight $3$ multiplies $0$ , contributing nothing. Why this step? A silent neuron carries no information forward no matter how big its outgoing weight — this is why ReLU creates sparse activity.

What the figure shows: the same 2-2-1 graph, but the second hidden node is greyed out at value $0$ — its outgoing edge is drawn faint to signal "no signal passes", so $y$ is built from neuron 1 and the bias alone.

Verify: $y = 6 + 0 + 1 = 7$ . ✓ Sanity: if we removed neuron 2 entirely the answer would be identical — confirming it is inactive here.

Forecast: With all inputs zero, what survives — and what does the output read?

Pre-activation. $W0 = 0$ for any weights, so $z = 0 + b = b$ ; the first two entries are $[0.5 - 0.5]$ . Why this step? Multiplying a grid by a zero column gives a zero column — the input contributes nothing, the machine falls back to bias alone. This is the meaning of bias: the output when the input says nothing.
Activate (sigmoid). $a = σ (z)$ : $σ (0.5) = \frac{1}{1 + e ^{- 0.5}}$ , $σ (- 0.5) = \frac{1}{1 + e ^{0.5}}$ . Numerically $e^{- 0.5} \approx 0.6065$ so $σ (0.5) \approx 0.6225$ ; $e^{0.5} \approx 1.6487$ so $σ (- 0.5) \approx 0.3775$ . Since this is the last layer, these are the outputs $y_{1}, y_{2}$ . Why this step? Sigmoid squashes each bias into $(0, 1)$ ; a positive bias lands above $0.5$ , a negative bias below $0.5$ — read it off the blue curve in the intro figure at $z = \pm 0.5$ .

Verify: Note $σ (0.5) + σ (- 0.5) = 0.6225 + 0.3775 = 1.000$ because sigmoid is symmetric: $σ (z) + σ (- z) = 1$ . ✓

Forecast: Guess $σ (20)$ and $σ (- 20)$ to two decimals before computing.

Compute $σ (20)$ . $e^{- 20} \approx 2 \times 1 0^{- 9}$ , so $y = σ (20) = \frac{1}{1 + 2 \times 1 0 ^{- 9}} \approx 0.999999998$ . Why this step? A large positive $z$ makes $e^{- z}$ vanish, pinning the output at the top rail $1$ .
Compute $σ (- 20)$ . $e^{20} \approx 4.85 \times 1 0^{8}$ , so $y = σ (- 20) = \frac{1}{1 + 4.85 \times 1 0 ^{8}} \approx 2 \times 1 0^{- 9}$ . Why this step? A large negative $z$ makes $e^{- z}$ explode, pinning the output at the bottom rail $0$ .

What the figure shows: the blue sigmoid with two marked dots at $z = \pm 20$ sitting flat against the dashed rails $y = 1$ and $y = 0$ — the curve is so flat there that moving $z$ barely moves $y$ , which is exactly the danger of saturation.

Limiting behaviour: as $z \to + \infty$ , $σ \to 1$ ; as $z \to - \infty$ , $σ \to 0$ . In the flat rails the curve barely changes, so once inputs are huge the network becomes insensitive — the reason we normalize inputs.

Verify: $σ (20) \approx 1$ and $σ (- 20) \approx 0$ to 6 decimals; also $σ (20) + σ (- 20) \approx 1$ . ✓

Forecast: Compute $z$ ; is it above or below $0$ (the $0.5$ line of sigmoid)?

Pre-activation. $z = 0.4 \cdot 1 + (- 0.1) \cdot 2 + 0.3 = 0.4 - 0.2 + 0.3 = 0.5$ . Why this step? One dot product plus bias gives the single decision score.
Sigmoid. $y = σ (0.5) = \frac{1}{1 + e ^{- 0.5}} \approx 0.6225$ . Why this step? Sigmoid turns the score into a probability so we can threshold it.
Threshold. $0.6225 > 0.5 \Rightarrow$ predict fault. Why this step? $σ (z) > 0.5$ exactly when $z > 0$ ; here $z = 0.5 > 0$ , consistent.

Verify: Because $z = 0.5 > 0$ , sigmoid must exceed $0.5$ ; indeed $0.6225 > 0.5$ . ✓ The output $y$ lies in $(0, 1)$ as a probability must.

Forecast: One neuron's score will be $0$ , the other nonzero. Predict the tanh of each.

Pre-activation. $z_{1} = 1 \cdot 1 + 1 \cdot (- 1) = 0$ ; $z_{2} = 1 \cdot 1 + (- 1) \cdot (- 1) = 2$ . Why this step? Neuron 1 sums a cancelling pair → exactly $0$ ; neuron 2 reinforces → $2$ .
tanh. $a^{(1)} = tanh (z^{(1)})$ : $tanh (0) = 0$ ; $tanh (2) = \frac{e ^{2} - e ^{- 2}}{e ^{2} + e ^{- 2}} \approx \frac{7.389 - 0.135}{7.389 + 0.135} \approx 0.964$ . Why this step? tanh passes zero straight through and gives a signed value near $+ 1$ for a positive score — its output range $(- 1, 1)$ lets hidden units be negative, unlike ReLU.

What the figure shows: the pink tanh curve with dots at $z = 0$ (lands on $0$ ) and $z = 2$ (lands near $0.964$ ), between the dashed $\pm 1$ rails — proof that tanh can output negative values, the feature ReLU lacks.

Verify: $tanh (0) = 0$ exactly; $tanh (2) \approx 0.9640$ . ✓ Both lie in $(- 1, 1)$ .

Forecast: Which class wins, and will the three probabilities add to exactly $1$ ?

Exponentiate each score. $e^{2} \approx 7.389$ , $e^{1} \approx 2.718$ , $e^{- 1} \approx 0.368$ . Why this step? $e^{z}$ is always positive and turns bigger scores into much bigger numbers — softmax's way of making a "soft argmax".
Sum. $S = 7.389 + 2.718 + 0.368 = 10.475$ . Why this step? Dividing by the total forces the outputs to sum to $1$ , so they read as probabilities.
Divide. $y_{1} = 7.389/10.475 \approx 0.7054$ , $y_{2} = 2.718/10.475 \approx 0.2595$ , $y_{3} = 0.368/10.475 \approx 0.0351$ . Why this step? Each ratio is that class's share of the total exponential mass.

What the figure shows: three chalk bars, tallest for Nominal ( $0.7054$ ), matching the largest raw score $2$ — the picture of softmax turning ranked scores into a probability bar chart that sums to $1$ .

Verify: $0.7054 + 0.2595 + 0.0351 = 1.0000$ . ✓ Argmax is class 1 (Nominal), matching the largest raw score $2$ . ✓

Forecast: High $α$ should push toward a stall warning. Will the output exceed $0.5$ ?

Hidden pre-activation. $z_{1} = 0.1 \cdot 12 + 0.5 \cdot (- 1) + 0.2 \cdot 0.5 - 1 = 1.2 - 0.5 + 0.1 - 1 = - 0.2$ . $z_{2} = - 0.2 \cdot 12 + 0.1 \cdot (- 1) + 0.3 \cdot 0.5 + 0 = - 2.4 - 0.1 + 0.15 = - 2.35$ . Why this step? Two dot products build the hidden scores from the sensors.
ReLU. $a^{(1)} = ReLU (z^{(1)})$ : $ReLU (- 0.2) = 0$ , $ReLU (- 2.35) = 0$ , so $a^{(1)} = [00]$ — both hidden neurons die. Why this step? Both scores are negative; ReLU zeroes both, a fully degenerate hidden layer.
Output. $z^{(2)} = 2 \cdot 0 + 1 \cdot 0 - 0.5 = - 0.5$ ; $y = σ (- 0.5) \approx 0.3775$ . Why this step? With the hidden layer silent, only the output bias speaks — and it says "no warning".
Decision. $0.3775 < 0.5 \Rightarrow$ no stall warning.

Verify: $σ (- 0.5) \approx 0.3775 < 0.5$ . ✓ Sanity: because $z^{(2)} = - 0.5 < 0$ , sigmoid must fall below $0.5$ — consistent. (Lesson: raw $α = 12$ was fine here; this same net does fire for larger $α$ , showing why the dead-neuron case is not "always no".)

Forecast: Two stacked linear maps — guess the single equivalent slope and intercept.

Layer 1. $a^{(1)} = 2 x + 1$ (identity leaves it linear). Why this step? With no squashing, the activation equals the pre-activation.
Layer 2. $y = 3 a^{(1)} - 4 = 3 (2 x + 1) - 4 = 6 x + 3 - 4 = 6 x - 1$ . Why this step? Substituting shows the whole network is the single affine map $y = 6 x - 1$ — proving stacked linear layers gain nothing. This is exactly why real networks need nonlinear $σ$ .
Evaluate. At $x = 5$ : $y = 6 \cdot 5 - 1 = 30 - 1 = 29$ . Why this step? Plugging into the collapsed line gives the same answer as walking both layers — the check below confirms it.

Verify: Direct two-step: $a^{(1)} = 2 \cdot 5 + 1 = 11$ , then $y = 3 \cdot 11 - 4 = 33 - 4 = 29$ . ✓ Matches the collapsed line $6 x - 1$ at $x = 5$ . ✓

Recall Quick self-check

ReLU of a negative pre-activation equals what? ::: Exactly $0$ — the neuron is silent. An activation vector $a$ is defined how, in terms of $z$ ? ::: $a = σ (z)$ — the curve applied to every entry of $z$ . With input $x = 0$ , the pre-activation equals what? ::: The bias $b$ (since $W0 = 0$ ). Softmax outputs always satisfy which one property? ::: They are positive and sum to exactly $1$ . Two stacked linear layers are equivalent to what? ::: A single affine map $W_{eff} x + b_{eff}$ .