Forward propagation computation: Visual walkthrough

We assume you know: what a number is, what "multiply then add" means, and that a picture is worth a page of algebra. Everything else — dot products, matrices, activations — we earn on the spot.

Step 1 — What is a single input, really?

WHAT. A network's input is just a short list of numbers. If we are recognising, say, a tiny handwritten digit from two measurements, the input is $x = [x_{1} x_{2}] .$ Here $x_{1}$ and $x_{2}$ are just two ordinary numbers stacked in a column. The bold $x$ is shorthand for "the whole stack at once."

WHY. We stack them because in the next steps we will feed all of them into every neuron simultaneously. Writing them as one column lets us talk about the whole input in a single symbol.

PICTURE. Two amber dials, each holding a number, feeding into the machine.

Figure — Forward propagation computation

Step 2 — One neuron: mix the inputs with a "recipe"

WHAT. A neuron has one weight per input, $w_{1}, w_{2}$ , and one extra number $b$ (the bias). It computes a single number: $z = trust in x_{1} w_{1} input 1 x_{1} + trust in x_{2} w_{2} input 2 x_{2} + fixed nudge b .$ Each weight says how much this neuron cares about that input. The bias adds a constant, independent of the inputs.

WHY multiply-then-add? Because we want a single "score" that goes up when an input we trust is large, and down when a distrusted (negative-weight) input is large. Multiplying weight by input, then summing, is the simplest rule that does exactly this. The bias lets the neuron have an opinion even when every input is zero — without it, $x = 0$ would force $z = 0$ always, which is a needless handicap.

PICTURE. Two inputs flowing along wires; each wire scaled by its weight; a bias splash dropped in; everything poured into one summing bowl labelled $z$ .

Step 3 — Why one neuron is not enough: bend it

WHAT. After computing $z$ , the neuron passes it through a bending function $ϕ$ (see Activation functions): $a = ϕ (z) .$ A popular choice is ReLU: $ϕ (z) = max (0, z)$ — "keep positive numbers, flatten negatives to zero."

WHY bend at all? This is the heart of everything. If we never bent, then stacking neuron after neuron would just be adding and multiplying — and a pile of "multiply-and-add" is still one big multiply-and-add. No matter how deep, a straight-line-only machine can only draw straight lines. The bend is what lets the network draw curves (this is the promise of the Universal approximation theorem).

PICTURE. The straight line $z$ enters ReLU; the left half (negatives) gets folded flat onto the floor; the right half passes through unchanged. That single fold is the source of all the machine's power.

Step 4 — Many neurons = a row of recipes = a matrix

WHAT. A layer is just several neurons side by side, each with its own weights. Neuron $1$ has weights $(w_{11}, w_{12})$ , neuron $2$ has $(w_{21}, w_{22})$ . Stack each neuron's weights as a row: $W = [w_{11} w_{21} w_{12} w_{22}], z = W x + b .$ Reading term by term for output slot $j$ : $z_{j} = row j of W dotted with x w_{j 1} x_{1} + w_{j 2} x_{2} + neuron j ’s bias b_{j} .$

WHY a matrix? Because "take a row, multiply each entry by the matching input, add up" — a dot product — is precisely the weighted sum from Step 2. Putting neuron $j$ 's weights in row $j$ makes the machinery of matrix-times-vector spit out all the $z_{j}$ at once. That's why $W$ is $m \times n$ (m output rows, n input columns): the multiply demands it.

PICTURE. Each row of the grid $W$ reaches across the input column, multiplies pairwise, and drops its sum into the matching output slot.

Step 5 — Chain layers: the assembly line

WHAT. Feed one layer's output into the next. With $a^{[0]} = x$ , for each layer $ℓ = 1, 2, \dots, L$ : $z^{[ℓ]} = W^{[ℓ]} a^{[ℓ - 1]} + b^{[ℓ]}, a^{[ℓ]} = ϕ^{[ℓ]} (z^{[ℓ]}) .$ The superscript $[ℓ]$ is just a station number on the assembly line, not a power.

WHY chain? One layer can only bend once — one crease. Chaining lets each layer bend the already-bent output again, and creases stacked on creases carve out richly curved decision regions. The output of the last station, $a^{[L]} = \overset{y}{^}$ , is the prediction.

PICTURE. Numbers flowing left-to-right through three stations; at each station the "weigh–bias–bend" box acts, and the shape of the data changes.

Step 6 — The output station: turning scores into probabilities

WHAT. For classification the last $ϕ$ is usually softmax (see Softmax and cross-entropy loss). Given final scores $z^{[L]}$ : $softmax (z)_{k} = \frac{e ^{z_{k}}}{j \sum e ^{z_{j}}} .$ Term by term: $e^{z_{k}}$ makes score $k$ positive; the denominator is the total of all such positives, so dividing forces the outputs to add up to $1$ .

WHY exponentials? We need every output positive (a probability can't be negative) and monotonic (a bigger score → bigger probability). $e^{(\cdot)}$ does both, and dividing by the sum normalises the pile into a valid probability distribution.

PICTURE. Three raw scores → exponentiated into positive bars → rescaled so the bars fill exactly one full "probability bar."

Step 7 — Edge cases: what breaks, and why it doesn't

WHAT & WHY (four degenerate inputs).

Zero input $x = 0$ : then $z = b$ . The neuron still fires because of the bias — this is the whole point of $b$ .
All-negative $z$ into ReLU: every output becomes $0$ — a "dead" layer. This is real (the dying ReLU problem) and connects to Vanishing and exploding gradients.
Huge logits in softmax: $e^{1000}$ overflows. The fix is subtracting the max, $e^{z_{k} - m a x z}$ , which gives the same probabilities (the constant cancels top and bottom).
All-equal logits $z = [c, c, c]$ : softmax returns $[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}]$ — maximal uncertainty, exactly what "no preference" should look like.

PICTURE. Four small panels, one per edge case, each showing the input and what the machine does with it.

Step 8 — Watch it run: the worked example, as a picture

Using $x = [12]$ , $W^{[1]} = [0.5 1 - 0.5 1]$ , $b^{[1]} = [0 - 1]$ , ReLU hidden, then $W^{[2]} = [2 - 1]$ , $b^{[2]} = 0.5$ , identity output.

Weigh + bias: $z^{[1]} = [- 0.5 2]$ .
Bend (ReLU): $a^{[1]} = [02]$ — neuron 1's negative score is flattened to $0$ : it is gated off.
Output: $z^{[2]} = 2 (0) + (- 1) (2) + 0.5 = - 1.5 = \overset{y}{^}$ .

PICTURE. The exact numbers flowing through, with the flattened neuron drawn greyed out.

The one-picture summary

Every idea on this page, compressed: raw numbers enter, each station does weigh → bias → bend, the shape gets progressively re-carved, and the final station outputs a prediction (a probability distribution if softmax).

Recall Feynman retelling of the whole walkthrough

Picture a water bucket-brigade. Your input is a couple of buckets of water. Each neuron is a person who scoops from the buckets behind them — scooping more from neighbours they trust (the weights) and always adding their own fixed splash (the bias). That's the weighted sum $z$ . Then they pour it through a funny funnel that flattens anything below a line (the ReLU bend) — this bending is the only reason the brigade can do something cleverer than a single scoop-and-pour. A whole row of people forms one layer (that's the matrix $W$ : one row of trust-values per person). The water passes station by station, re-shaped at each, until the last row of people — using the softmax funnel — hands you buckets that are guaranteed positive and add up to exactly one full tank: your probabilities. Forward propagation is just running the water forward through the brigade one time.