Visual walkthrough — Feedforward network — forward pass

---

Step 1 — A neuron is just a dot, holding a number

WHAT. Draw a single circle. Inside it lives one number. That's a neuron. The number it holds is called its activation — think of it as how strongly this neuron is firing, like a dimmer switch between "off" and "very on."

WHY. Before any math, we must agree on the smallest object. Everything else — layers, matrices, the whole network — is copies of this one circle wired together. If you understand the circle, you understand the network.

PICTURE. Look at the figure. One cyan circle, one amber number inside. The arrows into it are the wires that will feed it; the arrow out carries its activation onward.

---

Step 2 — A wire multiplies; a weight is the volume knob on that wire

WHAT. Connect two circles with a wire. The left circle holds activation $a_j^{(\ell -1)}$. The wire carries that number to the right circle, but scales it first by a number called the weight $w_{ij}^{(\ell )}$. What arrives on the other side is the product $w_{ij}^{(\ell )}{a}_j^{(\ell -1)}$.

WHY. A raw sensor reading (say Mach 0.8) means different things for different decisions. The weight is how much this decision cares about that reading. A big positive weight = "listen hard and agree." A negative weight = "listen, but do the opposite." Zero = "ignore this wire." Multiplication is the natural tool because doubling the input should double its influence — that is exactly what multiplying by a fixed number does.

PICTURE. The figure shows the input number, the weight sitting on the wire like a volume knob, and the scaled number coming out. Watch the amber label change size.

---

Step 3 — A neuron adds up all its wires (the weighted sum)

WHAT. A neuron usually has many wires coming in. It simply adds up every scaled number: $\underset{\text{add over all input neurons }j}{\underbrace{\sum _{j=1}^{n_{\ell -1}}}}\underset{\text{knob on wire }j}{\underbrace{w_{ij}^{(\ell )}}}\underset{\text{what neuron }j\text{ sent}}{\underbrace{a_j^{(\ell -1)}}}$

WHY. Each wire is one piece of evidence. Summing them is how the neuron pools evidence into a single score: strong-agree wires push the sum up, disagree wires (negative weights) pull it down. Addition is the right tool because evidence should accumulate — two half-reasons should combine into one full reason.

PICTURE. Three input circles, three knobs, three arrows all funnelling into one big plus-sign, giving one running total. The $\sum$ (capital Greek "S," for Sum) is just shorthand for "add these all up," with $j$ counting from the first input neuron to the last one, $n_{\ell -1}$ (the number of neurons in the previous layer).

---

Step 4 — Add a bias: the neuron's built-in opinion

WHAT. After summing the wires, the neuron adds one more number that does not depend on any input — the bias $b_i^{(\ell )}$. The full total is the pre-activation: $z_i^{(\ell )}=\underset{\text{evidence from wires}}{\underbrace{\sum _j{w}_{ij}^{(\ell )}{a}_j^{(\ell -1)}}}+\underset{\text{fixed offset}}{\underbrace{b_i^{(\ell )}}}$

WHY. Sometimes a neuron should fire even when all inputs are quiet, or refuse to fire unless the evidence is overwhelming. The bias shifts the whole total up or down, setting the neuron's "default mood." Without it, every neuron would be forced to output the same thing when the input is all zeros — too rigid.

PICTURE. The plus-sign total from Step 3, then a small amber $+b$ box nudging the bar up (positive bias) or down (negative bias) before we read off $z$.

---

Step 5 — Stack the wires into one matrix multiply

WHAT. Every neuron in layer $\ell$ does its own weighted sum. Instead of writing one equation per neuron, we stack all the neurons' activations into a column of numbers (a vector) and all the weights into a grid (a matrix). Then the whole layer is one line: $\underset{\text{column, one entry per neuron}}{\underbrace{{\mathbf{z}}^{(\ell )}}}=\underset{\text{grid of all weights}}{\underbrace{{\mathbf{W}}^{(\ell )}}}\underset{\text{column from previous layer}}{\underbrace{{\mathbf{a}}^{(\ell -1)}}}+\underset{\text{column of biases}}{\underbrace{{\mathbf{b}}^{(\ell )}}}$

WHY. Bold symbols just bundle many numbers so we write one clean line instead of hundreds. A matrix–vector product is defined to be exactly "each output = dot product of one matrix row with the input vector" — which is precisely the weighted sum from Steps 3–4. So the compact line hides nothing; it is the same arithmetic. Computers also multiply matrices blazingly fast, so this form is what actually runs on the aircraft.

PICTURE. Row $i$ of the grid ${\mathbf{W}}^{(\ell )}$ lights up, slides across the input column, multiplies-and-adds, and drops one number into slot $i$ of ${\mathbf{z}}^{(\ell )}$. The grid has $n_{\ell }$ rows (one per neuron in this layer) and $n_{\ell -1}$ columns (one per neuron feeding it).

---

Step 6 — Bend the line: the activation function

WHAT. Feed the raw tally ${\mathbf{z}}^{(\ell )}$ through a bending function ${\sigma }^{(\ell )}$, applied to each entry on its own: ${\mathbf{a}}^{(\ell )}={\sigma }^{(\ell )}\left({\mathbf{z}}^{(\ell )}\right)$ Common choices: ReLU $\sigma (z)=\max (0,z)$ (flatten anything negative to 0), and tanh (squash any number into the range $-1$ to $+1$).

WHY THIS TOOL and not plain addition? Because if every layer were only sums-and-scales (straight-line maths), then stacking layers would collapse into one straight-line map — no matter how many you stack. Proof-by-picture below: two straight lines composed are still one straight line. A curved function is the only way to build shapes that straight lines can't — and stall, shock waves, and drag curves are all bendy. The bend is where the "learning power" comes from.

PICTURE. Left panel: the ReLU elbow — flat on the left, then a $45^{\circ }$ ramp; a negative $z$ lands at $0$, a positive $z$ passes through unchanged. Right panel: the tanh S-curve flattening toward $\pm 1$ at the edges.

---

Step 7 — Why the bend is not optional (the degenerate case)

WHAT. Suppose we removed every activation — set each $\sigma$ to "do nothing." Then layer after layer is just multiply-and-add. Chain two of them: ${\mathbf{a}}^{(2)}={\mathbf{W}}^{(2)}({\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)})+{\mathbf{b}}^{(2)}=\underset{\text{one new grid }{\mathbf{W}}_{\text{eff}}}{\underbrace{{\mathbf{W}}^{(2)}{\mathbf{W}}^{(1)}}}\mathbf{x}+\underset{\text{one new offset }{\mathbf{b}}_{\text{eff}}}{\underbrace{{\mathbf{W}}^{(2)}{\mathbf{b}}^{(1)}+{\mathbf{b}}^{(2)}}}$

WHY show this. It proves the earlier claim visually and algebraically: a 100-layer linear network equals a 1-layer network ${\mathbf{W}}_{\text{eff}}\mathbf{x}+{\mathbf{b}}_{\text{eff}}$. All that depth bought nothing. This is the degenerate case the reader must never fall into.

PICTURE. Top: two straight-line transforms drawn as two rulers; below, they merge into a single ruler — same output, one step. Amber caption: "depth wasted."

---

Step 8 — Wire the layers together: the full forward pass

WHAT. Start by renaming the input as layer 0's activation, so every layer looks identical: ${\mathbf{a}}^{(0)}=\mathbf{x}$ Then repeat Steps 5–6 for $\ell =1,2,\dots ,L$, feeding each layer's output into the next: ${\mathbf{a}}^{(\ell )}={\sigma }^{(\ell )}({\mathbf{W}}^{(\ell )}{\mathbf{a}}^{(\ell -1)}+{\mathbf{b}}^{(\ell )}),\mathbf{y}={\mathbf{a}}^{(L)}$

WHY rename the input? So the same recipe runs at every layer — no special case for the first one. Uniformity makes the code a clean loop and the maths a clean recursion. We stop at layer $L$ (the last one); its activation is the prediction $\mathbf{y}$.

PICTURE. The full pipeline left-to-right: input column → grid → bend → column → grid → bend → output. Data only ever flows rightward — that one-way flow is why it's called "forward."

---

Step 9 — Watch the parent's numbers flow through (the 2-3-1 case)

WHAT. Run Example 1 from the parent note through our picture pipeline. Input $\mathbf{x}=\left[\begin{array}{c}0.8\\ 10000\end{array}\right]$, one hidden layer (3 neurons, ReLU), one output (linear).

• Layer 1 tally: ${\mathbf{z}}^{(1)}=\left[\begin{array}{c}1.5\\ 0.06\\ -0.84\end{array}\right]$ → after ReLU: ${\mathbf{a}}^{(1)}=\left[\begin{array}{c}1.5\\ 0.06\\ 0\end{array}\right]$ (neuron 3 flattened to 0 — a dead neuron for this input).
• Output tally: $z^{(2)}=1.0(1.5)-0.5(0.06)+2.0(0)+0.3=1.77$, linear → $\mathbf{y}=1.77$.

WHY revisit it here. Now every number has a picture: the ReLU elbow clipping neuron 3, the output dot-product summing what survived. The abstract recursion becomes a concrete waterfall of numbers.

PICTURE. Each neuron drawn as a circle sized by its activation; neuron 3 greyed-out (dead); the final amber $1.77$ glowing at the output.

---

The one-picture summary

Everything above compressed into a single blueprint: the input column enters left, at each layer a grid (matrix) does the weighted sums, a plus adds the bias, a bend ($\sigma$) curves the result, and the arrow marches strictly rightward until the amber output pops out. That's the entire forward pass.

Recall Feynman retelling — say it back in plain words

A neuron is a dot holding one number. Wires bring numbers from the layer before, each wire scaling its number by a weight (a volume knob). The neuron adds all those scaled numbers, then adds its own bias — that's the raw tally $z$. Then it bends the tally through a curved function (ReLU flattens negatives; tanh squashes into $-\mathrm{1..1}$) to get its activation. Stacking a whole layer of these is one matrix-multiply-plus-bias-then-bend. We rename the input as "layer 0," then repeat the recipe layer by layer, always feeding forward, never backward — that one-way march is why it's called the forward pass. The bend is mandatory: without it, a hundred layers collapse into one straight line and all the depth is wasted. We stop at the last layer; its activation is the prediction.

Recall

Why is the network called "feedforward"? ::: Because activations flow in one direction only, input → output, never looping back. What does the weight $w_{ij}^{(\ell )}$ do to a wire? ::: It multiplies (scales) the number travelling from neuron $j$ to neuron $i$ — a volume knob. What is the pre-activation $z_i^{(\ell )}$? ::: The weighted sum of inputs plus the bias, before the activation function is applied. Why must the activation function be nonlinear? ::: Otherwise stacked layers collapse into a single affine map ${\mathbf{W}}_{\text{eff}}\mathbf{x}+{\mathbf{b}}_{\text{eff}}$, so depth buys nothing. What is ReLU of $-0.84$? ::: $0$ — ReLU flattens every negative to zero. What is the output of Example 1's network? ::: $1.77$.