Foundations — Feedforward network — forward pass

Before you can read a single line of the forward pass note, you need to already own the meaning of a handful of symbols. The parent note uses them; here we build them, one at a time, each from the picture it stands for. If you can pass the checklist at the bottom, the parent note will read like plain English.

1. A number and a list of numbers (the vector $\mathbf{x}$)

Start with the simplest object: a single number, like the airspeed $250$. A sensor gives you one number.

But an airplane has many sensors at once — airspeed, throttle, angle-of-attack. Stack those readings into a column, one on top of the other. That stack is a vector. We write it in bold, $\mathbf{x}$, to say "this is a whole list, not one number".

Look at the figure. The three numbers become three stacked boxes. That vertical stack is the picture of $\mathbf{x}$. The little symbol ${\mathbb{R}}^{n_0}$ next to it is read "a list of $n_0$ real numbers" — $\mathbb{R}$ is the whole number line (any decimal, positive or negative), and the superscript $n_0$ counts how many slots the list has. So ${\mathbb{R}}^3$ means "three real numbers stacked".

Subscripts pick a slot. $x_1$ is the number in slot 1, $x_2$ in slot 2. The subscript is a street address inside the list.

2. A grid of numbers (the weight matrix $\mathbf{W}$)

Now, the whole point of a neuron is to mix the input numbers — take some of the airspeed, some of the throttle, add them up. "How much of each" is a set of dials called weights.

One neuron reading 3 inputs needs 3 dials. If we have 4 neurons, we need $4\times 3=12$ dials. Arrange those dials in a grid: one row per neuron, one column per input. That grid is a matrix, written bold and capital: $\mathbf{W}$.

The picture shows a $4\times 3$ grid. The red entry sits in row 2, column 3. We name it $w_{ij}$ where $i=2$ (the neuron it feeds into) and $j=3$ (the input it comes from). Read the subscripts right-to-left: "from input $j$ into neuron $i$." That ordering feels backwards but it is exactly what makes the matrix multiply work in the next section.

3. Multiplying a matrix by a vector (the core engine)

Here is the one operation the entire forward pass leans on. We have a grid $\mathbf{W}$ and a list $\mathbf{a}$, and we want the mixed result $\mathbf{Wa}$.

The rule, in words: to get the number in output slot $i$, walk along row $i$ of the matrix and along the input list at the same time, multiply the pairs, and add them up. That "multiply-pairs-and-add" move is called a dot product.

Follow the red row in the figure. Each dial in that row lines up with one number in the input column. Multiply each pair (red arrows), then sum the three products — that single sum is output slot $i$. Repeat for every row and you have filled the whole output list.

Shape rule (memorise the click). A $(n_{\ell },n_{\ell -1})$ matrix times a $(n_{\ell -1},1)$ column gives a $(n_{\ell },1)$ column. The inner numbers ($n_{\ell -1}$) must match — that is why the matrix has as many columns as the input has slots. When they match, they cancel and leave the outer numbers as the output shape.

4. Adding a bias (the vector $\mathbf{b}$)

A weighted sum can only produce $0$ when the inputs are $0$. But a neuron might need to be "already leaning on" or "already leaning off" regardless of input. The bias is that built-in lean: a single number added to each neuron's sum.

So the full mixing step is $\mathbf{z}=\mathbf{Wa}+\mathbf{b}$: mix, then shift. In the picture from section 3, the bias is simply one extra number dropped into each output slot after the dot product.

5. The bending curve (activation $\sigma$)

If all a network ever did was $\mathbf{Wa}+\mathbf{b}$, stacking layers would be pointless — one big grid could do the same job (the parent note proves this collapse). Straight-line mixing can only draw straight relationships. Real aerospace behaviour (stall, shock) is curved. So after every mix we bend the numbers with a nonlinear function $\sigma$ (Greek "sigma", lowercase).

The red ReLU curve is dead flat for negative inputs (the neuron "doesn't fire") then rises as a straight $45°$ line for positives. The black tanh curve flattens at $+1$ and $-1$ no matter how huge the input — a natural "confidence saturates" shape. Both are curves, not lines, and that is the whole reason they exist.

6. Layers, and the superscript $(\ell )$

The parent note writes ${\mathbf{W}}^{(1)}$, ${\mathbf{a}}^{(2)}$, ${\mathbf{z}}^{(\ell )}$. The superscript in parentheses is a layer label, not a power. ${\mathbf{a}}^{(1)}$ means "the activations of layer 1", never "$\mathbf{a}$ squared".

How it all feeds the forward pass

Read it top to bottom: numbers become vectors, weights become a matrix, the two meet in a matrix–vector product, bias shifts the result, the activation bends it, and that output becomes the next layer's input — looping until the last layer hands back the prediction $\mathbf{y}$.

Equipment checklist

Cover the right side and answer aloud. If any one stumps you, reread its section before opening the parent note.