Feedforward network — forward pass

Why "forward"? Because we move from input to output (the prediction direction). Later, backpropagation will move backward to compute gradients. The forward pass is how your trained model actually runs inference in production—it's the airplane's autopilot computing control commands from sensor readings.

The Architecture — What Are We Computing?

Each connection from neuron $j$ in layer $\ell-1$ to neuron $i$ in layer $\ell$ has a weight $w_{ij}^{(\ell)}$. Each neuron $i$ in layer $\ell$ has a bias $b_i^{(\ell)}$.

Key insight: A neuron computes a weighted sum of its inputs, adds bias, then applies a nonlinear activation function. Without nonlinearity, stacking layers would collapse to a single linear transformation (useless for complex aerospace dynamics).

The Forward Pass Algorithm — Step by Step Derivation

Starting Point: What Do We Have?

Inputs:

• Input vector $\mathbf{x} \in \mathbb{R}^{n_0}$ (sensor readings)
• Weight matrices $\mathbf{W}^{(1)}, \mathbf{W}^{(2)}, \ldots, \mathbf{W}^{(L)}$ where $\mathbf{W}^{(\ell)} \in \mathbb{R}^{n_\ell \times n_{\ell-1}}$
• Bias vectors $\mathbf{b}^{(1)}, \mathbf{b}^{(2)}, \ldots, \mathbf{b}^{(L)}$ where $\mathbf{b}^{(\ell)} \in \mathbb{R}^{n_\ell}$
• Activation functions $\sigma^{(1)}, \sigma^{(2)}, \ldots, \sigma^{(L)}$ (often ReLU, sigmoid, tanh, or linear)

Goal: Compute output $\mathbf{y} = \mathbf{a}^{(L)}$.

Layer-by-Layer Propagation

Step 1: Initialize $\mathbf{a}^{(0)} = \mathbf{x}$

Why? The "activation" of layer0 is just the input data itself. We denote it $\mathbf{a}^{(0)}$ to make the recursion uniform.

Step 2: For each layer $\ell = 1, 2, \ldots, L$:

(a) Compute pre-activation (weighted sum + bias)* $\mathbf{z}^{(\ell)} = \mathbf{W}^{(\ell)} \mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)}$

Why this form?

• Each neuron $i$ in layer $\ell$ computes $z_i^{(\ell)} = \sum_{j=1}^{n_{\ell-1}} w_{ij}^{(\ell)} a_j^{(\ell-1)} + b_i^{(\ell)}$
• In matrix form: $\mathbf{W}^{(\ell)}$ has shape $(n_\ell, n_{\ell-1})$, $\mathbf{a}^{(\ell-1)}$ is $(n_{\ell-1}, 1)$, so the product is $(n_\ell, 1)$
• This is a linear transformation of the previous layer's output

(b) Apply activation function element-wise $\mathbf{a}^{(\ell)} = \sigma^{(\ell)}(\mathbf{z}^{(\ell)})$

Why activation? Without $\sigma$, the entire network would be: $\mathbf{a}^{(L)} = \mathbf{W}^{(L)} (\mathbf{W}^{(L-1)} (\cdots (\mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}) + \cdots) + \mathbf{b}^{(L-1)}) + \mathbf{b}^{(L)}$ which colapses to $\mathbf{a}^{(L)} = \mathbf{W}_{\text{eff}} \mathbf{x} + \mathbf{b}_{\text{eff}}$, a single affine map. Aerospace systems are nonlinear (stall, shock waves, compressibility)—we need nonlinearity to approximate them.

Step 3: Output $\mathbf{y} = \mathbf{a}^{(L)}$

Why stop here? The final layer's activation is the network's prediction. For regression (predicting continuous control surfaces), often $\sigma^{(L)}$ is linear (identity). For classification (fault detection: nominal/critical), $\sigma^{(L)}$ might be softmax.

Worked Examples — Concrete Calculations

Setup: Predict thrust adjustment from 2 sensors (Mach number, altitude).

• Input: $\mathbf{x} = \begin{bmatrix} 0.8 \\ 10000 \end{bmatrix}$ (Mach 0.8, 10k ft)
• Hidden layer: 3 neurons, ReLU activation
• Output layer: 1 neuron, linear activation

Weights and biases (pretrained): $\mathbf{W}^{(1)} = \begin{bmatrix} 0.5 & 0.0001 \\ -0.3 & 0.00005 \\ 0.2 & -0.0001 \end{bmatrix}, \quad \mathbf{b}^{(1)} = \begin{bmatrix} 0.1 \\ -0.2 \\ 0.0\end{bmatrix}$ $\mathbf{W}^{(2)} = \begin{bmatrix} 1.0 & -0.5 & 2.0 \end{bmatrix}, \quad \mathbf{b}^{(2)} = \begin{bmatrix} 0.3 \end{bmatrix}$

Why these tiny second-column weights? The altitude feature (10000) is huge compared to Mach (0.8). Small weights on the altitude column keep contributions balanced—this is why in practice we normalize inputs before training.

Forward Pass Computation:

Layer 1 (Hidden):

Step 2a: Pre-activation $\mathbf{z}^{(1)} = \mathbf{W}^{(1)} \mathbf{a}^{(0)} + \mathbf{b}^{(1)}$ $= \begin{bmatrix} 0.5 & 0.0001 \\ -0.3 & 0.00005 \\ 0.2 & -0.0001 \end{bmatrix} \begin{bmatrix} 0.8 \\ 10000 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.2 \\ 0.0 \end{bmatrix}$ $= \begin{bmatrix} 0.5 \cdot 0.8 + 0.0001 \cdot 10000 \\ -0.3 \cdot 0.8 + 0.00005 \cdot 10000 \\ 0.2 \cdot 0.8 - 0.0001 \cdot 10000 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.2 \\ 0.0 \end{bmatrix}$ $= \begin{bmatrix} 0.4 + 1.0 \\ -0.24 + 0.5 \\ 0.16 - 1.0 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.2 \\ 0.0 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 0.06 \\ -0.84 \end{bmatrix}$

Why this step? Each neuron aggregates evidence from the input features. Neuron 1 gets strong signal (1.5), neuron 2 weak (0.06), neuron 3 negative (-0.84).

Step 2b: Activation (ReLU: $\max(0, z)$) $\mathbf{a}^{(1)} = \text{ReLU}(\mathbf{z}^{(1)}) = \begin{bmatrix} \max(0, 1.5) \\ \max(0, 0.06) \\ \max(0, -0.84) \end{bmatrix} = \begin{bmatrix} 1.5 \\ 0.06 \\ 0.0 \end{bmatrix}$

Why ReLU here? Negative pre-activation means "this feature combination doesn't fire this neuron." ReLU zeros it out, creating sparse representations (many neurons off). Neuron 3 is "dead" for this input—it detected a pattern that's absent.

Layer 2 (Output):

Step 2a: Pre-activation $\mathbf{z}^{(2)} = \mathbf{W}^{(2)} \mathbf{a}^{(1)} + \mathbf{b}^{(2)}$ $= \begin{bmatrix} 1.0 & -0.5 & 2.0 \end{bmatrix} \begin{bmatrix} 1.5 \\ 0.06 \\ 0.0 \end{bmatrix} + \begin{bmatrix} 0.3 \end{bmatrix}$ $= [1.0 \cdot 1.5 - 0.5 \cdot 0.06 + 2.0 \cdot 0] + 0.3 = 1.5 - 0.03 + 0.3 = 1.77$

Why this step? The output neuron combines the hidden features. Positive weights (1.0, 2.0) amplify neurons 1 and 3; negative weight (-0.5) suppresses neuron 2. Neuron 3 is off, so its connection (2.0) contributes nothing.

Step 2b: Activation (Linear: $\sigma(z) = z$) $\mathbf{a}^{(2)} = \mathbf{z}^{(2)} = 1.77$

Why linear output? For regression, we want unbounded predictions. The network predicts thrust adjustment of 1.77 N (or whatever units were used in training).

Setup: Fault detection—classify flight state into 3 categories: 0=Nominal, 1=Degraded, 2=Critical.

• Input: $\mathbf{x} = \begin{bmatrix} 250 \\ 0.95 \\ 35 \end{bmatrix}$ (airspeed m/s, throttle fraction, angle-of-attack deg)
• Hidden: 4 neurons, tanh activation
• Output: 3 neurons, softmax activation

Weights: $\mathbf{W}^{(1)} = \begin{bmatrix} 0.01 & 0.5 & 0.02 \\ -0.01 & -0.3 & 0.03 \\ 0.005 & 0.1 & -0.05 \\ 0.02 & 0.2 & 0.01 \end{bmatrix}, \quad \mathbf{b}^{(1)} = \begin{bmatrix} -2 \\ 1 \\ 0 \\ -1 \end{bmatrix}$ $\mathbf{W}^{(2)} = \begin{bmatrix} 1.5 & -0.8 & 0.3 & 1.0 \\ -1.0 & 0.5 & -0.2 & -0.9 \\ 0.2 & 1.2 & 0.9 & -0.5 \end{bmatrix}, \quad \mathbf{b}^{(2)} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$

Forward Pass:

Layer 1 — Pre-activation (computing $\mathbf{z}^{(1)} = \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}$ row by row):

• $z_1 = 0.01(250) + 0.5(0.95) + 0.02(35) - 2 = 2.5 + 0.475 + 0.7 - 2 = 1.675$
• $z_2 = -0.01(250) - 0.3(0.95) + 0.03(35) + 1 = -2.5 - 0.285 + 1.05 + 1 = -0.735$
• $z_3 = 0.005(250) + 0.1(0.95) - 0.05(35) + 0 = 1.25 + 0.095 - 1.75 = -0.405$
• $z_4 = 0.02(250) + 0.2(0.95) + 0.01(35) - 1 = 5.0 + 0.19 + 0.35 - 1 = 4.54$

$\mathbf{z}^{(1)} = \begin{bmatrix} 1.675 \\ -0.735 \\ -0.405 \\ 4.54 \end{bmatrix}$

Why check each row? The matrix–vector product is just a dot product per row. Doing them explicitly catches sign errors—a common source of bugs.

Step 2b: Activation (tanh) $\mathbf{a}^{(1)} = \tanh(\mathbf{z}^{(1)}) \approx \begin{bmatrix} 0.933 \\ -0.626 \\ -0.384 \\ 0.9997 \end{bmatrix}$

Why tanh? Output range $(-1, 1)$ centers activations around zero, helping with gradient flow. For aerospace, negative activations can encode "opposite conditions" (e.g., pitch-up vs pitch-down). Notice neuron 4 saturates near $+1$.

Layer 2 — Pre-activation ($\mathbf{z}^{(2)} = \mathbf{W}^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)}$):

• $z_1 = 1.5(0.933) - 0.8(-0.626) + 0.3(-0.384) + 1.0(0.9997) \approx 1.400 + 0.501 - 0.115 + 1.000 = 2.786$
• $z_2 = -1.0(0.933) + 0.5(-0.626) - 0.2(-0.384) - 0.9(0.9997) \approx -0.933 - 0.313 + 0.077 - 0.900 = -2.069$
• $z_3 = 0.2(0.933) + 1.2(-0.626) + 0.9(-0.384) - 0.5(0.9997) \approx 0.187 - 0.751 - 0.346 - 0.500 = -1.410$

$\mathbf{z}^{(2)} \approx \begin{bmatrix} 2.786 \\ -2.069 \\ -1.410 \end{bmatrix}$

Softmax activation: $\mathbf{a}^{(2)}_i = \frac{e^{z^{(2)}_i}}{\sum_{j=1}^{3} e^{z^{(2)}_j}}$

$e^{\mathbf{z}^{(2)}} \approx \begin{bmatrix} e^{2.786} \\ e^{-2.069} \\ e^{-1.410} \end{bmatrix} = \begin{bmatrix} 16.22 \\ 0.126 \\ 0.244 \end{bmatrix}$

Sum $\approx 16.59$, so: $\mathbf{a}^{(2)} \approx \begin{bmatrix} 0.978 \\ 0.008 \\ 0.015 \end{bmatrix}$

Interpretation: The network predicts 97.8% probability of Nominal flight state (class 0), 1.5% Critical, 0.8% Degraded. The argmax is class 0.

Why softmax? It converts arbitrary scores (logits) into a valid probability distribution (sums to 1, all positive). Critical for classification—the pilot needs confidence levels, not raw scores.

Computational Complexity — Why It Matters in Real-Time Systems

Flops per layer: For layer $\ell$:

• Matrix multiply $\mathbf{W}^{(\ell)} \mathbf{a}^{(\ell-1)}$: $n_\ell \times n_{\ell-1}$ multiplications + $n_\ell \times (n_{\ell-1}-1)$ additions $\approx 2 n_\ell n_{\ell-1}$ flops
• Bias add: $n_\ell$ additions
• Activation: $n_\ell$ function evaluations (ReLU: 1 flop each; tanh: ~10 flops)

Total forward pass: $\sum_{\ell=1}^{L} (2 n_\ell n_{\ell-1} + n_\ell \cdot c_\sigma)$ flops, where $c_\sigma$ is activation cost.

Example: For a 10-100-100-5 network (10 inputs, TWO hidden layers of 100, 5 outputs):

• Layer 1: $2 \cdot 100 \cdot 10 + 100 = 2100$ flops
• Layer 2: $2 \cdot 100 \cdot 100 + 100 = 20100$ flops
• Layer 3: $2 \cdot 5 \cdot 100 + 5 = 1005$ flops
• Total: ~23k flops

Why this matters in aerospace: Flight control computers run at high frequency (100-1000 Hz). A 23k-flop network on a 1 GFlop processor takes 23 microseconds—well within a 1 ms control loop. But a 1000-1000-1000-5 network (2M flops) could miss deadlines. Model size is a design constraint for embedded systems.

Common Mistakes — Steel-Manning the Confusion

Wrong approach: Always apply ReLU to output.

Why it feels right: "Every layer needs activation for nonlinearity."

The fix: Output activation depends on the task:

• Regression (continuous values): Linear (identity) or sometimes ReLU if outputs must be non-negative (e.g., thrust ≥ 0)
• Binary classification: Sigmoid (outputs probability in [0,1])
• Multi-class classification: Softmax (outputs probability distribution)

Applying ReLU to regression output would clip negative predictions to zero—catastrophic if you need to predict negative elevator deflection (nose-down command).

Wrong approach: $\mathbf{W}^{(\ell)}$ has shape $(n_{\ell-1}, n_\ell)$ (transposed).

Why it feels right: "Weights connect from previous layer (rows) to current layer (columns)."

The fix: In the standard convention, $\mathbf{W}^{(\ell)}$ is $(n_\ell \times n_{\ell-1})$ so that $\mathbf{W}^{(\ell)} \mathbf{a}^{(\ell-1)}$ gives a column vector of size $n_\ell$. Each row of $\mathbf{W}^{(\ell)}$ contains the incoming weights for one neuron in layer $\ell$.

Check: If $\mathbf{a}^{(\ell-1)}$ is $(n_{\ell-1}, 1)$ and $\mathbf{z}^{(\ell)}$ should be $(n_\ell, 1)$, then $\mathbf{W}^{(\ell)}$ must be $(n_\ell, n_{\ell-1})$ for the product to be valid.

Wrong approach: During backpropagation, recompute the forward pass from scratch to get activations.

Why it feels right: "I need $\mathbf{a}^{(\ell)}$ values for gradient calculations."

The fix: Cache $\mathbf{z}^{(\ell)}$ and $\mathbf{a}^{(\ell)}$ during the forward pass. Backprop needs these (especially $\mathbf{z}^{(\ell)}$ for computing activation derivatives), and recomputing doubles the cost. In aerospace applications running on resource-constrained hardware, this memory-vs-compute tradeoff is critical.

Implementation tip:

# Store in a dictionary during forward pass
cache = {}
cache[f'z{ell}'] = z
cache[f'a{ell}'] = a
# Access during backprop
z = cache[f'z{ell}']

Mnemonic & Recall

Remember: W·A → Z → σ → A for each layer. The cycle repeats $L$ times.

A feedforward network is like a chain of decision-makers sitting in a row. The first person (input layer) sees the raw numbers from the instruments. They do some simple math (multiply by weights, add bias) and pass their answers to the next person. That person does MORE math on those answers and passes them along. This keeps happening through several people (hidden layers) until the last person (output layer) shouts out the final decision: "Tilt up 5 degrees!"

The "activation function" is like a rule: "If your answer is negative, just say zero instead" (that's ReLU). It makes the math more interesting because without it, all those people in the chain would just be doing one big multiplication, and you'd only need one person.

The "forward pass" means we start at the be

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)