Forward propagation computation

WHY does forward propagation exist?

WHAT problem does it solve? A neural network is a function $\hat{y} = f(x)$ with millions of tunable knobs (weights). Before we can learn those knobs (backprop), we must be able to compute the prediction for a given input. Forward propagation is that computation — the "run the network" step.

WHY this design (linear + nonlinear)?

• A pure stack of linear layers collapses: $W_2(W_1 x) = (W_2 W_1)x$ — still linear, no matter how deep. So depth would be useless.
• Inserting a nonlinear activation between layers breaks that collapse, letting the network approximate curved, complex functions. This is the whole reason deep nets are powerful.

HOW: building it from first principles

Step 1 — A single neuron

A neuron receives inputs $x_1,\dots,x_n$, each with importance $w_i$, plus a bias $b$.

Why the bias? Without $b$, every neuron's decision boundary must pass through the origin. The bias shifts the activation left/right, giving freedom to fire even when inputs are zero.

Step 2 — Vectorize one layer

Stacking $m$ neurons in a layer, each has its own weight row. Collect them:

Why does row $j$ of $W$ contain neuron $j$'s weights? Because matrix–vector multiply computes each output as a dot product of a row with the input. Row $j$ · input $= z_j$ — exactly the weighted sum for neuron $j$.

Step 3 — Chain the whole network

With input $\mathbf{a}^{[0]} = \mathbf{x}$, repeat for $\ell = 1,2,\dots,L$:

$\mathbf{a}^{[0]} = \mathbf{x} \;\longrightarrow\; \mathbf{z}^{[1]},\mathbf{a}^{[1]} \;\longrightarrow\;\cdots\;\longrightarrow\; \mathbf{a}^{[L]} = \hat{y}$

The final activation $\mathbf{a}^{[L]}$ is the prediction. For classification $\phi^{[L]}$ is usually softmax; for regression it's often identity.

Step 4 — Batch form (why we use matrices)

Instead of one input vector, process $N$ examples at once as columns of $X$ ($n\times N$): $Z^{[\ell]} = W^{[\ell]}A^{[\ell-1]} + \mathbf{b}^{[\ell]},\qquad A^{[\ell]} = \phi(Z^{[\ell]})$ The bias $\mathbf{b}^{[\ell]}$ ($m\times 1$) is broadcast across all $N$ columns. Why batch? One big matmul uses hardware (GPU) far more efficiently than $N$ small ones.

Common activations (and WHY each)

Common hidden activations: ReLU $\phi(z)=\max(0,z)$ (cheap, no vanishing gradient for $z>0$), sigmoid $\phi(z)=\frac{1}{1+e^{-z}}$ (squashes to $(0,1)$), tanh (squashes to $(-1,1)$).

Worked Example 1 — a tiny 2-layer network by hand

Input $\mathbf{x} = \begin{bmatrix}1\\2\end{bmatrix}$. Hidden layer (2 units), ReLU: $W^{[1]}=\begin{bmatrix}0.5 & -0.5\\ 1 & 1\end{bmatrix},\quad \mathbf{b}^{[1]}=\begin{bmatrix}0\\-1\end{bmatrix}$ Output layer (1 unit), identity: $W^{[2]}=\begin{bmatrix}2 & -1\end{bmatrix},\quad b^{[2]}=0.5$

Layer 1 pre-activation — Why? Combine inputs with weights + bias: $\mathbf{z}^{[1]}=\begin{bmatrix}0.5(1)+(-0.5)(2)+0\\ 1(1)+1(2)-1\end{bmatrix}=\begin{bmatrix}-0.5\\2\end{bmatrix}$

Layer 1 activation — Why? Apply ReLU (clip negatives to 0): $\mathbf{a}^{[1]}=\begin{bmatrix}\max(0,-0.5)\\ \max(0,2)\end{bmatrix}=\begin{bmatrix}0\\2\end{bmatrix}$

Layer 2 — Why? Same recipe, identity activation: $z^{[2]}=2(0)+(-1)(2)+0.5=-1.5,\qquad \hat{y}=-1.5$

Worked Example 2 — softmax output

Logits $\mathbf{z}^{[L]}=\begin{bmatrix}2\\1\\0\end{bmatrix}$. Why exponentiate? To turn scores into positive, comparable weights: $e^{2}=7.389,\; e^{1}=2.718,\; e^{0}=1,\quad \text{sum}=11.107$ $\hat{y}=\begin{bmatrix}0.665\\0.245\\0.090\end{bmatrix}$ Why divide by the sum? So the outputs form a probability distribution ($\sum=1$). Class 0 is most likely.

Forecast-then-Verify

Common mistakes (Steel-manned)

Flashcards

Connections

• Activation functions — the nonlinearities $\phi$ that make forward prop expressive.
• Backpropagation — reuses the cached $\mathbf z,\mathbf a$ from forward prop to compute gradients.
• Matrix multiplication — the computational engine of each layer.
• Softmax and cross-entropy loss — output activation + how prediction becomes a loss.
• Universal approximation theorem — why one nonlinear hidden layer already gives huge power.
• Vanishing and exploding gradients — downstream consequence of activation choices made here.

Concept Map

Hinglish (regional understanding)