Forward propagation computation: Foundations — AI-ML deep dive

Before you can read the forward-propagation page comfortably, you need to see what each symbol stands for. Below we take every letter, subscript, and shape from that page and build it from nothing — plain words first, then a picture, then why the topic needs it.

0. The absolute starting point: a number and a list of numbers

Everything in a neural network is made of numbers. A single number is called a scalar — just one value like $3$ or $- 0.5$ .

When we line several scalars up in a column, we get a vector. Think of it as a stack of numbers, one on top of the other.

The picture: look at the figure. The vector $x$ is drawn as a tall box with three stacked cells. The label $x_{2}$ (in red) points at the second cell from the top. That subscript is nothing more than a house-number telling you which cell.

Why the topic needs it: the network's input (an image, a row of measurements) arrives as a vector. The subscript $x_{i}$ lets us talk about "the $i$ -th input" without drawing the whole stack.

1. The summation symbol $\sum$ — "add up a whole list"

The parent page writes $z = \sum_{i = 1}^{n} w_{i} x_{i} + b$ . The tall symbol $\sum$ (Greek capital sigma) is just shorthand for "add these up as $i$ counts from $1$ to $n$ ."

Why the topic needs it: a neuron may have thousands of inputs. Writing $w_{1} x_{1} + w_{2} x_{2} + \dots + w_{1000} x_{1000}$ is absurd. $\sum$ packs it into one tidy symbol.

2. The dot product — "how much do two lists agree, weighted?"

The heart of one neuron is $w_{1} x_{1} + w_{2} x_{2} + \dots$ . That specific pattern — multiply matching components, then add them all up — is so common it has its own name: the dot product.

The picture: the figure shows two stacks side by side — weights $w$ and inputs $x$ . Red arrows pair up cell $i$ with cell $i$ ; each pair is multiplied, then all the products drop into one box at the bottom and are summed. That single box is the dot product.

Why this tool and not another? We want a neuron to output a single "score" that grows when the input matches what the neuron cares about. Multiply-then-add is exactly the operation that measures weighted agreement: a weight $w_{i}$ says "how much do I care about input $i$ ," and $w_{i} x_{i}$ is that input's contribution. No other simple operation captures "importance-weighted total" so directly.

See Matrix multiplication — the dot product is its single building block.

3. Weights $w_{i}$ and bias $b$ — the tunable knobs

The picture: the figure plots the weighted sum $z = w x$ as a straight line through the origin (black), and $z = w x + b$ as the same line slid upward by $b$ (red). Notice the black line is forced through the point $(0, 0)$ : with no bias, zero input must give zero output. The red line is free to cross zero wherever it likes — that freedom is exactly what $b$ buys.

Why the topic needs the bias: without it, every neuron's "decision line" must pass through the origin, a severe and pointless restriction. The bias adds shifting to complement the weights' scaling.

4. Pre-activation $z$ (the "logit") vs activation $a$

Two closely-named quantities appear everywhere. Keep them straight:

Why two names? Because the network needs both: $z$ is the "linear score," and $a$ is the "bent, usable output" that gets passed to the next layer. Backprop (see Backpropagation) treats them differently, so we never merge them.

5. The activation function $ϕ$ — the "bend"

$ϕ$ (Greek phi) is the name for any curve we push $z$ through. Its job is to be not a straight line, because straight lines are the one thing that would make a deep network useless.

The picture: three activation curves on the same axes, each with the red part highlighting its defining behaviour:

ReLU $ϕ (z) = max (0, z)$ — flat at zero for negative $z$ , then a straight ramp. It gates: negatives become $0$ .
Sigmoid $ϕ (z) = \frac{1}{1 + e ^{- z}}$ — an S-curve squashing everything into $(0, 1)$ .
tanh — an S-curve squashing into $(- 1, 1)$ .

Why a nonlinear $ϕ$ and not just the identity? If every $ϕ$ were a straight line, stacking layers would collapse: $W_{2} (W_{1} x) = (W_{2} W_{1}) x$ is still one linear map. The bend is what lets the network draw curved boundaries. Full treatment in Activation functions and Universal approximation theorem; the failure mode of some curves is Vanishing and exploding gradients.

Recall Why can't we skip

$ϕ$ ? Without a nonlinearity ::: any number of linear layers multiply out into a single linear layer, so depth adds zero extra power.

6. The letter $e$ and $e^{z}$ — used inside softmax

The output page uses $e^{z_{k}}$ . Here $e \approx 2.718$ is a fixed special number (Euler's number), and $e^{z}$ is the exponential function.

Why softmax uses it: to turn arbitrary scores (which can be negative) into positive, comparable weights, then divide by their total so they sum to $1$ — a probability distribution. Because $e^{z}$ is monotonic (bigger input → bigger output), the biggest score stays the most likely class. Details in Softmax and cross-entropy loss.

7. Matrices and shapes — $W$ , and the notation $m \times n$

One neuron is a dot product. A layer is many neurons at once, so we stack their weight-vectors into a grid of numbers called a matrix, written with a capital $W$ .

Why the shape must be $m \times n$ : to turn an $n$ -long input into an $m$ -long output, the matrix must have $n$ columns (to line up with the input) and $m$ rows (one per output). Get this backwards and the multiply is illegal. See Matrix multiplication.

8. Superscripts in brackets: $a^{[ℓ]}$ , $W^{[ℓ]}$

The bracketed number on top, like $^{[ℓ]}$ (the Greek letter ell, just an index), names which layer a quantity belongs to. It is not a power.

9. The hat: $\overset{y}{^}$

$\overset{y}{^}$ (read "y-hat") is the network's prediction — its guess for the true answer $y$ . The little hat is the universal symbol for "estimated / predicted."

How these feed the topic

Read top to bottom: numbers become vectors, vectors combine via dot products, dot products plus bias give $z$ , bending $z$ gives $a$ , stacking neurons gives matrices, labelling layers chains it all into forward propagation.

Return to the parent: Forward propagation computation.

Equipment checklist

Test yourself — you are ready when you can answer each without peeking.

What is the difference between

x

(bold) and

x_{i}

(plain with subscript)?

x

is the whole vector (the stack);

x_{i}

is a single component, cell number

i

inside it.

What does

\sum_{i = 1}^{n} w_{i} x_{i}

compute in words?

Multiply each weight by its matching input and add all the products up — the dot product.

Why is the operation inside a neuron a dot product and not, say, just addition?

The weights let each input contribute by its importance; multiply-then-add measures importance-weighted agreement, exactly what a neuron scores.

Does the bias scale or shift the weighted sum?

It shifts (it is added after the sum), letting the neuron fire even when all inputs are zero.

What is the difference between

z

and

a

z

is the raw weighted sum plus bias (pre-activation);

a

z

after bending through the activation

ϕ

Why must the activation

ϕ

be nonlinear?

Otherwise stacked linear layers collapse into one linear map and depth is wasted.

If a layer turns an

n

-vector into an

m

-vector, what is the shape of

W

m \times n

— output rows, input columns.

What does the bracketed superscript in

W^{[3]}

mean?

The layer number (a label), not an exponent — layer 3's weight matrix.

What does the hat in

\overset{y}{^}

signify?

A predicted/estimated value — the network's guess for the true

y

Why does softmax use

e^{z}

e^{z}

is always positive and monotonic, turning raw scores into positive comparable weights that (after dividing by their sum) form a valid probability distribution.