Multi-layer perceptron architecture: Foundations

Before you can read a single formula in the parent note, you must own the tools it silently assumes. We build them in the exact order the machine uses them: a number → a list of numbers → a weighted sum → a bias shift → a bending curve → a whole layer at once → a probability.

1. A single number, and what a "neuron" holds

A neuron (also called a node) is nothing mystical. It is a little box that will eventually hold one number. That number is called its activation.

Why do we need this idea at all? Because a network's "thought" at any moment is just the collection of brightnesses across all its boxes. Learning = adjusting the machine so those brightnesses light up sensibly.

Figure — Multi-layer perceptron architecture

Look at the figure: three boxes glowing at different brightnesses. Each glow is one activation number. That is the entire content of a "neuron holding a value."

2. A list of numbers — the vector $x$

The parent note writes the input as $x = [x_{1}, x_{2}, \dots, x_{n}]^{T}$ . Let us dismantle that notation piece by piece — nothing here is allowed to be mysterious.

The bold $x$ ::: means "this is a whole list, treat it as one object."
The subscript $x_{i}$ ::: means "the single number sitting in slot $i$ ."
The T superscript ( $[\dots]^{T}$ ) ::: means transpose — flip a horizontal row into a vertical column. We stack numbers vertically because that is how the weighted-sum machinery (Section 5) lines them up.

Why does the topic need vectors? Because an input (10 tumor measurements, 784 pixels) is naturally many numbers at once. Bundling them into one bold $x$ lets us talk about "the input" as a single thing.

3. The weighted sum — earning $\sum$ , $w$ , and $\sum_{i} w_{i} x_{i}$

Here is the heart of the machine. A neuron does not treat all its inputs equally — it weights them.

Now the sum. The symbol $\sum$ (capital Greek "sigma") is just shorthand for "add up a bunch of terms":

$\sum_{i} w_{i} x_{i} = w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n}$

Why this exact tool and not plain addition? Plain addition $x_{1} + x_{2} + \dots$ treats every input as equally important — the machine could never learn that "radius matters a lot, texture barely at all." Multiplying each input by its own tunable weight before adding is what makes the neuron adjustable. Training is nothing but nudging these $w$ values.

4. The bias $b$ — a shift knob

The parent note adds a $+ b$ to every weighted sum. Why?

Putting Sections 3 and 4 together gives the parent's pre-activation:

$z = weighted vote i \sum w_{i} x_{i} + shift b$

The letter $z$ is just the name we give to "the number right before we bend it." The parent writes it as $z_{j}^{(l)}$ — read the superscript $(l)$ as "which layer" and the subscript $j$ as "which neuron in that layer." They are address labels, nothing more.

$z_{j}^{(l)}$ decoded ::: the pre-activation number of neuron $j$ living in layer $l$ .
$w_{j i}^{(l)}$ decoded ::: the weight feeding into neuron $j$ (layer $l$ ) from neuron $i$ of the previous layer.

5. Bending the result — the activation function $σ$

So far $z$ is a linear thing: doubling inputs doubles $z$ . Stacking linear steps gives you… still just one linear step (the parent proves $W_{3} W_{2} W_{1} x = W_{combined} x$ ). To get real power we must bend the number. That bender is $σ$ .

The three curves the parent uses, in plain words and pictures:

ReLU, $σ (z) = max (0, z)$ ::: "keep positives, flatten negatives to 0" — a hinge/ramp.
Sigmoid, $σ (z) = \frac{1}{1 + e ^{- z}}$ ::: a smooth S that squeezes any number into the range $(0, 1)$ .
Tanh, $σ (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$ ::: an S centred at 0, squeezing into $(- 1, 1)$ .

Here $e$ is Euler's number $\approx 2.718$ , and $e^{- z}$ means "raise $e$ to the power $- z$ ." Why exponentials? Because $e^{- z}$ shrinks toward 0 for big $z$ and blows up for very negative $z$ — exactly the tug-of-war that produces the smooth S-shape and, later, the probability-maker in the output layer. No simpler function bends smoothly between two flat ends the way $e$ does.

You now own the full single-neuron pipeline: $a = σ (\sum_{i} w_{i} x_{i} + b)$ .

6. Doing a whole layer at once — matrices $W$

A layer has many neurons, each with its own weighted sum. Writing $\sum$ for every neuron is exhausting, so we pack all the weights into one grid called a matrix.

How multiplication works — it is just dot products. The operation $W a$ means: take each row of $W$ and dot-product it with the column $a$ (the dot product you earned in Section 3 — multiply matching slots, add them up). Each row produces one number, so a $16 \times 10$ matrix hitting a $10 \times 1$ column yields $16$ numbers stacked into a $16 \times 1$ column. That is why "row times column" is precise: row-times-column = dot product.

Now the whole layer collapses into two lines:

$z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)}, a^{(l)} = σ (z^{(l)})$

Every bold letter here is a vector from Section 2: $a^{(l - 1)}$ is the incoming activations-vector, $b^{(l)}$ is the biases-vector, $z^{(l)}$ is the pre-activations stacked up, and $σ$ applied to a vector means "bend every entry, one at a time."

Why the matrix form at all? It is the same maths as Sections 3–5 — a pile of dot products — but written as a single symbol. Beyond neat notation, computers have hardware (and libraries like NumPy and PyTorch) built to crunch matrix multiplications extremely efficiently, so running all neurons in one matrix step is typically far faster in practice than writing an explicit loop over neurons in plain Python. (The exact speed-up depends on your hardware and problem size — the point is simply "one bulk operation beats many tiny ones.")

Recall

Shape check for a $16$ -neuron layer fed by $10$ inputs. Size of $W$ ::: $16 \times 10$ (rows = new neurons, columns = incoming numbers). Size of $a^{(l - 1)}$ ::: $10 \times 1$ (a column of the 10 inputs). Size of the result $z$ ::: $16 \times 1$ (one pre-activation per new neuron).

7. Turning scores into probabilities — softmax and $e$

The output layer for many classes needs numbers that behave like probabilities: all positive, all adding to 1. The tool that does this is called softmax, and the parent writes it as:

$softmax (z_{i}) = \frac{e ^{z_{i}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$

Read it slowly with the tools you now have:

$K$ ::: how many classes (e.g. 10 for digits).
$e^{z_{i}}$ ::: exponentiate score $i$ — guarantees a positive number (Section 5).
$\sum_{j = 1}^{K} e^{z_{j}}$ ::: add up the exponentials of all scores — the grand total (Section 3's $\sum$ ).
dividing ::: turns each positive number into a share of the whole, so the shares add to exactly 1.

That is why exponentiation appears: it is the cheapest smooth way to force positivity and amplify the biggest score into the biggest probability.

Prerequisite map

The diagram below shows how each foundation you just built feeds into the next, ending at the parent topic. Read it bottom-up: a single number becomes a list, a list plus weights becomes a dot-product sum, the sum plus bias becomes $z$ , $z$ bent by $σ$ becomes an activation, many neurons pack into a matrix, and the output stage uses $e$ for softmax — all converging on the MLP.

Each arrow is a "you need the tail before you can use the head" dependency: for example, you cannot understand the matrix step (G) until you own the dot product (D), which itself needs both a vector (B) and a weight (C).

Where the single neuron and its linear limits come from: 3.1.01-SinglePerceptron-and-Linear-Separability
How the weights actually get tuned: 3.1.03-Backpropagation-Algorithm and 3.2.01-Gradient-Descent-Optimization
A deeper tour of the bending curves $σ$ : 3.1.04-Activation-Functions
Where the raw input vector $x$ comes from: 2.1.05-Feature-Engineering
Prefer Hindi/Hinglish? 3.1.02 Multi-layer perceptron architecture (Hinglish)

Equipment checklist

Self-test — cover the right side and answer before revealing.

What does the bold $x$ signal, versus plain $x_{1}$ ? ::: Bold = the whole ordered list (vector); $x_{1}$ = the single number in slot 1.
What is a weights-vector $w$ , a biases-vector $b$ , an activations-vector $a$ ? ::: Bold columns holding, respectively, all of a neuron's weights, one bias per neuron in a layer, and every neuron's activation in a layer.
What does capital $\sum_{i} w_{i} x_{i}$ compute? ::: $w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n}$ , a single weighted-sum total.
What is the dot product $w \cdot x$ ? ::: Multiply matching slots and add them all up — exactly the weighted sum $\sum_{i} w_{i} x_{i}$ .
Why multiply inputs by weights before adding? ::: So the neuron can learn which inputs matter; plain addition treats all inputs equally and can't be tuned.
What does the bias $b$ let a neuron do? ::: Fire (or refuse) independent of the inputs — it shifts the decision boundary off the origin.
What is the pre-activation $z$ ? ::: The weighted sum plus bias, before the curve is applied: $z = \sum w_{i} x_{i} + b$ .
Why apply a non-linear $σ$ ? ::: Without bending, stacking layers collapses to one linear map $W_{combined} x$ ; $σ$ breaks that so depth adds real power.
Lowercase $σ$ vs capital $Σ$ ? ::: $σ$ bends one number with a curve; $\sum$ adds many numbers up.
What does "row times column" in $W a$ actually mean? ::: A dot product of that matrix row with the input column — one number per row.
What is $e$ and why does softmax use it? ::: Euler's number $\approx 2.718$ ; $e^{z}$ is always positive and smoothly amplifies larger scores, so shares can sum to 1.
What do the superscript $(l)$ and subscript $j$ in $z_{j}^{(l)}$ mean? ::: $(l)$ = which layer; $j$ = which neuron inside that layer — address labels only.
What is the shape of $W^{(l)}$ for a layer of $m$ neurons fed by $n$ inputs? ::: $m \times n$ (rows = new neurons, columns = incoming numbers).