Multi-layer perceptron architecture: Visual walkthrough

We assume you know nothing but this: a weight is a number that says "how much do I care about this input", and a bias is a number that shifts a decision. Every other symbol is earned below.

Step 1 — What a single neuron actually draws

WHAT. A single perceptron takes two inputs $x_{1}$ and $x_{2}$ , multiplies each by a weight, adds a bias, and asks "is the total positive or negative?"

$z = care about x_{1} w_{1} x_{1} + care about x_{2} w_{2} x_{2} + shift b$

Read left to right: $w_{1} x_{1}$ is " $x_{1}$ scaled by how much it matters", $w_{2} x_{2}$ is the same for $x_{2}$ , and $b$ nudges the whole thing up or down. The neuron outputs 1 if $z > 0$ , else 0.

WHY. We want the simplest possible decision-maker so we can see its limit clearly. This is it.

PICTURE. Set $z = 0$ . That equation $w_{1} x_{1} + w_{2} x_{2} + b = 0$ is a straight line on the $(x_{1}, x_{2})$ plane. One side of the line is "1", the other side is "0". A single neuron can only draw a straight line.

Figure — Multi-layer perceptron architecture

Step 2 — The problem no line can solve

WHAT. Meet XOR ("exclusive or"). Four points, two classes:

$x_{1}$	$x_{2}$	class
0	0	0
0	1	1
1	0	1
1	1	0

The two "1" points sit on opposite corners of a square. The two "0" points sit on the other two opposite corners.

WHY. This is the smallest problem that breaks a single line, so it is the perfect stress test.

PICTURE. Try to draw ANY straight line that puts both orange dots on one side and both blue dots on the other. You physically cannot — the classes interlock diagonally. This is the wall a single neuron hits.

Step 3 — The trick: bend space, don't bend the line

WHAT. Instead of one clever line, use two ordinary lines as helper questions, then a third neuron that combines their answers.

Neuron $h_{1}$ asks: "am I above line A?" → answer $a_{1} \in {0, 1}$
Neuron $h_{2}$ asks: "am I above line B?" → answer $a_{2} \in {0, 1}$

These two neurons are the hidden layer. They don't touch the final answer directly — they build new coordinates.

WHY. Each helper is still just a line (Step 1). But their answers $(a_{1}, a_{2})$ form a new space. If XOR becomes separable in that new space, a single output neuron finishes the job. We are not making the line smarter — we are moving the points to where a line suffices.

PICTURE. Below, line A separates "top-left corner" and line B separates "bottom-right corner". Together they carve the plane into regions.

Step 4 — Concrete weights that work

WHAT. Pick these hidden neurons (using the step function: output 1 if $z > 0$ ):

$h_{1} : z_{1} = w 1 x_{1} + w 1 x_{2} - - b 0.5$ $h_{2} : z_{2} = w 1 x_{1} + w 1 x_{2} - - b 1.5$

$h_{1}$ fires ( $a_{1} = 1$ ) whenever at least one input is on ( $x_{1} + x_{2} \geq 1$ ). This is OR.
$h_{2}$ fires ( $a_{2} = 1$ ) whenever both inputs are on ( $x_{1} + x_{2} \geq 2$ ). This is AND.

WHY. XOR = "at least one, but NOT both" = OR minus AND. So we want the output neuron to say "fire when $a_{1} = 1$ and $a_{2} = 0$ ".

PICTURE. Watch the four points land in the new $(a_{1}, a_{2})$ space. Three of the four map to distinct spots; crucially the two "1"-class points collapse to $(a_{1}, a_{2}) = (1, 0)$ while the two "0"-class points go elsewhere.

Mapping table (the heart of the whole page):

$x_{1}, x_{2}$	$x_{1} + x_{2}$	$a_{1}$ (OR)	$a_{2}$ (AND)	class
0,0	0	0	0	0
0,1	1	1	0	1
1,0	1	1	0	1
1,1	2	1	1	0

Step 5 — In the new space, a single line finishes the job

WHAT. In the transformed $(a_{1}, a_{2})$ space, the two class-1 points both sit at $(1, 0)$ , class-0 points sit at $(0, 0)$ and $(1, 1)$ . Now build the output neuron:

$z_{out} = trust a_{1} 1 a_{1} - veto a_{2} 2 a_{2} - shift 0.5$

$+ 1 \cdot a_{1}$ : reward "at least one input on".
$- 2 \cdot a_{2}$ : strongly punish "both on" (the $- 2$ overpowers the $+ 1$ ).
$- 0.5$ : keeps the all-zero case negative.

WHY. These weights are exactly "OR and not AND". A single line now separates the points — because we already re-shaped them in Step 4.

PICTURE. One clean straight line in $(a_{1}, a_{2})$ space splits the orange point from the two blue points. The wall of Step 2 is gone.

Check all four cases:

$a_{1}, a_{2}$	$z_{out} = a_{1} - 2 a_{2} - 0.5$	output	correct?
0,0	$- 0.5$	0	✓ (class 0)
1,0	$+ 0.5$	1	✓ (class 1)
1,1	$- 1.5$	0	✓ (class 0)

Every row matches. XOR solved by a 2-neuron hidden layer + 1 output neuron.

Step 6 — Why the bend was essential (the degenerate case)

WHAT. Suppose we delete the non-linearity — the hidden neurons just pass $z$ straight through (no step, no activation). Then each hidden output is a linear combo of inputs, and the output is a linear combo of those.

$z_{out} = W_{out} (W_{hid} x) = one matrix (W_{out} W_{hid}) x$

WHY. Two matrices multiplied together are still one matrix. So a "deep" linear network collapses back to Step 1 — a single line. No number of linear layers escapes the XOR wall.

PICTURE. The stacked linear layers fold flat into one line — the same failing line from Step 2. The kink (the step / ReLU / sigmoid) is the only thing that keeps the layers from collapsing.

The one-picture summary

The full journey on one canvas: left — XOR unsolvable by any line; middle — two hidden neurons re-shape the four points into a new space; right — in that new space, one line cleanly separates them. Depth + non-linearity = the ability to reshape space until a line works.

Recall Feynman retelling — explain it to a friend

Imagine four coins on a table: two blue on one diagonal, two orange on the other. Your only tool is a single straight ruler, and you must put both blues on one side, both oranges on the other. Impossible — they interlock. So instead you hire two assistants. Assistant One raises a flag if at least one coin near you is heads-up. Assistant Two raises a flag only if both are. Now you stop looking at the coins — you look only at the two flags. In flag-world, the orange cases all say "flag-1 up, flag-2 down", and the blue cases say something different. Suddenly one ruler separates them perfectly. That's an MLP: the assistants are the hidden layer, their yes/no bends are the activation function, and you with the final ruler are the output neuron. The magic isn't a smarter ruler — it's moving the coins into a room where a plain ruler already works. And the twist: if the assistants only ever add and scale (no yes/no flag, no bend), their room is just the old table rotated — the coins still interlock. The bend is everything.

Recall

A single neuron can only draw what shape? ::: A straight line (a linear decision boundary). Why can't a single perceptron solve XOR? ::: XOR's two classes interlock on opposite corners — no single straight line separates them; XOR is not linearly separable. What does the hidden layer actually do geometrically? ::: It maps the input points into a new coordinate space where the classes become linearly separable. If you remove all activation functions, what does a deep MLP collapse to? ::: A single linear transformation (one matrix), equivalent to one perceptron — still just one line. In the XOR solution, what two logic gates do the hidden neurons compute? ::: OR ( $h_{1}$ , threshold at $x_{1} + x_{2} \geq 1$ ) and AND ( $h_{2}$ , threshold at $x_{1} + x_{2} \geq 2$ ); XOR = OR minus AND.

Related: 3.1.01-SinglePerceptron-and-Linear-Separability · 3.1.03-Backpropagation-Algorithm · 3.1.04-Activation-Functions · 3.2.01-Gradient-Descent-Optimization · 2.1.05-Feature-Engineering · 4.1.01-Convolutional-Neural-Networks · 3.3.02-Dropout-Regularization