Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Visual walkthrough

Everything below starts from a child who has seen multiplication and addition and nothing else. Every symbol is earned before it is used.

Step 1 — One input, one weight, one bias: the straight line

WHAT. The smallest possible neuron takes one number in — call it $x$ — and produces one number out. Before any fancy function, the neuron does just two things: it scales $x$ by a number $w$ , then shifts the result by a number $b$ . We name this scaled-and-shifted number $z$ :

z = how steep w \cdot the input x + where it crosses b

$x$ — the input number (say, an airspeed reading).
$w$ — the weight: it stretches or squashes $x$ . Big $w$ = steep line, $x$ matters a lot.
$b$ — the bias: it lifts the whole line up or down without touching its steepness. This $+ b$ is exactly what makes the map affine rather than strictly linear.
$z$ — the weighted sum, the neuron's raw score before any bending.

WHY these two operations and no others? Scaling ( $w \cdot x$ ) is the only way to say "this input is 3× more important." Shifting ( $+ b$ ) is the only way to let the neuron output something non-zero when $x = 0$ . Together they are the most general straight line you can write — a line that need not pass through the origin, i.e. an affine line. That is exactly the point — and also exactly the limitation we will hit.

PICTURE. The graph of $z = w x + b$ is a ruler-straight line. $w$ is its tilt; $b$ is the height where it crosses the vertical axis.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Step 2 — Many inputs: the line becomes a weighted sum

WHAT. Real neurons watch many numbers at once: airspeed, altitude, angle-of-attack. Call them $x_{1}, x_{2}, \dots, x_{n}$ . Each gets its own weight $w_{1}, \dots, w_{n}$ . The neuron multiplies each pair and adds them all, then adds one shared bias:

z = every input, scaled by its importance w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n} + one shared shift b

The tidy shorthand for "add up all the $w_{i} x_{i}$ " is the sum symbol $\sum$ (a capital Greek "S", for Sum):

z = i = 1 \sum n w_{i} x_{i} + b

$\sum_{i = 1}^{n}$ — "let $i$ run from $1$ to $n$ and add every term." Nothing mysterious: it is just a compact for loop that means $w_{1} x_{1} + \dots + w_{n} x_{n}$ .
Everything else is exactly Step 1, repeated once per input.

WHY still just addition and scaling? Because we have not yet added anything else. Even with a thousand inputs, $z$ is still a flat (affine) plane in disguise — the multi-input twin of the straight line. Hold that thought; it is about to bite us.

PICTURE. With two inputs the "line" becomes a tilted flat sheet floating over the $(x_{1}, x_{2})$ floor. Its height at any point is $z$ .

Step 3 — Stack two neurons and watch the line refuse to bend

WHAT. Deep learning's promise is stacking: feed one layer's output into another. First we must upgrade our shorthand from single numbers (scalars) to lists of numbers (vectors) and grids of numbers (matrices), because a real layer has many neurons.

Definition Vectors, matrices, and what

$W x + b$ means

A vector is a column of numbers, e.g. an input $x = [x_{1} x_{2}]$ with $2$ entries. We say $x$ has dimension $2$ .
A matrix $W$ is a grid of numbers. If layer 1 has $3$ neurons each reading those $2$ inputs, $W_{1}$ is a $3 \times 2$ grid (3 rows, 2 columns) — one row of weights per neuron.
Matrix times vector, $W_{1} x$ : take each row of $W_{1}$ , multiply it entry-by-entry with $x$ , and sum — that is exactly the weighted sum $\sum w_{i} x_{i}$ from Step 2, done once per neuron. The result is a vector with $3$ entries (one per neuron).
Adding the bias vector $b_{1}$ (also $3$ entries) shifts each neuron's score. So $h = W_{1} x + b_{1}$ is just "run Step 2 for all $3$ neurons at once," giving a $3$ -entry vector $h$ .
Layer 2 with (say) $1$ neuron reading those $3$ has $W_{2}$ of shape $1 \times 3$ , bias $b_{2}$ a single number, output $y$ a single number.

With that vocabulary, layer 1 turns the input vector $x$ into the hidden vector $h$ , and layer 2 turns $h$ into $y$ :

Layer 1: h = W_{1} x + b_{1}, Layer 2: y = W_{2} h + b_{2}

Substitute the first into the second:

y = W_{2} (this is h W_{1} x + b_{1}) + b_{2} = one new weight box, call it W_{eff} (W_{2} W_{1}) x + one new bias, call it b_{eff} (W_{2} b_{1} + b_{2})

Here $W_{2} W_{1}$ is matrix times matrix ( $1 \times 3$ times $3 \times 2$ gives a $1 \times 2$ grid) — one combined weight box — and $W_{2} b_{1} + b_{2}$ is one combined bias number.

WHY does this matter? Look at the final line: it has the exact shape $y = W_{eff} x + b_{eff}$ — a single affine map again. Two layers collapsed into one. Multiplying two affine functions and adding shifts can never escape straightness. Stack a hundred; still one straight (affine) map.

PICTURE. Two straight ramps chained together produce one straight ramp — the fold you hoped for never appears.

Step 4 — The problem a straight line cannot solve: XOR

WHAT. Here is a task that needs a curve. Four points on a floor:

$x_{1}$	$x_{2}$	wanted label
0	0	0 (blue)
0	1	1 (orange)
1	0	1 (orange)
1	1	0 (blue)

This is XOR: output 1 when the inputs disagree. We want to draw a boundary that keeps the two orange points on one side and the two blue points on the other.

WHY it is impossible with Step 3's machine. The blue points sit at opposite corners; so do the orange. No single straight line can separate two diagonal pairs — try it and one point is always on the wrong side. Since stacked affine layers = one straight boundary (Step 3), the whole affine network cannot learn XOR.

PICTURE. Every straight cut you attempt leaves one point stranded.

Step 5 — Insert one bend: the activation function

WHAT. Between the two layers we slip in a function $f$ that is not a straight line — an activation function. Now:

y = W_{2} \cdot apply the bend before layer 2 f (W_{1} x + b_{1}) + b_{2}

The simplest possible bend is ReLU (Rectified Linear Unit) — "let positives through, flatten negatives to zero":

f (z) = max (0, z) = {z 0 z > 0 z \leq 0

For $z > 0$ : output equals input — a $4 5^{\circ}$ ramp.
For $z \leq 0$ : output is flat zero — the "door is shut."
The corner at $z = 0$ is the crucial thing: it is a kink, the very non-straightness Step 3 lacked.

WHY does a single kink rescue us? A concrete 1-D check. Take two affine pieces of the same input $x$ : $a = w_{1} x + b_{1}$ and $c = w_{2} x + b_{2}$ . Adding them, $a + c = (w_{1} + w_{2}) x + (b_{1} + b_{2})$ — still one straight line, no escape. But push each through ReLU first and then add:

g (x) = max (0, w_{1} x + b_{1}) + max (0, w_{2} x + b_{2}) .

Try numbers $w_{1} = 1, b_{1} = 0$ and $w_{2} = - 1, b_{2} = 0$ , i.e. $g (x) = max (0, x) + max (0, - x) = ∣ x ∣$ . That is the absolute-value "V" — provably not of the form $(constant) \cdot x + constant$ , because its slope is $- 1$ on the left and $+ 1$ on the right. The $max$ refuses to distribute across the sum: $max (0, a) + max (0, c) \neq = max (0, a + c)$ in general (at $x = - 1$ : $1 + 0 = 1$ on the left, but $max (0, - 1 - 1 \cdot (- 1))$ -style bookkeeping gives $0$ ). One bend has bought us a genuine corner — the collapse of Step 3 is broken.

PICTURE. ReLU: flat on the left, a rising ramp on the right, with a sharp elbow at the origin.

Step 6 — Watch the bends fold space and solve XOR

WHAT. Give the tiny network two hidden ReLU neurons. Each ReLU contributes one fold. Combining two folds, the network builds a boundary shaped like a bent line — a "V" or a corner — instead of a straight cut.

WHY this works where Step 4 failed. A corner can hug the two blue points into one region and push the two orange points outside it. The impossible straight-line separation becomes an easy folded-region separation. This is the payoff of Step 5's kink, made visible.

PICTURE. The learned boundary bends around the blue diagonal; orange and blue now sit cleanly apart.

Blue region ← both points labelled 0.
Orange region ← both points labelled 1.
The boundary is piecewise-straight (made of ReLU segments) but globally curved — exactly what one affine layer could never produce.

Step 7 — Not every bend is equal: three activations side by side

WHAT. ReLU is one bend; there are smoother ones. All three from the parent note, on one axis:

ReLU: max (0, z), σ (z) = \frac{1}{1 + e ^{- z}}, tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}

Reading each symbol where it sits:

$e^{- z}$ — the number $e \approx 2.718$ raised to $- z$ ; it shrinks fast as $z$ grows, so $σ$ climbs toward $1$ .
$σ (z)$ — sigmoid, squashes any input into the open interval $(0, 1)$ : a confidence meter.
$tanh (z)$ — same S-shape but squashed into $(- 1, 1)$ , so it is zero-centred (negatives allowed).

WHY care about which one? Their steepness differs, and steepness is what gradient descent rides on:

ReLU's slope is exactly $1$ for $z > 0$ — learning signals pass undiminished.
Sigmoid's slope never exceeds $0.25$ (its steepest point, at $z = 0$ ); tanh's never exceeds $1$ but both flatten to nearly $0$ at the extremes ("saturation").

PICTURE. ReLU's hard elbow, sigmoid's gentle $(0, 1)$ S with its $0.25$ max-slope tangent marked, tanh's symmetric $(- 1, 1)$ S with its slope- $1$ tangent and its $\pm 1$ asymptotes — all overlaid.

Step 8 — The degenerate case: a flat activation is no activation

WHAT. What if someone "simplifies" and picks a linear activation, $f (z) = z$ ? Then

y = W_{2} (W_{1} x + b_{1}) + b_{2} = W_{eff} x + b_{eff}

— we are right back at Step 3's collapse. Likewise a constant activation $f (z) = c$ kills all input dependence entirely ( $y$ ignores $x$ ).

WHY show this. It nails the boundary of the whole idea: the bend must be genuinely non-linear and input-dependent. A straight or flat $f$ silently undoes everything. This is also why a ReLU neuron whose $z$ is negative for all data is dead — it is locally the flat, useless case, and (see the parent's Dead-ReLU note) its gradient is $0$ so it never recovers.

PICTURE. Two "bad bends": the identity line (collapses to affine) and a flat constant (ignores input) — neither can fold space.

The one-picture summary

Read the four panels left-to-right, top-to-bottom — they replay the whole page:

Top-left "linear = straight": one affine layer $z = w x + b$ is a ruler line (Steps 1–3). Stacking more never bends it.
Top-right "line fails XOR": the four XOR dots; the red dashed line is any straight cut, and it always strands a point (Step 4).
Bottom-left "add a bend (ReLU)": insert $max (0, z)$ — the green ramp with the red kink at the origin is the new non-linear ingredient (Step 5).
Bottom-right "folds solve XOR": two ReLU folds carve a corner-shaped boundary that finally separates blue from orange (Step 6).

The through-line: stretch (weights) → slide (bias, affine) → BEND (activation). Only the bend escapes the straight-boundary trap.

Recall Feynman retelling — say it back in plain words

Start with a machine that only knows two tricks: stretch a number and slide it. Stretch-and-slide is what grown-ups call an affine map — a straight ruler line that need not pass through zero. Chain a hundred of these machines and — surprise — you still only get one straight line, because stretching a stretch is just a bigger stretch and sliding a slide is just a bigger slide. Now here's a puzzle four dots make, called XOR: two dots want "yes," two want "no," and they sit on opposite corners. Try to separate them with a ruler — you can't, ever; someone is always on the wrong side. So we add one new trick: after stretching, we bend the number — the simplest bend just flattens all the negatives to zero (that's ReLU, a door that shuts below zero). That single kink means the chained machines are no longer one straight ruler; they can fold the paper. (Proof you can feel: $max (0, x) + max (0, - x)$ is the V-shape $∣ x ∣$ — no ruler makes a V.) Two folds make a corner, and a corner can finally wrap around the two "no" dots and leave the two "yes" dots outside. That's the whole secret of neural networks: stretch, slide, and — crucially — bend. Pick a bend that's actually curved (ReLU, sigmoid, tanh); pick a flat or straight one and you've thrown the magic away. And that one sharp point at the bottom of ReLU has no proper slope, so the computer just picks any value between $0$ and $1$ there and carries on.

Recall Quick self-test

Why can't stacked affine layers learn XOR? ::: Because stacking affine layers collapses to a single affine layer ( $W_{2} (W_{1} x + b_{1}) + b_{2} = W_{eff} x + b_{eff}$ ), which can only draw one straight boundary — and no straight line separates XOR's diagonal pairs. What does the kink in ReLU buy you? ::: A genuine non-linearity: $max (0, z)$ cannot be rewritten as a straight line (e.g. $max (0, x) + max (0, - x) = ∣ x ∣$ ), so layers stop collapsing and the network can fold its input space into curved boundaries. What is the difference between a linear and an affine map? ::: Linear is pure scaling $w x$ (through the origin); affine adds a shift, $w x + b$ . A "linear layer" $W x + b$ is really affine whenever $b \neq = 0$ . What happens at $z = 0$ for ReLU? ::: It is non-differentiable (left slope $0$ , right slope $1$ ); software uses a subgradient — any value in $[0, 1]$ , usually $0$ . Which activation is zero-centred, and why is that nice? ::: $tanh$ , range $(- 1, 1)$ ; being symmetric about zero lets it output signed values and generally helps gradient descent converge.

Where this leads: the steepness of each bend controls how learning signals flow backward — the subject of Backpropagation and Gradient Descent and its failure mode, Vanishing and Exploding Gradients. Choosing the bend and the starting weights together is Weight Initialization Strategies.