Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Worked examples

You have met the neuron and its three activation functions in the parent note. This page does one thing: it drags every activation function through every kind of number you could ever feed it — big positive, big negative, exactly zero, tiny, and real aircraft data — so that when you sit an exam or debug a network, you have already seen the case.

Before we start, a two-line recap of the only three rules we use, stated in plain words so no symbol is unearned.

The scenario matrix

Every neuron input is a single number $z$ . There are only so many shapes of number, and only so many activations. This grid is the complete world — each worked example below fills one row.

Cell	Scenario class	What we test	Example
A	$z > 0$ large	ReLU passes, sigmoid → 1, tanh → +1 (saturation)	Ex. 1
B	$z < 0$ large	ReLU kills to 0, sigmoid → 0, tanh → −1	Ex. 2
C	$z = 0$ exactly	The pivot point of all three	Ex. 3
D	$z$ small (near 0), both signs	Where gradients are strongest; symmetry	Ex. 4
E	Limiting behaviour	$z \to \pm \infty$ : what value is approached but never reached?	Ex. 5
F	Gradient / vanishing	derivative values, why deep sigmoid dies	Ex. 6
G	Real-world word problem	full neuron: weights + bias + activation	Ex. 7
H	Exam twist	dead ReLU + Leaky ReLU rescue	Ex. 8

See the whole landscape at once first — the shape of each curve tells you the answer before any arithmetic.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Cell A — large positive $z$

Worked example Example 1:

$z = + 6$ through all three Forecast: guess each output before reading on. What does ReLU give? What does sigmoid approach but not reach? What about tanh?

Step 1 — ReLU. $f (6) = max (0, 6) = 6$ . Why this step? When $z > 0$ , ReLU is the identity — it copies the number straight through. No squashing.

Step 2 — Sigmoid. $σ (6) = \frac{1}{1 + e ^{- 6}} = \frac{1}{1 + 0.002479} \approx 0.99753$ . Why this step? $e^{- 6}$ is tiny (a big positive $z$ makes $- z$ big negative, and $e$ to a big-negative power is near zero), so the denominator is just over 1, and the whole fraction sits just under 1.

Step 3 — Tanh. $tanh (6) = \frac{e ^{6} - e ^{- 6}}{e ^{6} + e ^{- 6}} = \frac{403.43 - 0.00248}{403.43 + 0.00248} \approx 0.99999$ . Why this step? $e^{- 6}$ is negligible next to $e^{6}$ , so top and bottom are both $\approx e^{6}$ and the ratio is $\approx 1$ .

Verify: All three are increasing functions, so a big positive input should give large outputs — it does. Sigmoid stays strictly below 1, tanh strictly below 1 (both approach but never touch). Sanity ✔.

Cell B — large negative $z$

Worked example Example 2:

$z = - 6$ through all three Forecast: ReLU on a negative number does what? And sigmoid/tanh head toward which floor?

Step 1 — ReLU. $f (- 6) = max (0, - 6) = 0$ . Why this step? $0$ is bigger than $- 6$ , so the max is $0$ . ReLU "switches the neuron off."

Step 2 — Sigmoid. $σ (- 6) = \frac{1}{1 + e ^{6}} = \frac{1}{1 + 403.43} \approx 0.002473$ . Why this step? Now $- z = + 6$ , so $e^{- z} = e^{6}$ is huge, dominating the denominator; the fraction becomes tiny → near the floor of $0$ .

Step 3 — Tanh. $tanh (- 6) \approx - 0.99999$ . Why this step? Tanh is an odd function — flipping the sign of the input flips the sign of the output: $tanh (- 6) = - tanh (6)$ .

Verify: Big-negative input → smallest outputs. ReLU hit its floor exactly ( $0$ ); sigmoid neared its floor ( $0^{+}$ ); tanh neared its floor ( $- 1^{+}$ ). Note sigmoid $σ (- 6) + σ (6) = 0.002473 + 0.99753 \approx 1$ — a sign of the symmetry $σ (- z) = 1 - σ (z)$ . ✔

Cell C — exactly zero

The most important single point: all three curves pass through the origin region, and this is where their derivatives (steepness) peak.

Cell D — small $z$ near the origin (both signs)

Worked example Example 4:

$z = + 0.5$ and $z = - 0.5$ — the near-linear zone and its symmetry Forecast: near zero the curves are nearly straight lines. Guess whether tanh or sigmoid moves more for the same small input — and whether feeding $- 0.5$ mirrors feeding $+ 0.5$ .

Step 1 — ReLU at $+ 0.5$ . $f (0.5) = max (0, 0.5) = 0.5$ . Why this step? Positive input → identity, so ReLU just copies it through.

Step 2 — Sigmoid at $+ 0.5$ . $σ (0.5) = \frac{1}{1 + e ^{- 0.5}} = \frac{1}{1 + 0.60653} \approx 0.62246$ . Why this step? A small positive input pushes the output a little above the $0.5$ centre.

Step 3 — Tanh at $+ 0.5$ . $tanh (0.5) = \frac{e ^{0.5} - e ^{- 0.5}}{e ^{0.5} + e ^{- 0.5}} = \frac{1.64872 - 0.60653}{1.64872 + 0.60653} \approx 0.46212$ . Why this step? Same input, but tanh climbs from $0$ and is steeper near the origin, so it moves further from its centre.

Step 4 — Now the negative twin, $z = - 0.5$ . ReLU: $f (- 0.5) = max (0, - 0.5) = 0$ . Sigmoid: $σ (- 0.5) = \frac{1}{1 + e ^{0.5}} \approx 0.37754$ . Tanh: $tanh (- 0.5) \approx - 0.46212$ . Why this step? This tests the symmetry each function has around its centre. ReLU is not symmetric — it flatly zeroes the negative side. Sigmoid is symmetric about its centre $0.5$ : notice $0.37754 = 1 - 0.62246$ , i.e. $σ (- z) = 1 - σ (z)$ . Tanh is anti-symmetric about $0$ : $tanh (- 0.5) = - tanh (0.5)$ exactly.

Verify: Distance moved from centre (magnitude), for $+ 0.5$ : sigmoid $0.62246 - 0.5 = 0.122$ ; tanh $0.46212 - 0 = 0.462$ . For $- 0.5$ : sigmoid $0.5 - 0.37754 = 0.122$ (same magnitude, opposite side); tanh $0 - (- 0.46212) = 0.462$ (same magnitude, opposite side). So the near-origin gradient is identical for $\pm 0.5$ — the curves are locally symmetric — and tanh moves ~ $3.8 \times$ further than sigmoid either way. ✔

Cell E — limiting behaviour ( $z \to \pm \infty$ )

Worked example Example 5: What value is approached but never reached?

Forecast: name the four horizontal lines the curves flatten onto.

Step 1 — Sigmoid limits. As $z \to + \infty$ , $e^{- z} \to 0$ , so $σ \to \frac{1}{1 + 0} = 1$ . As $z \to - \infty$ , $e^{- z} \to + \infty$ , so $σ \to \frac{1}{\infty} = 0$ . Why this step? Limits describe the asymptotes — the ceiling and floor a curve creeps toward forever without touching. Sigmoid is trapped strictly inside $(0, 1)$ .

Step 2 — Tanh limits. As $z \to + \infty$ , $tanh \to \frac{e ^{z}}{e ^{z}} = + 1$ ; as $z \to - \infty$ , $tanh \to - 1$ . So tanh lives strictly inside $(- 1, 1)$ . Why this step? At large $+ z$ the $e^{- z}$ terms vanish and top and bottom both reduce to $e^{z}$ , giving $+ 1$ ; the odd symmetry then hands us $- 1$ on the far left. These are tanh's two asymptotes.

Step 3 — ReLU limit. As $z \to + \infty$ , $f \to + \infty$ (no ceiling — that's why ReLU never saturates on the positive side). As $z \to - \infty$ , $f = 0$ (a hard floor). Why this step? On the positive side ReLU is the identity $f (z) = z$ , which grows without bound, so there is no upper asymptote; on the negative side $max (0, z)$ is pinned at exactly $0$ , a genuine floor rather than an approached limit. This asymmetry is precisely why ReLU dodges saturation on the active side.

Verify (numeric proxy). Plug a large $z = 20$ : $σ (20) = 0.99999999794 \approx 1$ but $< 1$ ; $tanh (20) \approx 1$ but $< 1$ . Never equal, always just short. ✔

This is the whole story of saturation: once you are out in the flat tails, moving $z$ barely changes the output, so the derivative $\approx 0$ — see Vanishing and Exploding Gradients.

Cell F — gradients and the vanishing problem

The derivative is the steepness of the curve at a point. In training (see Backpropagation and Gradient Descent) these steepness values get multiplied layer after layer, so we must know how small they can get.

Worked example Example 6: Sigmoid's largest possible gradient, and 10 layers of it

Forecast: the sigmoid derivative is $σ^{'} (z) = σ (z) (1 - σ (z))$ . Where is it biggest, and how big?

Step 1 — Find the peak. The product $p (1 - p)$ (with $p = σ (z) \in (0, 1)$ ) is largest when $p = 0.5$ , i.e. at $z = 0$ . Why this step? A number times "one minus itself" is a downward parabola peaking at $0.5$ ; that's basic and worth memorising.

Step 2 — Value at the peak. $σ^{'} (0) = 0.5 \times (1 - 0.5) = 0.25$ . Why this step? This is the maximum slope sigmoid can ever have — so every sigmoid gradient satisfies $σ^{'} (z) \leq 0.25$ .

Step 3 — Chain 10 layers. Backprop multiplies one factor per layer. Best case: $0.2 5^{10}$ . $0.2 5^{10} = 9.5367 \times 1 0^{- 7} \approx 1 0^{- 6} .$ Why this step? Even in the best case the gradient is a millionth of what it started as — the deep layers barely learn. That is the vanishing gradient.

Step 4 — ReLU contrast. For active neurons $f^{'} (z) = 1$ , so $1^{10} = 1$ — no shrinkage. Compare tanh's peak: $tanh^{'} (0) = 1 - tanh^{2} (0) = 1 - 0 = 1$ , four times steeper than sigmoid's peak.

Verify: $0.2 5^{10} = 9.5367431640625 \times 1 0^{- 7}$ ✔; $tanh^{'} (0) = 1$ ✔; $σ^{'} (0) = 0.25$ ✔. This is precisely why you use ReLU in hidden layers (see the parent's mistake callout).

Cell G — real-world word problem (full neuron)

Worked example Example 7: A pitch-hold neuron with real sensor data

A neuron in an autopilot estimates a "climb tendency" score. It has two inputs — normalised airspeed $x_{1} = 0.7$ and normalised altitude-rate $x_{2} = - 0.4$ — with weights $w_{1} = 1.5$ , $w_{2} = 2.0$ and bias $b = - 0.3$ . It uses tanh because the output is a signed control tendency (climb +, dive −).

Forecast: will the score be positive (climb) or negative (dive)? Guess the sign before computing.

Step 1 — Weighted sum. $z = w_{1} x_{1} + w_{2} x_{2} + b = 1.5 (0.7) + 2.0 (- 0.4) + (- 0.3)$ . Why this step? This is the neuron's core: importance-weighted evidence plus a resting nudge. $z = 1.05 - 0.8 - 0.3 = - 0.05.$

Step 2 — Activation. $tanh (- 0.05) = \frac{e ^{- 0.05} - e ^{0.05}}{e ^{- 0.05} + e ^{0.05}} \approx - 0.049958$ . Why this step? Near zero, tanh is almost linear, so a small negative $z$ gives an almost-equal small negative output.

Step 3 — Interpret. Output $\approx - 0.05$ : a very slight dive tendency, essentially neutral. About $5%$ of full down authority. Why this step? Tanh's $[- 1, 1]$ range maps directly to "full dive … full climb," so magnitude = how hard, sign = which way.

Verify: Units/range check — the output lands inside $(- 1, 1)$ ✔ and its sign matches $z$ 's sign (both negative) because tanh preserves sign ✔. Had we (wrongly) used sigmoid here, we'd get $σ (- 0.05) \approx 0.4875$ — a positive number that cannot express "dive." Right tool chosen. ✔

Cell H — exam twist: dead ReLU rescued by Leaky ReLU

Worked example Example 8: A neuron stuck at

$z = - 3$ for all data A hidden neuron produces $z = - 3$ on every training sample. Show it is dead under ReLU, and rescue it with Leaky ReLU ( $f (z) = max (0.01 z, z)$ ).

Forecast: what is ReLU's output and its gradient here? Why does "gradient = 0" mean the neuron can never recover?

Step 1 — ReLU output. $f (- 3) = max (0, - 3) = 0$ . Why this step? $0$ is bigger than $- 3$ , so the max is $0$ — ReLU clamps every negative pre-activation to the same flat floor, giving no output signal at all.

Step 2 — ReLU gradient. $f^{'} (- 3) = 0$ (flat zero region). Why this step? In training, weight updates are proportional to this gradient. A gradient of exactly $0$ means the update is $0$ — the weights never move, $z$ stays $- 3$ forever. Dead neuron.

Step 3 — Leaky ReLU output. $f (- 3) = max (0.01 \times (- 3), - 3) = max (- 0.03, - 3) = - 0.03$ . Why this step? $- 0.03$ is bigger than $- 3$ , so the leak wins: the neuron emits a tiny signal instead of nothing.

Step 4 — Leaky ReLU gradient. For $z < 0$ the slope is $0.01$ , so $f^{'} (- 3) = 0.01 \neq = 0$ . Why this step? A nonzero gradient means updates still flow — the neuron can crawl back toward positive territory and revive. (See Weight Initialization Strategies for preventing this in the first place.)

Verify: Compare the two at $z = - 3$ : ReLU $(0, 0)$ vs Leaky ReLU $(- 0.03, 0.01)$ . The rescue is exactly the nonzero derivative. And for a positive input, say $z = + 3$ , both agree: $max (0.03, 3) = 3$ — Leaky ReLU changes nothing on the good side. ✔

Recall Self-check: name the cell, then the answer

$max (0, - 6)$ ::: $0$ (Cell B — ReLU on large negative) $σ (0)$ ::: $0.5$ (Cell C — the pivot) $tanh (0)$ ::: $0$ (Cell C — zero-centered) The value sigmoid approaches as $z \to + \infty$ ::: $1$ , but never reaches it (Cell E) The maximum possible value of $σ^{'} (z)$ ::: $0.25$ , at $z = 0$ (Cell F) Leaky ReLU gradient for $z < 0$ ::: $0.01$ (nonzero → neuron stays alive, Cell H) Best-case gradient through 10 sigmoid layers ::: about $1 0^{- 6}$ — the vanishing gradient (Cell F) $σ (- 0.5)$ in terms of $σ (0.5)$ ::: $1 - σ (0.5) \approx 0.37754$ (Cell D — sigmoid symmetry)

Related: Vanishing and Exploding Gradients · Backpropagation and Gradient Descent · Binary Cross-Entropy Loss (sigmoid outputs feed here) · Sensor Fusion in Aerospace (where these neurons read real telemetry).