Activation functions - sigmoid, tanh: Worked examples

Before we start, three symbols we will lean on, each in plain words:

Figure — Activation functions - sigmoid, tanh

Look at the figure. The orange curve (sigmoid) never dips below $0$ or above $1$ . The teal curve (tanh) is the same shape stretched to live between $- 1$ and $+ 1$ . The flat tails on both far left and far right are where trouble hides — we will land there in cell E.

The scenario matrix

Every case sigmoid/tanh can throw at you falls into one of these cells. Each later example is tagged with its cell letter.

Cell	Case class	What makes it special	Example
A	$z > 0$ (positive input)	output above the midpoint	Ex 1
B	$z < 0$ (negative input)	output below the midpoint; symmetry test	Ex 2
C	$z = 0$ (exact zero / degenerate)	the crossing point; max slope	Ex 3
D	Backprop use of the slope	derivative multiplied into a chain	Ex 4
E	$	z	$ large (saturated tail / limit)
F	Sigmoid vs tanh on the SAME $z$	zero-centering, 4× slope	Ex 6
G	Real-world word problem	reading output as a probability	Ex 7
H	Exam twist (inverse / solve for $z$ )	going backwards through $σ$	Ex 8

We will hit all eight.

Example 1 — Cell A: positive pre-activation

Example 2 — Cell B: negative pre-activation & the symmetry law

Recall Why does

$σ (- z) = 1 - σ (z)$ ? Multiply top and bottom of $σ (- z) = \frac{1}{1 + e ^{z}}$ by $e^{- z}$ : you get $\frac{e ^{- z}}{e ^{- z} + 1} = 1 - \frac{1}{1 + e ^{- z}} = 1 - σ (z)$ . One-line proof ::: $\frac{1}{1 + e ^{z}} = \frac{e ^{- z}}{1 + e ^{- z}} = 1 - \frac{1}{1 + e ^{- z}} = 1 - σ (z)$ .

Example 3 — Cell C: the degenerate input $z = 0$

Worked example Exactly zero: the crossing point

A dead neuron produces $z = 0$ (e.g. all-zero input with zero bias). Find $σ (0)$ , $tanh (0)$ , and both slopes there.

Forecast: zero input is "no evidence either way", so guess the outputs sit dead-center of each range.

Sigmoid value. $σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{1 + 1} = 0.5$ . Why this step? $e^{0} = 1$ exactly; this is the midpoint, "50% confidence".
Tanh value. $tanh (0) = \frac{e ^{0} - e ^{0}}{e ^{0} + e ^{0}} = \frac{0}{2} = 0$ . Why this step? Numerator is $1 - 1 = 0$ ; tanh's center is $0$ , which is why we call it zero-centered.
Slopes at zero. $σ^{'} (0) = σ (0) (1 - σ (0)) = 0.5 \times 0.5 = 0.25$ ; $tanh^{'} (0) = 1 - tanh^{2} (0) = 1 - 0 = 1.0$ . Why this step? Both curves are steepest at their center — this is the fastest-learning region.

Verify: $0.5, 0, 0.25, 1.0$ . Sanity: tanh's slope $1.0$ is $4 \times$ the sigmoid's $0.25$ — matches the mnemonic "four-times-as-steep".

Example 4 — Cell D: the slope inside backpropagation

Worked example Using

$σ^{'}$ in a chain A sigmoid neuron output $a = σ (z) = 0.9526$ (from Ex 1). The gradient arriving from the layer above is $\frac{\partial L}{\partial a} = 1.5$ . Find $\frac{\partial L}{\partial z}$ .

Forecast: $z = 3$ is already fairly far right, so the local slope should be smallish — expect the passed-down gradient to shrink noticeably.

Local slope, self-referential form. $σ^{'} (z) = a (1 - a) = 0.9526 \times 0.0474 = 0.04516$ . Why this step? We already computed $a$ ; the identity $σ^{'} = a (1 - a)$ means no exponentials needed again — this is why it is loved in Backpropagation.
Chain rule. $\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot σ^{'} (z) = 1.5 \times 0.04516 = 0.06774$ . Why this step? The chain rule multiplies the upstream gradient by the local slope; a small slope throttles the signal.

Verify: $0.06774$ . Sanity: the incoming $1.5$ got shrunk to $\approx 0.068$ — a $\times 0.045$ attenuation, exactly the "already leaning on, hard to change" behavior forecast.

Example 5 — Cell E: the saturated tail & vanishing gradient (limit)

Worked example Large

$∣ z ∣$ : where learning stalls Compare the local sigmoid slope at $z = 6$ vs $z = 0$ , then chain four such $z = 6$ layers.

Forecast: $z = 6$ is deep in the flat right tail (figure), so the slope should be nearly $0$ ; chaining four of them should be catastrophic.

Value at $z = 6$ . $σ (6) = \frac{1}{1 + e ^{- 6}} = \frac{1}{1 + 0.002479} = 0.99753$ . Why this step? We need $σ$ to get its slope via $σ (1 - σ)$ .
Slope at $z = 6$ . $σ^{'} (6) = 0.99753 \times (1 - 0.99753) = 0.99753 \times 0.00247 = 0.002461$ . Why this step? One factor is near $1$ , the other near $0$ , so the product is tiny — the curve is flat here.
Chain four layers. $(0.002461)^{4} = 3.67 \times 1 0^{- 11}$ . Why this step? Backprop multiplies these local slopes; four saturated layers multiply four tiny numbers, and the gradient collapses to essentially zero.

Verify: $0.99753, 0.002461, \approx 3.67 \times 1 0^{- 11}$ . Compare to $z = 0$ where the slope was $0.25$ — a $\approx 100 \times$ drop per layer. This is the vanishing gradient problem and the reason ReLU and Leaky ReLU exists. See Vanishing and Exploding Gradients and Weight Initialization (which keeps $z$ off these tails).

Example 6 — Cell F: sigmoid vs tanh on the same input

Worked example Same

$z$ , two squashers, and the link formula For $z = 0.8$ , compute $σ (0.8)$ and $tanh (0.8)$ , then confirm $tanh (z) = 2 σ (2 z) - 1$ .

Forecast: tanh, being two-sided and steeper, should give a value further from $0$ (in its own scale) than sigmoid is from $0.5$ .

Sigmoid. $σ (0.8) = \frac{1}{1 + e ^{- 0.8}} = \frac{1}{1 + 0.44933} = 0.68997$ . Why this step? Baseline value to compare.
Tanh directly. $tanh (0.8) = \frac{e ^{0.8} - e ^{- 0.8}}{e ^{0.8} + e ^{- 0.8}} = \frac{2.22554 - 0.44933}{2.22554 + 0.44933} = \frac{1.77621}{2.67487} = 0.66404$ . Why this step? Independent value from tanh's own definition.
Confirm the link. $2 σ (2 \cdot 0.8) - 1 = 2 σ (1.6) - 1 = 2 (0.83201) - 1 = 1.66403 - 1 = 0.66403$ . Why this step? This proves tanh is a rescaled, shifted sigmoid — same shape, recentered on $0$ , doubled in slope.

Verify: $tanh (0.8) = 0.66404$ vs the link value $0.66403$ (agree to rounding) ✓. Note tanh's output $0.664$ is a strong "yes" while sigmoid's $0.690$ , measured from its center $0.5$ , is only $+ 0.19$ — tanh spreads decisions further, which is why hidden layers prefer it (see the Softmax and Logistic Regression output-layer notes for where sigmoid still wins).

Example 7 — Cell G: real-world word problem

Worked example Spam filter confidence

An email scorer outputs a single pre-activation $z = - 1.2$ (mildly "not spam" evidence). The team ships an email as spam only if $σ (z) \geq 0.7$ . Do we flag this email?

Forecast: $z$ is negative, so $σ < 0.5$ ; almost certainly below the $0.7$ threshold — do not flag.

Convert score to probability. $σ (- 1.2) = \frac{1}{1 + e ^{1.2}} = \frac{1}{1 + 3.32012} = 0.23148$ . Why this step? Sigmoid turns the raw score into a probability in $(0, 1)$ — the only form a threshold rule can compare against.
Apply the decision rule. Is $0.23148 \geq 0.7$ ? No. Why this step? The business threshold operates on the probability, not the raw $z$ .

Verify: $0.23148 < 0.7$ ⇒ do not flag ✓. Units check: a probability, dimensionless, in $(0, 1)$ — correct type for a threshold. This is exactly the single-sigmoid decision of Logistic Regression.

Example 8 — Cell H: exam twist (invert the sigmoid)

Worked example Solve backwards: given output, find

$z$ A neuron should output $σ (z) = 0.8$ . What pre-activation $z$ achieves this? (This is the logit / inverse-sigmoid question.)

Forecast: $0.8 > 0.5$ , so we expect a positive $z$ , somewhere around $1$ – $2$ .

Set up the inverse. From $0.8 = \frac{1}{1 + e ^{- z}}$ , invert: $1 + e^{- z} = \frac{1}{0.8} = 1.25$ , so $e^{- z} = 0.25$ . Why this step? To "undo" sigmoid we rearrange for the exponential, isolating $z$ — the logistic function is one-to-one, so a unique $z$ exists for every output in $(0, 1)$ .
Take the logarithm. $- z = ln (0.25) = - 1.38629$ , so $z = 1.38629$ . Why this step? $ln$ is the tool that undoes $e^{(\cdot)}$ ; it is the only operation that frees $z$ from an exponent.
General formula (the logit). $z = ln \frac{p}{1 - p} = ln \frac{0.8}{0.2} = ln 4 = 1.38629$ . Why this step? $ln \frac{p}{1 - p}$ is the standard logit, the exact inverse of sigmoid — worth memorizing for exams.

Verify: plug back: $σ (1.38629) = \frac{1}{1 + e ^{- 1.38629}} = \frac{1}{1 + 0.25} = 0.8$ ✓, and $z = 1.386$ is positive as forecast.

Coverage check

Recall Did we hit every cell of the matrix?

A (Ex 1, $z > 0$ ), B (Ex 2, $z < 0$ + symmetry), C (Ex 3, $z = 0$ degenerate + max slope), D (Ex 4, backprop chain), E (Ex 5, saturated limit / vanishing gradient), F (Ex 6, sigmoid vs tanh + link), G (Ex 7, word problem), H (Ex 8, inverse/logit). All eight ✓.

Connections

Backpropagation — Example 4 is one node of it.
Vanishing and Exploding Gradients — Example 5 is the sigmoid case.
ReLU and Leaky ReLU — what people switched to after Example 5.
Softmax — the normalized multi-class cousin (contrast Example 7's independent sigmoid).
Logistic Regression — Examples 7 and 8 are exactly its forward and inverse maps.
Weight Initialization — keeps $z$ in the steep region (Cell C), away from Cell E.