Visual walkthrough — Activation functions - sigmoid, tanh

We will build, in order:

1. The number line (the raw input).
2. The exponential curve $e^{-z}$ (the one ingredient).
3. The squash into $(0,1)$ — the sigmoid.
4. Its slope (the derivative), by watching the curve.
5. The self-referential shortcut $\sigma (1-\sigma )$.
6. Tanh as the same picture, recentered.
7. The degenerate/edge cases (the flat tails).

---

Step 1 — The raw signal lives on the whole number line

WHAT. A neuron first computes one plain number, the pre-activation $z$. We call it $z$; it can be any real number: $-1000$, $0$, $+7.3$, anything.

WHY. Before we squash anything, we must be honest about the input. The input is unbounded — it stretches to $-\infty$ on the left and $+\infty$ on the right. Our whole job is to tame this unbounded line into a bounded answer.

PICTURE. Look at the horizontal arrow: it never ends in either direction. The green dot at $z=0$ is "no opinion". Right of it, evidence for firing; left of it, evidence against.

---

Step 2 — One ingredient: the exponential $e^{-z}$

WHAT. We introduce exactly one new function, $e^{-z}$. Read it as "$e$ raised to minus $z$", where $e\approx 2.718$ is a fixed number.

WHY THIS TOOL and not another? We need a gadget that is:

• always positive (so we never divide by a negative surprise),
• huge when $z$ is very negative, and tiny when $z$ is very positive,
• smooth (no corners), so the network can be nudged.

$e^{-z}$ is the simplest function doing all three. When $z\to -\infty$, $-z\to +\infty$, so $e^{-z}\to +\infty$. When $z\to +\infty$, $e^{-z}\to 0$. At $z=0$, $e^0=1$. That "flips from big to small, passing through 1" is exactly the switch we want.

PICTURE. The coral curve dives from a towering height on the left down toward the floor $0$ on the right, kissing the value $1$ right at $z=0$ (mint dot).

Term-by-term, the expression $e^{-z}$:

• $e$ ::: the fixed base $\approx 2.718$, chosen because its own slope equals itself — i.e. $\frac{d}{dz}{e}^z=e^z$. This is the one calculus fact we will lean on.
• $-z$ ::: the "minus" flips the input, so large $z$ makes the exponent very negative, driving $e^{-z}$ toward $0$.

---

Step 3 — Squash it: build the sigmoid $\sigma (z)$

WHAT. Define the sigmoid: $\sigma (z)=\underset{\text{one over (one plus the ingredient)}}{\underbrace{\frac{1}{1+e^{-z}}}}$

WHY this exact recipe? We want a number that lives strictly between $0$ and $1$. Look at the denominator $1+e^{-z}$:

• It is always bigger than $1$ (since $e^{-z}>0$), so $\sigma <1$. ✔ never reaches 1.
• It is always finite, so $\sigma >0$. ✔ never reaches 0.

Reading each piece where it sits:

• the $e^{-z}$ ::: the "how much evidence against" term. When it is huge (very negative $z$), the whole fraction $\to 0$.
• the $+1$ ::: the guardrail that keeps the denominator from ever hitting $0$, so we never blow up.
• the outer $\frac{1}{\cdots }$ ::: the flip that turns "big denominator" into "small output".

PICTURE. The lavender S-curve: floor at $0$ on the far left, ceiling at $1$ on the far right, passing through exactly $0.5$ at $z=0$ (the green dot — "maximum uncertainty"). This is the tamed version of Step 1's endless line.

The mirror symmetry $\sigma (-z)=1-\sigma (z)$

WHY prove it? It says the S-curve is point-symmetric about the center dot $(0,0.5)$ — flipping $z$'s sign reflects the output through $0.5$. We will lean on this to understand the tails.

Derivation, step by step. First, plug $-z$ into the definition $\sigma (z)=\frac{1}{1+e^{-z}}$ — every place we see $z$, write $-z$: $\sigma (-z)=\frac{1}{1+e^{-(-z)}}=\frac{1}{1+e^z}$ Now multiply top and bottom by $e^{-z}$ (a legal move, since $\frac{e^{-z}}{e^{-z}}=1$): $\sigma (-z)=\frac{1}{1+e^z}\cdot \underset{\text{=1, legal}}{\underbrace{\frac{e^{-z}}{e^{-z}}}}=\frac{e^{-z}}{e^{-z}+1}$ Now compare with $1-\sigma (z)=1-\frac{1}{1+e^{-z}}=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}$. Same thing. ✔ $\boxed{\sigma (-z)=1-\sigma (z)}$

PICTURE. The S-curve with the point $(z,\sigma (z))$ and its mirror $(-z,1-\sigma (z))$ marked — both sit symmetrically across the center dot.

---

Step 4 — The slope: what does "steepness" mean, and why do we want it?

WHAT. The derivative of $\sigma$, written ${\sigma }^{\prime }(z)$, is the slope of the S-curve at each point — how much the output rises for a tiny nudge in $z$. The little tick mark ${}^{\prime }$ ("prime") is shorthand for "differentiate with respect to $z$"; that is, ${\sigma }^{\prime }(z)$ means $\frac{d\sigma }{dz}$, the same object written two ways.

WHY do we need it AT ALL? Backpropagation learns by asking "if I wiggle $z$ a hair, how much does the output move?" That sensitivity is precisely the slope. Flat spot → wiggling $z$ does nothing → no learning there.

PICTURE. Three little tangent segments drawn on the S-curve. In the far-left tail the segment is nearly flat. Near $z=0$ it is steep. In the far-right tail it flattens again. The slope is biggest in the middle.

The two calculus rules we use, in pictures.

• Chain rule — if a value depends on $z$ through an inner box, its slope is (slope of the outer wrapping) × (slope of the inner box). Picture a nudge $\Delta z$ (a tiny change in $z$) entering the inner box, coming out scaled, then scaled again by the outer wrapping: the two scalings multiply. Here the outer wrapping is "take the reciprocal / raise to $-1$", the inner box is $1+e^{-z}$.
• Power rule — for $u^{-1}$, a tiny change $\Delta u$ in $u$ changes $u^{-1}$ by $-u^{-2}\Delta u$; picture the reciprocal curve $1/u$ sloping downward (bigger $u$ → smaller $1/u$), so its slope is negative, and steeper when $u$ is small.
• Slope of $e^{-z}$ — by the fact in Step 2 ($\frac{d}{dz}{e}^z=e^z$) plus the chain rule on the inner $-z$: $\frac{d}{dz}{e}^{-z}=e^{-z}\cdot (-1)=-e^{-z}$.

Now compute. Write $\sigma =(1+e^{-z}{)}^{-1}$ and apply the chain rule (outer $\times$ inner): ${\sigma }^{\prime }(z)=\underset{\text{outer: power rule on }{u}^{-1}}{\underbrace{-(1+e^{-z}{)}^{-2}}}\cdot \underset{\text{inner box}}{\underbrace{\frac{d}{dz}(1+e^{-z})}}=-(1+e^{-z}{)}^{-2}\cdot \underset{\text{slope of }{e}^{-z}}{\underbrace{(-e^{-z})}}$ The two minus signs cancel: ${\sigma }^{\prime }(z)=\frac{e^{-z}}{(1+e^{-z}{)}^2}$ Reading it: the numerator $e^{-z}$ is our ingredient again; the squared denominator makes the slope shrink fast whenever $e^{-z}$ is far from balanced.

---

Step 5 — The magic shortcut: ${\sigma }^{\prime }=\sigma (1-\sigma )$

WHAT. We rewrite the slope using only $\sigma$ itself — no exponentials.

WHY bother? During Backpropagation we have already computed $\sigma (z)$ in the forward pass. If the slope can be built from that stored number, we save the whole expensive $e^{-z}$ recomputation. That is why this identity is beloved.

The trick, one line at a time. Split the numerator by writing $e^{-z}=(1+e^{-z})-1$: ${\sigma }^{\prime }(z)=\frac{(1+e^{-z})-1}{(1+e^{-z}{)}^2}=\underset{=\frac{1}{1+e^{-z}}=\sigma }{\underbrace{\frac{1+e^{-z}}{(1+e^{-z}{)}^2}}}-\underset{={\sigma }^2}{\underbrace{\frac{1}{(1+e^{-z}{)}^2}}}=\sigma -{\sigma }^2$ $\boxed{{\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))}$

Term-by-term of the final form:

• $\sigma (z)$ ::: how "on" the neuron is (its output height).
• $(1-\sigma (z))$ ::: how much room is left before it saturates to 1 (and, by Step 3's mirror, this equals $\sigma (-z)$).
• their product ::: the slope is largest when both factors are balanced — i.e. at $\sigma =0.5$.

PICTURE. The lavender S-curve and, beneath it, the coral bump $\sigma (1-\sigma )$. The bump peaks exactly under the steepest part of the S, at $z=0$, where its height is $0.5\times 0.5=0.25$. Away from center, at least one factor is near $0$, so the product collapses.

---

Step 6 — Same picture, recentered: tanh

WHAT. Tanh is $\tanh (z)=2\sigma (2z)-1$ — literally the sigmoid, stretched and shifted.

WHY this reframing? Sigmoid's whole output is positive (it lives in $(0,1)$), which biases every gradient of a weight vector to share one sign and causes zig-zag updates. Recentering to $(-1,1)$ fixes that. Reading the transform:

• $\sigma (2z)$ ::: double the input first → makes the S twice as steep.
• $2(\cdots )$ ::: stretch the height from $(0,1)$ to $(0,2)$.
• $-1$ ::: slide down so the range becomes $(-1,1)$, centered on $0$.

Check the recentering is real. Recall $\sigma (2z)=\frac{1}{1+e^{-2z}}$. Multiply top and bottom by $e^z$: $\sigma (2z)=\frac{e^z}{e^z+e^{-z}}$. Then $2\sigma (2z)-1=\frac{2e^z}{e^z+e^{-z}}-\frac{e^z+e^{-z}}{e^z+e^{-z}}=\frac{e^z-e^{-z}}{e^z+e^{-z}}=\tanh (z).✔$

Now derive its slope, from first principles. We differentiate $\tanh (z)=2\sigma (2z)-1$ with the same chain rule as Step 4 (outer $\times$ inner, and $\frac{d}{dz}(2z)=2$): ${\tanh }^{\prime }(z)=2\cdot {\sigma }^{\prime }(2z)\cdot 2=4\sigma (2z)(1-\sigma (2z))$ Substitute $\sigma (2z)=\frac{\tanh (z)+1}{2}$ (just rearranging $\tanh =2\sigma (2z)-1$): ${\tanh }^{\prime }(z)=4\cdot \frac{1+\tanh (z)}{2}\cdot \frac{1-\tanh (z)}{2}=(1+\tanh (z))(1-\tanh (z))=1-{\tanh }^2(z)$ Reading the factors: $(1+\tanh )$ is "room above the floor $-1$", $(1-\tanh )$ is "room below the ceiling $+1$" — their product is the slope, biggest when tanh is at $0$. $\boxed{{\tanh }^{\prime }(z)=1-{\tanh }^2(z)}$

PICTURE. The lavender sigmoid and the mint tanh on the same axes. Tanh passes through the origin (not $0.5$), and near $z=0$ it climbs four times faster — its peak slope is ${\tanh }^{\prime }(0)=1-0^2=1.0$.

---

Step 7 — The edge cases: the flat tails (where learning dies)

WHAT. We check the extremes: $z\to +\infty$, $z\to -\infty$, and the exact center.

WHY. The reader must never meet a scenario we skipped. The tails are not a curiosity — they are the vanishing-gradient problem.

Case by case:

• $z\to +\infty$: $e^{-z}\to 0$, so $\sigma \to 1$. Then ${\sigma }^{\prime }=\sigma (1-\sigma )\to 1\cdot 0=0$. Flat.
• $z\to -\infty$: $e^{-z}\to \infty$, so $\sigma \to 0$. Then ${\sigma }^{\prime }\to 0\cdot 1=0$. Flat.
• $z=0$ (center): $\sigma =0.5$, slope $=0.25$ — the one and only peak.
• Concrete tail number ($z=6$): $\sigma (6)=0.99753$, so ${\sigma }^{\prime }(6)=0.99753\times 0.00247\approx 0.00246$ — nearly dead.

PICTURE. The slope bump again, but now with the two tails shaded: both ends hug the floor. An arrow marks the lonely peak of $0.25$ at the center. Chain many such tiny slopes in a deep net and the product races to $0$.

---

The one-picture summary

Everything at once: the endless input line (Step 1) → the exponential ingredient → the S-shaped squash into $(0,1)$ → its slope bump peaking at $0.25$ in the middle and dying in both tails → tanh as the recentered twin.

Recall Feynman retelling — the whole walk in plain words

Start with a number that can be anything, huge or tiny — it's the weighted sum $w\cdot x+b$ a neuron computes. I build one helper, $e^{-z}$, which is enormous when the number is very negative and almost nothing when it's very positive. I drop it into $\frac{1}{1+e^{-z}}$: the $+1$ stops any blow-up, the flip turns "lots of evidence against" into a value near $0$ and "lots for" into a value near $1$. That's the S-curve — a soft yes/no dial that reads $0.5$ when it can't decide, and it's mirror-symmetric: flipping the input reflects the output through $0.5$. Then I ask, "how touchy is this dial?" — that's the slope. Doing the calculus (multiply the outer scaling by the inner scaling — the chain rule), an astonishing thing happens: the slope equals the dial's height times how much room it has left, $\sigma (1-\sigma )$. So it's touchiest right in the middle (slope $0.25$) and goes numb at both ends. Tanh is the same dial, doubled and slid down so it swings from $-1$ to $+1$ through zero, and it's four times as touchy at the center. The catch, seen in the very last picture: when a neuron is pushed to an extreme — far out in either tail — wiggling it barely changes the output, so its slope is nearly zero. Stack many such numb neurons and Backpropagation multiplies all those tiny slopes together, driving the gradient toward zero. That vanishing gradient is exactly why very deep networks built from sigmoid or tanh stop learning — and why we start weights small (to sit near the steep center) and often switch to non-saturating activations.

---

Connections

• Parent · Sigmoid & Tanh — the results this page draws from scratch.
• Backpropagation — where ${\sigma }^{\prime }=\sigma (1-\sigma )$ pays off (reuse the forward value).
• Vanishing and Exploding Gradients — the flat tails of Step 7, at scale.
• ReLU and Leaky ReLU — the non-saturating fix.
• Weight Initialization — keeps $z$ near the steep center.
• Logistic Regression — a single sigmoid neuron in the wild.
• Softmax — the normalized multi-class cousin.