Activation functions - sigmoid, tanh: Question bank

Quick symbol refresher so nothing here is unexplained:

$x$ = the input vector to the neuron — a list of numbers (features), e.g. $x = [x_{1}, x_{2}, \dots]$ , one number per incoming signal.
$W$ = the weight matrix (for a single neuron, a weight vector $w$ ): one number per input, controlling how strongly that input pushes the neuron.
$b$ = the bias, a single number added to shift the sum up or down regardless of the inputs.
$z$ = the pre-activation, the raw weighted sum $W x + b$ a neuron computes before squashing — for one neuron this is $w \cdot x + b = w_{1} x_{1} + w_{2} x_{2} + \dots + b$ . Any real number.
$σ (z) = \frac{1}{1 + e ^{- z}}$ = sigmoid, squashes $z$ into $(0, 1)$ .
$a$ = the activation = the output of the squasher for this neuron, i.e. $a = σ (z)$ . It is the number stored during the forward pass and reused later.
$tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$ = hyperbolic tangent, squashes $z$ into $(- 1, 1)$ .
$σ^{'} (z)$ = derivative = the local slope of the curve = how much the output moves per tiny nudge in $z$ . This slope is what Backpropagation multiplies by.
Saturation = the flat tails of the S-curve, where $z$ is very large positive or negative and the slope is nearly zero.

True or false — justify

TF1. "Sigmoid can output exactly 0 or exactly 1 for large enough input."

False —

e^{- z}

is always strictly positive, so

σ (z)

approaches but never reaches

0

1

; the range is the open interval

(0, 1)

TF2. "A network of 50 linear layers with no activation is more powerful than a single linear layer."

False — composing linear maps gives another linear map, so all 50 layers collapse into one

(W_{2} W_{1} \dots) x + const

; the activation's nonlinearity is what breaks the collapse.

TF3. "Sigmoid outputs across several neurons in a layer form a valid probability distribution."

False — each sigmoid is independent and they do not sum to 1; for a distribution over mutually-exclusive classes you need Softmax.

TF4. "Tanh is just a sigmoid with a different label; the shapes are unrelated."

False — write

σ (2 z) = \frac{1}{1 + e ^{- 2 z}} = \frac{e ^{z}}{e ^{z} + e ^{- z}}

, then

2 σ (2 z) - 1 = \frac{2 e ^{z} - ( e ^{z} + e ^{- z} )}{e ^{z} + e ^{- z}} = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}} = tanh (z)

, so tanh is a sigmoid stretched horizontally by 2 and scaled/shifted to span

(- 1, 1)

TF5. "The maximum slope of tanh is exactly four times the maximum slope of sigmoid."

True — differentiate

tanh (z) = 2 σ (2 z) - 1

by the chain rule:

tanh^{'} (z) = 2 \cdot σ^{'} (2 z) \cdot 2 = 4 σ (2 z) (1 - σ (2 z))

; at

z = 0

this is

4 \cdot 0.5 \cdot 0.5 = 1

versus

σ^{'} (0) = 0.25

, so the factor of 2 inside the argument produces the

4 \times

slope.

TF6. "Both sigmoid and tanh saturate, so both suffer from vanishing gradients in deep networks."

True — both have flat tails where the derivative

\to 0

, so chaining many layers multiplies many near-zero slopes and the gradient vanishes.

TF7. "Because tanh is zero-centered, it never causes vanishing gradients."

False — zero-centering fixes the sign-of-gradient zig-zag issue, not saturation; tanh still flattens in its tails and can vanish.

TF8. "Sigmoid's derivative requires recomputing

e^{- z}

during backprop."

False — the self-referential form

σ^{'} (z) = σ (z) (1 - σ (z)) = a (1 - a)

reuses the already-computed forward activation

a = σ (z)

, so no exponential is recomputed.

TF9. "Logistic regression is essentially a single sigmoid neuron."

True — Logistic Regression computes

σ (w \cdot x + b)

and reads it as a probability, which is exactly one sigmoid unit.

TF10. "The average output of a random-input sigmoid layer is near zero."

False — sigmoid outputs live in

(0, 1)

so their average is near

0.5

(positive); tanh, centered on 0, is the one with near-zero average.

Spot the error

SE1. "Since

σ^{'} (z) = σ (z) (1 - σ (z))

and

σ

ranges over

(0, 1)

, the derivative can be as large as 1."

Writing the sigmoid value as

σ

, the product

σ (1 - σ)

is maximized at

σ = 0.5

, giving

0.25

, not 1 — the derivative never exceeds

0.25

anywhere.

SE2. "To get a probability that a sample belongs to one of 3 classes, apply sigmoid to each of the 3 outputs and you're done."

Three independent sigmoids don't normalize to sum 1; for mutually-exclusive classes you must use Softmax so the outputs form a proper distribution.

SE3. "Tanh maps

R

(0, 1)

just like sigmoid."

Tanh maps to

(- 1, 1)

; the negative range is precisely its zero-centered advantage over sigmoid's

(0, 1)

SE4. "We should use sigmoid in hidden layers because its probability interpretation helps learning."

The probability reading matters only at a binary output; in hidden layers sigmoid's all-positive, saturating behavior hurts training, which is why tanh or ReLU is preferred there.

SE5. "

tanh^{'} (z) = 1 - tanh^{2} (z)

, so at large

z

the slope grows because

tanh^{2} \to 1

At large

z

tanh^{2} \to 1

makes

1 - tanh^{2} \to 0

, so the slope shrinks (saturates), not grows.

SE6. "Making the network deeper always increases what sigmoid/tanh nets can learn."

Extra depth with saturating activations amplifies vanishing gradients, so deeper can actually train worse — a historical reason ReLU took over.

SE7. "All-positive sigmoid activations are fine because the weights can still be positive or negative."

The problem is that with all-positive inputs to a neuron, every gradient component of its weight vector shares the same sign, forcing inefficient zig-zag updates — independent of the weights' signs.

Why questions

WHY1. "Why can't we just use a step function (0 or 1) instead of the smooth sigmoid?"

A step has zero derivative almost everywhere and an undefined jump at 0, so backprop gets no usable slope; the smooth sigmoid provides gradients everywhere so the network can be nudged.

WHY2. "Why is the sigmoid's derivative largest at

z = 0

z = 0

is the steepest point of the S-curve (output

= 0.5

, the middle of

(0, 1)

); away from it the curve flattens toward its horizontal asymptotes, so slope falls off.

WHY3. "Why does zero-centering (tanh) speed up hidden-layer training versus sigmoid?"

Zero-centered activations let a downstream neuron receive both positive and negative inputs, so its weight-gradient signs can differ, avoiding the shared-sign zig-zag that slows sigmoid-fed layers.

WHY4. "Why does the vanishing gradient get worse the deeper the network, not better?"

Backprop multiplies the per-layer slopes together; each saturated sigmoid contributes a factor

\leq 0.25

, and multiplying many small factors drives the product exponentially toward 0.

WHY5. "Why is careful Weight Initialization tied to sigmoid/tanh?"

If initial weights make pre-activations

z

large, neurons start in the flat saturated tails where slopes are tiny, so good initialization keeps

z

near 0 where gradients are healthy.

WHY6. "Why do we write

tanh (z) = 2 σ (2 z) - 1

instead of treating them as separate functions?"

It shows tanh is a rescaled-shifted sigmoid, explaining why they share S-shape and saturation while differing only in centering and steepness — one insight instead of two memorized curves.

WHY7. "Why is a sigmoid output called a 'soft' version of firing?"

Instead of a hard yes/no it returns a continuous confidence in

(0, 1)

, so small changes in input produce small graded changes in output — 'soft' meaning smoothly graded rather than binary.

Edge cases

EC1. "What is

σ (0)

and

tanh (0)

, and why do these anchor points matter?"

σ (0) = 0.5

and

tanh (0) = 0

; these are the centers where each curve is steepest, so a fresh neuron with

z \approx 0

learns fastest.

EC2. "What happens to the gradient when

z = 6

for sigmoid?"

σ (6) \approx 0.9975

, so

σ^{'} (6) = 0.9975 \times 0.0025 \approx 0.0025

— a nearly dead slope, the vanishing-gradient tail in action.

EC3. "As

z \to - \infty

, what do

σ (z)

and

σ^{'} (z)

approach?"

σ (z) \to 0

and

σ^{'} (z) = σ (1 - σ) \to 0 \cdot 1 = 0

; the neuron is 'off' and its gradient dies, so it barely learns.

EC4. "As

z \to + \infty

, what do

tanh (z)

and

tanh^{'} (z)

approach?"

tanh (z) \to 1

and

tanh^{'} (z) = 1 - tanh^{2} \to 1 - 1 = 0

; fully saturated 'on' with a vanishing slope.

EC5. "If two neurons both saturate — one at

z = + 8

, one at

z = - 8

— do their gradients differ in health?"

No; both tails are flat, so both have near-zero slope and both learn poorly regardless of which extreme they sit at.

EC6. "A sigmoid output neuron gives

0.5

. What does that mean and is it a problem?"

It means maximum uncertainty (50/50), which is a perfectly valid — and gradient-healthy — state, not an error; it's exactly the steepest, most learnable point.

EC7. "What is the derivative of tanh at its center, and how does it compare to sigmoid's center slope?"

tanh^{'} (0) = 1

versus

σ^{'} (0) = 0.25

, so tanh is four times steeper at the origin, giving stronger gradients through the un-saturated region.

Connections

ReLU and Leaky ReLU — the non-saturating alternative that dodges most traps above.
Softmax — the correct tool when you catch yourself misusing sigmoid for multi-class.
Backpropagation — where every derivative fact here is actually consumed.
Vanishing and Exploding Gradients — the theme behind the edge-case tail questions.
Logistic Regression — the single-sigmoid special case.
Weight Initialization — the practical antidote to starting in the tails.