Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Question bank

Before we start, one word to unpack so nothing below is mysterious: the pre-activation $z$ is just the plain number you get from $z = \sum_{i} w_{i} x_{i} + b$ before any activation function touches it. The activation $y = f (z)$ is what comes out after. Everything below hinges on keeping those two straight.

True or false — justify

Stacking 100 linear layers with no activation can learn a curved decision boundary.

False — the whole stack collapses to a single

W_{eff} x + b_{eff}

, which only draws straight lines/planes. The number of layers changes nothing without a non-linearity between them.

A neuron with all weights zero and bias zero is useless forever.

True at that instant (output is

f (0)

for every input), but training can move the weights via gradients — unless the activation kills the gradient too, which is why initialization matters (Weight Initialization Strategies).

ReLU is a non-linear function even though each of its two pieces is a straight line.

True — the "kink" at

z = 0

is the non-linearity. A function is linear only if it's a single straight line through the origin scaled everywhere; the bend at zero breaks that.

Sigmoid always outputs a valid probability, so it's a safe choice for any output layer.

False — its range

(0, 1)

suits binary probability, but it can't represent signed quantities (control deflection) or multi-class distributions, and in hidden layers it causes vanishing gradients.

Tanh is just sigmoid shifted and stretched.

True — exactly

tanh (z) = 2 σ (2 z) - 1

. It has the same S-shape but is re-centered to span

(- 1, 1)

instead of

(0, 1)

The bias term is optional because you can always rescale the inputs instead.

False — rescaling inputs changes slope, not the vertical shift. Without

b

the activation is forced to pass through a fixed point, so the neuron can't move its threshold.

ReLU never suffers from vanishing gradients.

Mostly true on its positive side (derivative is exactly 1), but on the negative side the gradient is 0, causing the dead neuron problem instead — a different failure, not vanishing.

A larger weight always means that input is more important.

False in general — importance depends on weight magnitude relative to input scale. A big weight on an input that's always tiny may matter less than a small weight on a large input; this is why input normalization matters.

Spot the error

"We use ReLU in the output layer so the network gives a probability."

Error — ReLU outputs

[0, \infty)

, not

[0, 1]

. Probabilities need sigmoid (binary) or softmax (multi-class); ReLU belongs in hidden layers.

"Sigmoid's derivative can be as large as 1, so gradients survive deep networks."

Error —

σ^{'} (z) = σ (z) (1 - σ (z))

peaks at

z = 0

with value

0.25

, not 1. Multiplying many factors

\leq 0.25

shrinks gradients fast (Vanishing and Exploding Gradients).

"tanh is zero-centered, so it can never saturate."

Error — zero-centering and saturation are unrelated. tanh still flattens to

\pm 1

for large

∣ z ∣

, where its derivative

1 - tanh^{2} (z)

goes to 0.

"A dead ReLU neuron will recover on its own after a few epochs."

Error — if

z < 0

for all data, the gradient through that neuron is 0, so its weights never update. It stays dead unless you use Leaky ReLU or re-initialize.

"Non-linearity comes from the weighted sum, so the more weights, the more non-linear the neuron."

Error — the weighted sum

\sum w_{i} x_{i} + b

is perfectly linear no matter how many terms. Non-linearity comes only from the activation function

f

"Since tanh gives negative values, its gradient can be negative, which stops learning."

Error — the output can be negative, but its derivative

1 - tanh^{2} (z)

is always in

(0, 1]

, i.e. non-negative. Learning direction is set by the loss, not by the sign of the activation output.

"We should always use the same activation for every layer for consistency."

Error — best practice mixes them: ReLU (or variants) in hidden layers for healthy gradients, and sigmoid/tanh/softmax at the output chosen by the task (probability vs. signed vs. class).

Why questions

Why does inserting

f (z)

between two linear layers rescue expressiveness?

Because

f

breaks the algebraic collapse —

W_{2} f (W_{1} x + b_{1}) + b_{2}

can no longer be folded into a single

A x + b

, so the composition can bend and fold input space into curves.

Why is ReLU the default for hidden layers despite being non-differentiable at

z = 0

The single kink at exactly

z = 0

almost never occurs on real data, and its constant slope-1 gradient elsewhere avoids the vanishing problem — cheapness plus healthy gradients outweigh the one undefined point.

Why does sigmoid cause learning to stall in deep networks?

In backprop, per-layer derivatives multiply. Since each

σ^{'} (z) \leq 0.25

, ten layers give a factor

\leq 0.2 5^{10} \approx 1 0^{- 6}

, so the earliest layers get near-zero updates (Backpropagation and Gradient Descent).

Why prefer tanh over sigmoid for hidden layers when you must use an S-shape?

tanh is zero-centered, so activations aren't all-positive; this keeps gradient directions balanced and typically speeds convergence. Its slope near zero (up to 1) is also steeper than sigmoid's (up to 0.25).

Why does the bias act like a neuron's "resting potential"?

It sets the output when all inputs are zero:

y = f (b)

. This lets a neuron fire (or stay silent) without any input signal, shifting the activation threshold left or right.

Why is careful weight initialization tied to whether a network even starts learning?

Poor scaling can push every

z

deep into a saturated region (sigmoid/tanh) or below zero (ReLU), zeroing gradients from step one. He-initialization keeps

z

in the sensitive range (Weight Initialization Strategies).

Edge cases

What does ReLU output at exactly

z = 0

Zero, by the convention

max (0, z)

with the boundary assigned to the "off" branch. The subderivative there is ambiguous, but frameworks pick

0

(or

1

) and it rarely matters in practice.

What is

σ (z)

z \to + \infty

and

z \to - \infty

It approaches

1

and

0

respectively but reaches neither — the range is the open interval

(0, 1)

, so a sigmoid output is never exactly 0 or 1.

What is

tanh (0)

and why does that make tanh "zero-centered"?

tanh (0) = 0

, so a zero pre-activation gives a zero output. Symmetry about the origin means positive and negative inputs map symmetrically, keeping the mean activation near zero.

If every input

x_{i} = 0

, what does the neuron output?

y = f (b)

— purely the bias passed through the activation. This is exactly why bias exists: without it the answer would be forced to

f (0)

for all-zero inputs.

For a saturated neuron (

∣ z ∣

very large) using sigmoid, what happens to its gradient contribution in backprop?

It's nearly zero, so upstream weights receive almost no update through this path — the neuron is effectively frozen even though it's still producing an output.

Can a Leaky ReLU neuron become permanently dead like a plain ReLU?

No — for

z < 0

it still passes a small slope (e.g.

0.01

), so a nonzero gradient always flows and the weights can keep updating back toward the active region.

Recall One-line summary to hold onto

Linearity lives in the weighted sum; all expressive power and all gradient trouble live in the activation function — pick $f$ by what the layer needs (gradient health for hidden, output meaning for the last).