ReLU and variants (Leaky ReLU, ELU, GELU): Question bank

Before you start, a few words we lean on constantly:

Pre-activation — the raw number $x$ that goes into the activation function (the weighted sum $W x + b$ before it is bent).
Gradient of the activation — the slope of the activation curve at that $x$ ; this slope gets multiplied into the signal flowing backward during Backpropagation.
Saturate — the curve flattens out to a constant, so its slope goes to $0$ and it stops responding to bigger inputs.
$α$ (alpha) — a small positive constant, $α > 0$ . For Leaky ReLU it is the slope of the negative region (typically $α \approx 0.01$ ); for ELU it is the scale of the negative branch (typically $α = 1$ ). It is a fixed hyperparameter, not something the network normally learns (except in PReLU).
$Φ (x)$ (Phi) — the standard normal cumulative distribution function: $Φ (x) = P (Z \leq x)$ where $Z$ is a standard Gaussian (mean $0$ , variance $1$ ). It slides smoothly from $0$ (at $x \to - \infty$ ) through $0.5$ (at $x = 0$ ) to $1$ (at $x \to + \infty$ ).

Below, the shapes drawn so the traps are visual, not memorised. The top row is the activation $y$ ; the bottom row is its slope (gradient) $\frac{d y}{d x}$ .

Figure — ReLU and variants (Leaky ReLU, ELU, GELU)

Notice already: on the right half all four gradient curves ride at $1$ ; all the drama is on the left. Let that guide every answer.

True or false — justify

TRUE/FALSE: A network built only from ReLU layers (no linear layers between) can never be non-linear.

False — ReLU itself is non-linear (the kink at

0

), but you still need the linear layers around it; the point is the combination

W_{2} ReLU (W_{1} x)

carves curves, not ReLU alone.

TRUE/FALSE: ReLU is a linear function.

False — it is piecewise linear. Each of its two pieces is a straight line, but the corner at

0

makes the whole map non-linear, which is exactly what gives the network its power.

TRUE/FALSE: Because ReLU's positive-side gradient is exactly

1

, ReLU completely eliminates the vanishing gradient problem.

False — it only reduces it. The gradient of

1

helps in active regions, but the flat

0

-gradient region reintroduces a different failure (dying ReLU), and exploding gradients from large weights are untouched.

TRUE/FALSE: Leaky ReLU can suffer from dead neurons just like ReLU.

False — its negative-side gradient is

α \neq = 0

, so some signal always flows back and a stuck neuron can recover; the "death" trap needs a genuinely

0

gradient.

TRUE/FALSE: ELU can output negative numbers, so it can push the mean activation toward zero.

True — for

x \leq 0

it gives

α (e^{x} - 1) < 0

, letting outputs sit on both sides of zero, which self-centres the mean and speeds training.

TRUE/FALSE: ELU's negative branch grows without bound as

x \to - \infty

, so very negative inputs are dangerous.

False —

α (e^{x} - 1)

saturates to

- α

x \to - \infty

because

e^{x} \to 0

; it is bounded below, which is one of its noise-robustness selling points.

TRUE/FALSE: GELU is a hard gate that keeps or drops the input with a crisp threshold.

False — it is a smooth stochastic gate; the input is kept with probability

Φ (x)

, a value that slides smoothly from

0

1

, so the expected output

x Φ (x)

has no crisp corner.

TRUE/FALSE: The

tanh

formula for GELU is a genuinely different activation from

x Φ (x)

False — the

tanh

form is a numerical approximation of

x Φ (x)

; the curves agree to about

1 0^{- 3}

, chosen because

Φ

has no elementary closed form.

TRUE/FALSE: Making Leaky ReLU's slope

α

larger always gives a better activation.

False — as

α \to 1

the two pieces merge into

y = x

, a pure line, and you lose all non-linearity;

α

is kept small (

\sim 0.01

) on purpose.

TRUE/FALSE: All four functions (ReLU, Leaky, ELU, GELU) agree exactly for large positive

x

True — for large

x > 0

each returns

x

(ReLU and Leaky exactly, ELU exactly, GELU because

Φ (x) \to 1

), so they only differ meaningfully near and below zero.

Spot the error

SPOT THE ERROR: "ReLU's derivative at

x = 0

1

, so gradients flow fine there."

The derivative at exactly

x = 0

is undefined (a corner, not a slope); frameworks pick a convention like

0

. More importantly, the danger is the whole

x < 0

region where the slope is genuinely

0

SPOT THE ERROR: "A dead ReLU neuron will fix itself as soon as the input distribution shifts."

It cannot self-fix — with pre-activation stuck negative, its gradient is

0

, so no weight update ever happens; nothing pushes it back into the active region. Only a change from elsewhere (or a different activation) rescues it.

SPOT THE ERROR: "ELU with

α = 1

has a sharp corner at

x = 0

just like ReLU."

No — at

x = 0

the negative branch's slope is

α e^{0} = α = 1

, matching the positive branch's slope of

1

, so the curve joins smoothly with no corner (this is a key ELU advantage).

SPOT THE ERROR: "GELU always outputs a non-negative number, since it's a smoothed ReLU."

False — for small negative

x

Φ (x)

is small but positive and

x < 0

, so

x Φ (x) < 0

; GELU dips slightly negative near the origin, unlike ReLU which floors at

0

SPOT THE ERROR: "Leaky ReLU is smooth everywhere because it lets negatives through."

It still has a corner at

x = 0

where the slope jumps from

1

α

. "Leaky" fixes the dead-gradient problem, not the smoothness problem — that's what ELU and GELU address.

SPOT THE ERROR: "Because sigmoid saturates and ELU saturates, ELU has the same vanishing-gradient weakness."

The difference is where: sigmoid saturates on both ends (killing gradients for all large

∣ x ∣

), while ELU stays slope-

1

for all

x > 0

, so its positive region never vanishes — see Sigmoid and Tanh Activations.

Why questions

WHY does replacing sigmoid with ReLU help deep networks train?

Sigmoid's derivative peaks at

0.25

and shrinks, so many multiplied through depth shrink toward

0

; ReLU's active-region derivative is exactly

1

, so the signal is not attenuated layer by layer.

WHY does the exponential specifically appear in ELU, rather than any other decaying function?

e^{x}

makes the negative branch

α (e^{x} - 1)

saturate smoothly to

- α

and have slope

α e^{x}

that equals

1

x = 0

when

α = 1

— giving both boundedness and a seamless join, which a plain linear leak cannot.

WHY is He initialization the natural partner for ReLU rather than Xavier?

ReLU zeroes half its inputs on average, halving the variance passed forward; He init scales weights to compensate for that lost variance, whereas Xavier assumes a symmetric activation — see Weight Initialization (He vs Xavier).

WHY is GELU the default in Transformers instead of plain ReLU?

Its smooth probabilistic gate gives a gentler, differentiable response near zero and a slight negative dip, which empirically trains large language models more stably than ReLU's hard corner.

WHY does a kink (non-differentiable point) not break gradient descent for ReLU?

Gradient descent only needs a subgradient — pick any slope between the two sides at the corner (frameworks use

0

); the corner is a single point of measure zero, so it rarely lands there exactly and training proceeds.

WHY does making a network deeper and using ReLU still rely on the Universal Approximation Theorem?

The theorem guarantees that enough ReLU kinks, combined by linear layers, can approximate any continuous function; ReLU's role is to supply those non-linear kinks so the guarantee applies.

Edge cases

EDGE CASE: What does each activation output at exactly

x = 0

All four give

0

: ReLU

max (0, 0) = 0

, Leaky

α \cdot 0 = 0

, ELU

α (e^{0} - 1) = 0

, GELU

0 \cdot Φ (0) = 0

. They all pass through the origin.

EDGE CASE: What is the gradient of each at

x = 0^{-}

(approaching zero from the negative side)?

ReLU

\to 0

, Leaky

\to α

, ELU

\to α e^{0} = α

, GELU

\to

a smooth value near

0.5

; only ReLU has a vanishing slope right at the boundary.

EDGE CASE: As

x \to + \infty

, what does each gradient approach?

All four

\to 1

: ReLU and Leaky are exactly

1

on the positive branch, ELU is exactly

1

for

x > 0

, and GELU's slope

\to 1

because

Φ (x) \to 1

and its extra term fades. This is why they all behave identically for large positive inputs.

EDGE CASE: As

x \to - \infty

, what value does each activation approach?

ReLU

\to 0

, Leaky

\to - \infty

(slope

α

forever), ELU

\to - α

(bounded), GELU

\to 0

. Leaky is the only one that grows unboundedly negative.

EDGE CASE: As

x \to - \infty

, what does each gradient approach?

ReLU

\to 0

(dead), Leaky

\to α

(constant, alive), ELU

\to 0

(via

α e^{x}

, alive but fading), GELU

\to 0

. Only Leaky keeps a fixed non-zero gradient at the far negative end.

EDGE CASE: If you set Leaky ReLU's

α = 0

, what have you built?

Exactly plain ReLU — the negative branch

α x

becomes

0

, so Leaky ReLU is a strict generalization of ReLU with

α = 0

as the special case.

EDGE CASE: If you set ELU's

α = 0

, what happens to the negative branch?

It collapses to

0

for all

x \leq 0

, recovering ReLU's behaviour but without the smooth slope match — so

α = 0

throws away exactly the property (boundedness, negative outputs) that ELU was introduced for.

Recall One-line summary of the traps

The recurring theme ::: the difference between the four activations lives entirely in the negative region and at the corner — positive $x$ is identical, so every trap is about what happens when the pre-activation is $\leq 0$ .

Connections

Parent topic — ReLU and variants
Vanishing and Exploding Gradients — the disease behind most of these traps.
Backpropagation — why the gradient of each activation is what actually matters.
Weight Initialization (He vs Xavier) — pairs with ReLU's half-zeroing behaviour.
Sigmoid and Tanh Activations — the two-sided saturation contrast.
Transformers and Attention — where GELU became standard.
Universal Approximation Theorem — why piecewise-linear kinks suffice.