ReLU and variants (Leaky ReLU, ELU, GELU): Worked examples

This is the D3 ("Deep Dive 3 — worked-examples grind") companion to the parent note on ReLU and variants. "D3" is just this vault's label for the third deep-dive child of a topic: the one where we stop deriving theory and instead hit the four functions with every kind of input so you never meet a case you haven't seen solved.

Before we start, one shared reminder of the four functions and their derivatives (with $α$ the small leak/scale constant):

Formula The four activations and their slopes

$ReLU (x) = max (0, x), ReLU^{'} (x) = ⎩ ⎨ ⎧ 10 undefined x > 0 x < 0 x = 0$ $LReLU (x) = {x α x x > 0 x \leq 0, LReLU^{'} (x) = ⎩ ⎨ ⎧ 1 α undefined x > 0 x < 0 x = 0$ $ELU (x) = {x α (e^{x} - 1) x > 0 x \leq 0, ELU^{'} (x) = {1 α e^{x} x > 0 x \leq 0$ $GELU (x) = x Φ (x), GELU^{'} (x) = Φ (x) + x φ (x)$ The "undefined at $x = 0$ " entries for ReLU and Leaky are not a typo: at the join the left slope and right slope disagree, so no single tangent exists there (a corner). We prove this in Example 4, and in practice libraries just pick one side (usually $0$ ). ELU with $α = 1$ and GELU have matching left/right slopes at $0$ , so their derivative there is well-defined. Here == $Φ (x)$ is the standard-normal CDF== ("chance a bell-curve random number lands below $x$ ") and == $φ (x) = \frac{1}{2 π} e^{- x^{2} /2}$ is the standard-normal PDF== (the height of the bell curve at $x$ ). We derive that GELU derivative in Example 6.

The scenario matrix

Every input you can feed an activation falls into one of these cells. The worked examples below are tagged with the cell(s) they cover, and together they fill the whole table.

Cell	Input regime	What it stress-tests	Covered by
A	$x > 0$ (positive)	the "pass-through" branch, all four	Ex 1
B	$x < 0$ small	the negative branch: leak vs saturate vs gate	Ex 2
C	$x \to - \infty$ (limit)	saturation / blow-up behaviour	Ex 3
D	$x \to + \infty$ (limit)	do they all agree with $y = x$ ?	Ex 3
E	$x = 0$ (boundary)	continuity and kink, all four	Ex 4, Ex 6
F	Gradient at a dead point	dying-ReLU vs recovery	Ex 5
G	Degenerate parameter ( $α = 0$ or $α = 1$ )	when a variant collapses into another function	Ex 7
H	Word problem	a neuron in a real forward pass	Ex 8
I	Exam twist	sign trap and $tanh$ -GELU vs exact	Ex 9

Worked examples

Worked example Example 1 — Cell A (positive input, all four agree)

Compute all four activations at $x = 3$ (Leaky $α = 0.01$ , ELU $α = 1$ , per the page convention).

Forecast: guess before reading — do the four answers differ much here?

ReLU: $max (0, 3) = 3$ . Why this step? $x > 0$ selects the $x$ branch.
Leaky ( $α = 0.01$ ): positive branch is also $x$ , so $= 3$ . Why? the leak only changes the negative side; positives are untouched, so $α$ never appears here.
ELU ( $α = 1$ ): positive branch is $x$ , so $= 3$ . Why? for $x > 0$ ELU is literally the identity $y = x$ ; the $α (e^{x} - 1)$ piece only lives on the negative side.
GELU: $3 \cdot Φ (3) = 3 \times 0.99865 = 2.99595$ . Why? $Φ (3) \approx 0.99865$ — the gate is nearly fully open, so GELU is almost identical to $x$ .

Verify: all four sit at $\approx 3$ ; the largest gap is GELU, only $0.004$ below $x$ . This confirms Cell A: on the positive side every activation behaves like the identity line.

Reading the figure: in

, follow the four curves to the right of

x = 0

. They visibly collapse onto the black diagonal

y = x

; at

x = 3

you cannot separate them by eye — exactly the "all four

\approx 3

" you just computed. The red accent (ELU) is the last to peel away, and only on the negative side.

Figure — ReLU and variants (Leaky ReLU, ELU, GELU)

Worked example Example 2 — Cell B (small negative input, the branches split)

Compute all four at $x = - 2$ (Leaky $α = 0.01$ , ELU $α = 1$ , per the page convention).

Forecast: which one dips the most negative? Which stays nearest zero?

ReLU: $max (0, - 2) = 0$ . Why? the negative is clamped to zero — the gate slams shut.
Leaky ( $α = 0.01$ ): $0.01 \times (- 2) = - 0.02$ . Why? a trickle leaks through with slope $0.01$ , so output is tiny but non-zero.
ELU ( $α = 1$ ): $1 \cdot (e^{- 2} - 1) = 0.13534 - 1 = - 0.86466$ . Why? the exponential branch pulls the output toward $- α = - 1$ .
GELU: $- 2 \cdot Φ (- 2) = - 2 \times 0.02275 = - 0.04550$ . Why? $Φ (- 2)$ is small (only $\sim 2.3%$ of the bell curve is below $- 2$ ), so the gate is nearly closed.

Verify: ordering from least to most negative is ReLU $(0)$ , Leaky $(- 0.02)$ , GELU $(- 0.046)$ , ELU $(- 0.865)$ .

Reading the figure: on the left half of Figure s01, drop a vertical line at $x = - 2$ and read off where each curve crosses it. The red ELU curve sits lowest (near $- 0.86$ ), the black ReLU curve hugs the axis at $0$ , and Leaky/GELU sit just barely below the axis — the same top-to-bottom order as your computed numbers. Cell B covered.

Worked example Example 3 — Cell C and Cell D (the two limits)

Find $lim_{x \to - \infty}$ and $lim_{x \to + \infty}$ for each function (Leaky $α = 0.01$ , ELU $α = 1$ ).

Forecast: which function blows up as $x \to - \infty$ , and which saturate?

As $x \to + \infty$ (Cell D): all four $\to + \infty$ along the line $y = x$ . Why? every positive branch is $x$ ; and $Φ (x) \to 1$ so GELU $\to x$ too.
ReLU as $x \to - \infty$ : output is fixed at $0$ . Why? the clamp holds it flat forever.
Leaky as $x \to - \infty$ : $0.01 x \to - \infty$ — it does not saturate, it keeps sinking. Why? the negative branch is still a straight line, just shallow.
ELU as $x \to - \infty$ : $e^{x} \to 0$ , so $α (e^{x} - 1) \to α (0 - 1) = - 1$ . It saturates at $- α = - 1$ . Why? the exponential vanishes, leaving the constant $- α$ .
GELU as $x \to - \infty$ : $x Φ (x) \to 0$ . Why? $Φ (x)$ shrinks toward $0$ far faster than $x$ grows negative (Gaussian tail beats linear), so the product goes to $0$ .

Verify: numerically at $x = - 10$ : ELU $= e^{- 10} - 1 = - 0.99995$ (essentially $- 1$ ✓ saturated); Leaky $= - 0.1$ (still sliding ✓); GELU $= - 10 Φ (- 10) \approx - 10 \times 7.6 \times 1 0^{- 24} \approx - 7.6 \times 1 0^{- 23} \approx 0$ ✓.

Reading the figure: Figure s02 zooms into the far-negative tail from $x = - 10$ to $0$ . Watch the red ELU curve flatten onto the dotted line at $- 1$ (its saturation floor $- α$ ), while the dashed Leaky curve keeps descending in a straight line with no floor — a direct picture of "saturate vs slide." The dotted ReLU line stays pinned at $0$ .

Worked example Example 4 — Cell E (the boundary

$x = 0$ : continuity + the kink, all four) Check the value and left/right slopes at $x = 0$ for all four functions (Leaky $α = 0.01$ , ELU $α = 1$ ). Who has a corner, who is smooth?

Forecast: are all four continuous? Which ones are smooth (matching left and right slopes) at $0$ ?

ReLU value at $0$ : left branch gives $0$ , right branch gives $0$ → continuous. Why? both pieces meet at the origin, no jump.
ReLU slope at $0$ : left slope $= 0$ , right slope $= 1$ → they disagree → a corner (kink), so ReLU $^{'} (0)$ is undefined (matching the definition block). Why? $max (0, x)$ bends sharply; the tangent jumps $0 \to 1$ across the origin, so no single slope exists there.
Leaky value at $0$ : left $= α \cdot 0 = 0$ , right $= 0$ → continuous. Why? the leak line $α x$ also passes through the origin, so no jump.
Leaky slope at $0$ : left slope $= α = 0.01$ , right slope $= 1$ → they disagree → still a kink (undefined at $0$ , just a gentler corner than ReLU). Why? Leaky only softens the negative-side value, not the corner; the two straight pieces still meet at different angles.
ELU value at $0$ ( $α = 1$ ): left $= α (e^{0} - 1) = 1 \cdot (1 - 1) = 0$ ; right $= 0$ → continuous. Why? $e^{0} - 1 = 0$ , so the exponential branch also starts exactly at the origin.
ELU slope at $0$ : left slope $= α e^{0} = 1$ ; right slope $= 1$ → they match → smooth (no corner). Why? choosing $α = 1$ makes the exponential branch leave the origin with slope exactly $1$ , erasing the corner.
GELU value at $0$ : $0 \cdot Φ (0) = 0 \times 0.5 = 0$ → continuous. Why? GELU is a single smooth product $x Φ (x)$ , not a piecewise formula, so it can't have a jump at all.
GELU slope at $0$ : $GELU^{'} (0) = Φ (0) = 0.5$ , identical from both sides → smooth (we derive this general formula in Example 6). Why? one continuous formula ⇒ one well-defined slope; the value $0.5$ is between ReLU's $0$ and $1$ .

Verify: continuity holds for all four (each value $= 0$ at $x = 0$ ✓). Smoothness splits: ReLU (kink, undefined slope) and Leaky (kink, undefined slope) have corners; ELU (slope $1$ both sides) and GELU (slope $0.5$ both sides) are smooth. In Figure s01's origin region: sharp elbows on ReLU/Leaky, glides on ELU/GELU. Cell E covered for all four.

Worked example Example 5 — Cell F (gradient at a dead point: who learns?)

A neuron sits at pre-activation $x = - 5$ across every training batch. Compute each activation's gradient (Leaky $α = 0.01$ , ELU $α = 1$ ). Who can still learn?

Forecast: guess which functions give a zero gradient (no learning) versus a tiny but non-zero one.

ReLU $^{'} (- 5) = 0$ . Why this step? $x < 0$ → slope $0$ . No gradient flows back → the neuron is dead, weights frozen.
Leaky $^{'} (- 5) = α = 0.01$ . Why? the leak slope is constant $α$ regardless of how negative $x$ is → small but alive → can recover.
ELU $^{'} (- 5) = α e^{- 5} = 1 \times 0.006738 = 0.006738$ . Why? derivative of $α (e^{x} - 1)$ is $α e^{x}$ ; here it is tiny (smoothly vanishing) but still non-zero.
GELU $^{'} (- 5) = Φ (- 5) + (- 5) φ (- 5)$ . $Φ (- 5) = 2.867 \times 1 0^{- 7}$ , $φ (- 5) = \frac{1}{2 π} e^{- 12.5} = 1.487 \times 1 0^{- 6}$ . So $GELU^{'} (- 5) = 2.867 \times 1 0^{- 7} - 5 \times 1.487 \times 1 0^{- 6} = - 7.15 \times 1 0^{- 6}$ . Why? GELU's gradient is near zero but slightly negative here — a whisper of signal still leaks.

Verify: ReLU gives exactly $0$ (dead ✓); Leaky $0.01$ , ELU $0.00674$ , GELU $\approx - 7.15 \times 1 0^{- 6}$ — all non-zero, so all three variants keep the neuron trainable. This is the whole reason the variants exist. Cell F covered.

Worked example Example 6 — Cell E + derivative derivation (where does GELU

$^{'}$ come from?) Derive $GELU^{'} (x)$ and evaluate the gradient at $x = 0$ .

Forecast: GELU $(x) = x Φ (x)$ is a product — guess the rule we need.

Use the product rule on $x \cdot Φ (x)$ : $\frac{d}{d x} [x Φ (x)] = Φ (x) \cdot 1 + x \cdot Φ^{'} (x)$ . Why this step? two things multiply and both depend on $x$ , so neither can be treated as a constant — that is exactly when the product rule is the right tool.
The derivative of a CDF is its PDF: $Φ^{'} (x) = φ (x)$ . Why? $Φ (x) = \int_{- \infty}^{x} φ (t) d t$ , and differentiating an integral with respect to its upper limit returns the integrand (Fundamental Theorem of Calculus).
So $GELU^{'} (x) = Φ (x) + x φ (x)$ . Why? substitute step 2 into step 1 — this is the exact GELU gradient (the formula quoted back in Example 4).
At $x = 0$ : $Φ (0) = 0.5$ and $x φ (x) = 0$ , so $GELU^{'} (0) = 0.5$ . Why? at the origin the gate is exactly half-open, giving slope $\frac{1}{2}$ — smoother than ReLU's ambiguous corner.

Verify: $GELU^{'} (0) = Φ (0) + 0 = 0.5$ ✓. Compare: ReLU's slope is undefined at $0$ , GELU's is a clean $0.5$ . This smoothness is why Transformers (BERT/GPT) prefer GELU.

Worked example Example 7 — Cell G (degenerate parameters: variants collapse)

What happens to Leaky ReLU and ELU at the extreme choices $α = 0$ and $α = 1$ ? (This is the one place we deliberately break the page's fixed- $α$ convention, to show why it exists.)

Forecast: guess which choice turns a variant back into plain ReLU, and which turns it into a plain line.

Leaky with $α = 0$ : negative branch $= 0 \cdot x = 0$ → this is exactly ReLU. Why? zero leak = the gate fully closes on negatives.
Leaky with $α = 1$ : negative branch $= 1 \cdot x = x$ → the whole function is $y = x$ , a pure line. Why? if positives and negatives both map to $x$ , there is no kink → no non-linearity left (the parent note's warning: keep $α$ small).
ELU with $α = 0$ : negative branch $= 0 \cdot (e^{x} - 1) = 0$ → again ReLU on the negative side. Why? scaling the exponential piece by $0$ flattens it to the zero clamp.
ELU with $α = 1$ : the smooth default, slope-matched at $0$ (Example 4). Why? $α = 1$ is exactly the value that makes the left slope $α e^{0} = 1$ equal the right slope $1$ .

Verify: at $x = - 3$ : Leaky $_{α = 0} (- 3) = 0 =$ ReLU $(- 3)$ ✓; Leaky $_{α = 1} (- 3) = - 3 = x$ ✓ (pure linear). Cell G covered — this is why $α \approx 0.01$ lives between the two failure modes.

Worked example Example 8 — Cell H (word problem: one forward pass)

A hidden neuron in a backprop-trained net computes pre-activation $x = w^{⊤} a + b, w = [0.5, - 1.0], a = [2.0, 3.0], b = + 0.5.$ It uses ELU ( $α = 1$ ). Find the neuron's output and its local gradient.

Forecast: first compute $x$ — is it positive or negative? That decides everything.

Dot product: $w^{⊤} a = 0.5 \times 2.0 + (- 1.0) \times 3.0 = 1.0 - 3.0 = - 2.0$ . Why this step? a neuron's raw signal is the weighted sum of its inputs before any activation.
Add bias: $x = - 2.0 + 0.5 = - 1.5$ . Why? the bias shifts the pre-activation; here it stays negative.
$x < 0$ so use ELU's exp branch: output $= 1 \cdot (e^{- 1.5} - 1) = 0.22313 - 1 = - 0.77687$ . Why? negative pre-activation → smooth saturating branch.
Local gradient: $ELU^{'} (- 1.5) = α e^{- 1.5} = 0.22313$ . Why? this is the factor multiplied into the upstream gradient during backprop — non-zero, so this neuron learns even though it fired negative.

Verify: output $\approx - 0.777$ , gradient $\approx 0.223$ ; both non-zero and finite. A plain-ReLU neuron here would output $0$ with gradient $0$ (dead). Cell H covered.

Worked example Example 9 — Cell I (exam twist: a sign trap AND exact-vs-

$tanh$ GELU) Part (a) — the sign trap. A student computes $ReLU (- 3)$ and writes $- 3$ "because ReLU passes the input through." Then they compute $ELU (3)$ and write $e^{3} - 1 \approx 19.09$ . Both are wrong. Fix them.

Part (b). Show the two GELU forms agree at $x = 1$ . Exact: $x Φ (x)$ . Approx: $0.5 x (1 + tanh [2/ π (x + 0.044715 x^{3})])$ .

Forecast: in part (a), which branch does each input actually select? In part (b), will the gap really be under $1 0^{- 3}$ ?

(a) ReLU $(- 3)$ : the input is negative, so the correct branch is the clamp: $max (0, - 3) = 0$ , not $- 3$ . Why? ReLU passes positives unchanged but zeroes negatives — the student used the wrong branch by ignoring the sign.
(a) ELU $(3)$ : the input is positive, so the branch is the identity $x = 3$ , not $e^{3} - 1$ . Why? the $α (e^{x} - 1)$ formula only applies for $x \leq 0$ ; using it on a positive input is the classic sign trap.
(b) Exact at $x = 1$ : $Φ (1) = 0.84134$ , so $1 \times 0.84134 = 0.84134$ . Why? $Φ (1)$ is the fraction of the bell curve below $1$ .
(b) Inner term of approx: $2/ π = 0.79788$ ; $x + 0.044715 x^{3} = 1 + 0.044715 = 1.044715$ ; product $= 0.83356$ . Why? the cubic correction $0.044715 x^{3}$ bends the $tanh$ argument so it mimics the Gaussian shape.
(b) $tanh (0.83356) = 0.68236$ ; then $0.5 \times 1 \times (1 + 0.68236) = 0.5 \times 1.68236 = 0.84118$ .
(b) Compare: exact $0.84134$ vs approx $0.84118$ ; difference $\approx 1.6 \times 1 0^{- 4}$ . Why? well under $1 0^{- 3}$ — confirming they are the same function, one just cheaper.

Verify: (a) correct answers are $ReLU (- 3) = 0$ and $ELU (3) = 3$ (sign selects the branch ✓). (b) $∣0.84134 - 0.84118∣ \approx 1.6 \times 1 0^{- 4} < 1 0^{- 3}$ ✓. Cell I covered — read the sign first, and never treat the two GELU forms as different activations.

Recall Did every cell get filled?

Which examples cover the two limits ( $x \to \pm \infty$ )? ::: Example 3 (Cells C and D). Which example shows a variant collapsing back into ReLU or into a pure line? ::: Example 7 (Cell G, $α = 0$ and $α = 1$ ). Which example proves GELU $^{'} (0) = 0.5$ and why we need the product rule? ::: Example 6. Which example is the exam twist (sign trap plus tanh-vs-exact GELU)? ::: Example 9 (Cell I). In Example 8, why does the ELU neuron still learn while a ReLU one would be dead? ::: Its gradient $α e^{- 1.5} = 0.223$ is non-zero, whereas ReLU's negative-side gradient is $0$ . In Example 4, which two of the four are smooth at $x = 0$ and which two have a kink? ::: ELU and GELU are smooth; ReLU and Leaky have a corner (undefined slope).

Connections

Vanishing and Exploding Gradients — Examples 3 and 5 are the saturation/dead-gradient story made numeric.
Backpropagation — Example 8 shows the local gradient that gets chained backward.
Weight Initialization (He vs Xavier) — good init keeps pre-activations off the dead-negative region of Example 5.
Transformers and Attention — Example 6's smooth $0.5$ slope is why GELU rules Transformer blocks.
Sigmoid and Tanh Activations — the $tanh$ of Example 9 reused as a cheap Gaussian mimic.
Universal Approximation Theorem — the many kinks of Examples 1–2 are the pieces that approximate any curve.