ReLU and variants (Leaky ReLU, ELU, GELU): Visual walkthrough

We start from literally nothing: a dot on a number line.

Step 1 — What "activation" even means

WHAT. A neuron first computes one number. It takes its inputs, multiplies each by a weight, adds them up, adds a bias, and gets a single value we call $x$ . That $x$ is called the pre-activation. Then a function $f$ turns $x$ into the neuron's output $y = f (x)$ .

WHY we need $f$ at all. If $f$ just handed back $x$ unchanged ( $y = x$ ), stacking neurons would give you one big straight-line map — no curves, no learning of complicated shapes. (The parent proves this collapse: $W_{2} W_{1} x = W x$ .) So $f$ must bend.

PICTURE. Below, one number $x$ enters on the left axis, a function box bends it, and $y$ comes out. The number line at the bottom is our entire universe for now: every input is just a position, negative on the left, positive on the right.

Figure — ReLU and variants (Leaky ReLU, ELU, GELU)

Step 2 — The hard gate: ReLU

WHAT. The simplest useful bend: if $x$ is positive, keep it; if $x$ is negative, output $0$ . $ReLU (x) = max (floor 0, input x)$ Read $max (0, x)$ as "whichever is larger, $0$ or $x$ ". When $x > 0$ , $x$ is larger, so you get $x$ . When $x \leq 0$ , $0$ is larger, so you get $0$ .

WHY this shape. It is the cheapest possible non-linearity — one comparison, no $exp$ , no division. The kink at $x = 0$ (where the graph changes direction) is the genuine non-linear feature: two straight pieces meeting at a corner is not a straight line.

PICTURE. Look at the graph. To the right of $x = 0$ the red line is exactly the $45°$ diagonal $y = x$ . To the left it lies flat on the floor $y = 0$ . The corner is marked in yellow — that corner is where the "power" lives.

Step 3 — The slope story (why gradients care)

WHAT. During learning, a network passes a correction signal backward. At each activation, that signal gets multiplied by the slope of $f$ at the point $x$ . Slope = "how steep is the graph right here" = the derivative $\frac{df}{d x}$ .

WHY slope, not value. The learning rule (see Backpropagation) chains slopes together. If a slope is $0$ , the correction gets multiplied by $0$ and dies — no update reaches earlier layers. This is the whole drama.

For ReLU the slope is dead simple: $\frac{d}{d x} ReLU (x) = {10 x > 0 (diagonal has steepness 1) x < 0 (flat floor has steepness 0)$

PICTURE. Two flat plateaus: slope $= 1$ on the right (signal passes untouched), slope $= 0$ on the left (signal blocked). Compare the pale sigmoid slope curve underneath — it never even reaches $0.25$ , so deep stacks of it shrink the signal toward nothing (the vanishing gradient disease). ReLU's flat $1$ is the cure.

Recall Why is slope

$= 1$ so precious? Multiplying a backward signal by $1$ many times leaves it unchanged; multiplying by $0.25$ many times shrinks it toward $0$ . ::: ReLU keeps deep signals alive because its active-region slope is exactly $1$ .

Step 4 — The disaster: dying ReLU (the degenerate case)

WHAT. Suppose a neuron's $x$ is always negative (say a big negative bias pushed it there). Then its output is always $0$ and its slope is always $0$ . No output, no backward signal, no update — forever.

WHY it is permanent. To escape the negative region the weights would need to change; but they only change when a gradient flows; but the gradient is $0$ because it is negative. The neuron is trapped. This is the dying ReLU problem, and it is the entire reason the variants exist.

PICTURE. A red dot sits deep in the negative zone. Both arrows above it — output arrow and gradient arrow — read $0$ . The neuron is a dead pipe.

Step 5 — First fix: Leaky ReLU tilts the floor

WHAT. Instead of a flat floor, give the negative side a tiny downward slope $α$ : $LReLU (x) = ⎩ ⎨ ⎧ x \approx 0.01 α x x > 0 x \leq 0$ Here $α$ is a small fixed number. On the negative side the output is $α x$ — a gentle slope instead of dead flat.

WHY it revives neurons. Now the slope on the left is $α$ , not $0$ : $\frac{d}{d x} LReLU (x) = {1 α x > 0 x < 0$ A tiny non-zero gradient still flows, so a stuck neuron can slowly crawl back toward positive territory.

PICTURE. The floor is no longer flat — it tilts slightly below zero (green line). The same negative dot from Step 4 now has a non-zero gradient arrow. Barely alive is still alive.

Step 6 — Smooth fix: ELU rounds the corner with an exponential

WHAT. Replace the negative straight piece with a curve that bends up from $- α$ and flattens: $ELU (x) = {x α (e^{x} - 1) x > 0 x \leq 0$ Read the negative piece: $e^{x}$ is the exponential (grows/shrinks smoothly, always positive); subtract $1$ so that at $x = 0$ we get $α (1 - 1) = 0$ — it meets the positive piece exactly. As $x \to - \infty$ , $e^{x} \to 0$ , so the curve saturates to $α (0 - 1) = - α$ .

WHY an exponential and not a line. Two wins a straight leak can't give:

Bounded negatives. The Leaky line $α x$ keeps diving down forever; $α (e^{x} - 1)$ levels off at $- α$ , so a very noisy negative input can't produce a huge negative output.
Smooth corner. Its slope on the left is $α e^{x}$ , which at $x = 0$ equals $α e^{0} = α$ . With $α = 1$ that is exactly $1$ — the same slope as the right piece, so no abrupt corner (unlike ReLU/Leaky, which change slope suddenly at $0$ ).

PICTURE. The blue ELU curve glides smoothly through the origin and flattens onto the dashed line $y = - α$ far to the left. No kink; a rounded valley.

Recall Why subtract the

$1$ in $e^{x} - 1$ ? So the curve passes through $(0, 0)$ and joins the positive piece with no jump. ::: At $x = 0$ , $α (e^{0} - 1) = α (1 - 1) = 0$ , matching $ReLU$ -style continuity.

Step 7 — Probabilistic fix: GELU, a soft gate

WHAT. Instead of a hard "keep or drop", weight the input $x$ by the probability it should be kept: $GELU (x) = x \cdot keep-probability Φ (x)$ Here $Φ (x)$ is the standard normal CDF: the chance that a random standard Gaussian $Z$ lands at or below $x$ . It rises smoothly from $0$ (far left) through $0.5$ (at $x = 0$ ) to $1$ (far right).

WHY multiply by $Φ$ . ReLU multiplies $x$ by a hard gate — $1$ if $x > 0$ , else $0$ . GELU replaces that step-jump with a smooth S-shaped gate: the bigger $x$ is, the more likely we keep it. The output is the expected value $x \cdot Φ (x)$ of a stochastic on/off gate — a soft ReLU.

Trace the three regimes:

Small $x$ : $Φ \approx 0.5$ , output $\approx x /2$ — gentle.
Large positive $x$ : $Φ \to 1$ , output $\to x$ (like ReLU).
Large negative $x$ : $Φ \to 0$ , output $\to 0$ — but smoothly, dipping slightly negative just left of the origin.

PICTURE. Yellow S-curve = the gate $Φ (x)$ ; red curve = the product $x Φ (x)$ . Notice the tiny negative dip near $x = - 0.7$ where a small $x$ meets a still-nonzero $Φ$ .

Step 8 — All four on one axis (the case sweep)

WHAT. Put every function on the same picture and read off what each does with a negative input — the case that separates them.

WHY. Every neuron's $x$ can be positive, zero, or negative. On the positive side all four agree ( $y = x$ ). All the design choices live on the negative side:

Function	$x < 0$ output	$x < 0$ slope	negative dead?
ReLU	$0$	$0$	yes (dies)
Leaky	$α x$	$α$	no
ELU	$α (e^{x} - 1)$ , saturates to $- α$	$α e^{x}$	no
GELU	$x Φ (x)$ , tiny dip then $\to 0$	smooth	no

PICTURE. Four coloured curves overlaid. Right half: they merge into the one diagonal. Left half: they fan out — flat (ReLU), tilted (Leaky), curved valley (ELU), soft dip (GELU).

The one-picture summary

WHAT. One figure holds the whole story: the shared positive diagonal, the four negative-side treatments, and — underneath — the four slopes at a deeply negative point, showing which neurons can still learn.

The top panel is the family of shapes; the bottom bar chart is the punchline — at $x = - 5$ , ReLU's learning slope is a dead $0$ while Leaky/ELU stay alive. That single gap is why the family exists.

Recall Feynman retelling of the whole walkthrough

A neuron makes one number $x$ and must decide what to send onward. ReLU is a one-way gate: push forward, water flows; push backward, it slams shut — and once shut with no push, it's a dead pipe forever, because the "how much would opening help" signal is also zero. So we tinker with the backward side. Leaky cuts a hairline crack so a trickle always leaks and the pipe stays revivable. ELU replaces the crack with a smooth curved wall that lets a little backward flow but never floods — it levels off. GELU is the smartest gate: it opens more the harder the water pushes, deciding softly instead of all-or-nothing, using "how likely should I keep this?" as its opening amount. On the forward (positive) side, all four behave identically — a plain open channel $y = x$ . Every clever idea is spent on the negative side, and it's all about one thing: never let the backward learning signal become exactly zero.

Connections

Vanishing and Exploding Gradients — the slope-multiplication drama of Step 3.
Sigmoid and Tanh Activations — the saturating slope we out-run with slope $1$ .
Backpropagation — why the slope, not the value, decides who learns.
Weight Initialization (He vs Xavier) — He init keeps early $x$ out of the dead zone.
Universal Approximation Theorem — many ReLU kinks approximate any curve.
Transformers and Attention — where GELU actually ships.