Foundations — ReLU and variants (Leaky ReLU, ELU, GELU)

Before you can read the parent note on ReLU and variants, you need to earn every symbol it throws at you. This page builds each one from nothing — plain words first, then a picture, then why the topic needs it. Read top to bottom; each block leans on the one above.

0. What even is a "function"?

The picture is a graph: put the input $x$ on the horizontal axis, the output $f(x)$ on the vertical axis. Every input gets exactly one dot. Join the dots and you see the machine's "shape".

Look at the left panel: a straight line — this is a linear machine, output is just input scaled. The right panel has a kink — that bend is what "non-linear" means, and it is the whole reason ReLU exists.

Why the topic needs this: every activation ($\text{ReLU}$, $\text{ELU}$, $\text{GELU}$) is a function. If you can read a graph, you can read them all.

1. $x$ — the pre-activation

Picture a single number sliding along a horizontal line (a number line). To the right of the middle mark it is positive; to the left it is negative; the middle mark is zero.

Why the topic needs it: every activation's behaviour is described case by case — "for $x>0$ do this, for $x\le 0$ do that". You cannot follow those cases until $x$-the-number-line is second nature.

2. The sign of $x$: positive, negative, zero

The symbol $\le$ means "less than or equal to" — it includes the boundary $x=0$. The parent note writes $x \le 0$ precisely to decide which side owns the zero point. This is not fussiness: the whole "dying neuron" story lives right at that boundary.

3. $\max (a,b)$ — pick the bigger one

Picture two heights; $\max$ hands you the taller one. That's the entire operation — one comparison, no arithmetic.

Why the topic needs it: $\text{ReLU}(x)=\max (0,x)$ is defined by this. When $x$ is positive it beats $0$, so you get $x$; when $x$ is negative $0$ wins, so you get $0$. The kink in that right-hand graph of Figure 1 is the $\max$ switching winners at $x=0$.

4. Piecewise functions and the "case bracket"

Read aloud: "if $x>0$, output $x$; otherwise output $0$." One machine, two rules glued at $x=0$.

Why the topic needs it: every function in the parent note is written this way. Master the bracket once and Leaky ReLU / ELU are free.

5. Slope — how steep is the line?

The chalk-blue segment climbs at slope $1$ (up as much as it goes across). The pink segment is flat, slope $0$. The pale-yellow segment leaks downhill gently, slope $\alpha$ (a small number like $0.01$).

Why the topic needs it: the "gradient" that trains a network is nothing but the slope of the activation. A flat piece (slope $0$) means no learning signal passes — that is the dying-neuron catastrophe. Everything the variants do is "stop the slope from being exactly zero."

6. Derivative $\frac{d}{dx}f(x)$ — the slope, as its own machine

Why this tool and not another? Training a network asks one question over and over: "if I nudge this input a hair, how much does the output move?" That sensitivity is exactly slope-at-a-point, which is exactly the derivative. No other tool answers "how sensitive?" as directly — that is why the parent note reports a derivative for every activation.

For ReLU the derivative is itself piecewise:

Slope $1$ on the climbing piece, slope $0$ on the flat piece — read straight off Figure 2.

7. $e$ and $e^x$ — the exponential machine

Look at the curve: to the right it rockets upward; to the left ($x\to -\infty$) it flattens toward $0$ but never touches it. That gentle flattening on the left is the whole point of ELU.

Why the topic needs it: ELU uses $\alpha (e^x-1)$ for negative inputs. As $x$ marches to $-\infty$, $e^x\to 0$, so $\alpha (e^x-1)\to \alpha (0-1)=-\alpha$. The output saturates — settles to a floor of $-\alpha$ instead of falling forever. That bounded, smooth negative side is exactly what the exponential buys you.

Also: $e^0=1$, so $\alpha (e^0-1)=\alpha (1-1)=0$. This is why ELU joins smoothly to $0$ at the boundary (the parent's continuity check).

8. $\alpha$ (alpha) — the small tuning knob

Picture the pale-yellow leaky segment in Figure 2: $\alpha$ is literally how steeply it tilts below zero. Bigger $\alpha$ = steeper leak; $\alpha =0$ recovers plain ReLU; $\alpha =1$ makes the line perfectly straight (and kills the non-linearity — see the parent's mistake box).

Why the topic needs it: $\alpha$ is the single dial that turns ReLU into Leaky ReLU or ELU. Knowing it is "just a small slope number" demystifies all the formulas.

9. Probability and $\Phi (x)$ — the Gaussian gate

The bell is the Gaussian. $\Phi (x)$ is the shaded fraction of area to the left of $x$. Far left → almost no area → $\Phi \approx 0$. At the centre $x=0$ → exactly half the bell → $\Phi =0.5$. Far right → nearly all the area → $\Phi \approx 1$.

Why the topic needs it: $\text{GELU}(x)=x\Phi (x)$ multiplies the input by this probability. It's a soft gate: keep the input with probability $\Phi (x)$. Large positive $x$ → gate ~fully open ($\Phi \to 1$, output $\to x$, like ReLU). Large negative $x$ → gate ~shut ($\Phi \to 0$, output $\to 0$). But it opens gradually instead of snapping — that smoothness is GELU's edge, used in Transformers and Attention.

10. $\tanh$ — the S-shaped squasher

Why the topic needs it: the practical GELU formula uses $\tanh$ because $\Phi$ has no elementary formula (you can't write it with $+,-,\times ,÷,$ powers). $\tanh$ is a cheap S-curve that mimics the shape of $\Phi$ closely enough. It also appears in Sigmoid and Tanh Activations, the older activations ReLU replaced.

How these feed the topic

Non-linearity is the reason for all this — it links to the Universal Approximation Theorem (many kinks approximate any curve), while the derivative story links to Backpropagation, Vanishing and Exploding Gradients, and Weight Initialization (He vs Xavier).

Equipment checklist

Test yourself — you're ready for the parent note when each of these is obvious: