ReLU and variants (Leaky ReLU, ELU, GELU)

Activation functions inject non-linearity into a network. Without them, stacked linear layers collapse into a single linear map. This note derives the ReLU family from first principles and shows why each variant exists.

Why do we need non-linearity at all?

So an activation must be non-linear, ideally cheap, and must not destroy gradients during backprop.

ReLU — the Rectified Linear Unit

WHY it works so well. Its derivative is: $\frac{d}{dx}\text{ReLU}(x) = \begin{cases} 1 & x>0 \\ 0 & x<0 \end{cases}$

The gradient in the active region is exactly $1$. Compare with sigmoid, whose derivative $\sigma(1-\sigma)$ peaks at $0.25$ and shrinks toward $0$ for large $|x|$. Multiplying many such small numbers through a deep network causes the vanishing gradient problem. ReLU's flat gradient of $1$ keeps signals alive through many layers.

HOW it helps computation. $\max(0,x)$ is a single comparison — no $\exp$, no division. Cheap forward and backward passes.

The Dying ReLU problem

Leaky ReLU — give the negatives a small slope

WHY. By letting a tiny signal $\alpha x$ leak through for $x<0$, the derivative is $\alpha$ (not $0$) on the negative side: $\frac{d}{dx}\text{LReLU}(x)=\begin{cases}1 & x>0\\ \alpha & x<0\end{cases}$ A dead neuron can now recover because a non-zero gradient still flows. (PReLU makes $\alpha$ a learned parameter.)

ELU — Exponential Linear Unit

WHY the exponential. For $x\le 0$, $\alpha(e^x-1)$ smoothly saturates to $-\alpha$ as $x\to-\infty$. Two benefits:

• Smooth: differentiable-ish and its output can go negative, pushing the mean activation toward $0$ (self-normalizing effect → faster training).
• Bounded negatives: unlike Leaky ReLU it doesn't blow up for very negative inputs; it saturates, giving robustness to noise.

Derivative. For $x\le0$, $\frac{d}{dx}\alpha(e^x-1)=\alpha e^x$, which continuously matches slope $1$ at $x=0$ when $\alpha=1$ (unlike ReLU's abrupt corner).

GELU — Gaussian Error Linear Unit

WHY multiply by $\Phi(x)$? Think of a stochastic gate: keep the input $x$ with probability $\Phi(x)$ (how likely a standard Gaussian is below $x$), drop it otherwise. The expected output is $x\cdot\Phi(x)$. So GELU is a smooth, probabilistic version of ReLU's hard $0/1$ gate.

• Small $x$: $\Phi(x)\approx 0.5$, so output $\approx x/2$ — gentle.
• Large positive $x$: $\Phi\to 1$, output $\to x$ (like ReLU).
• Large negative $x$: $\Phi\to 0$, output $\to 0$ (like ReLU) but smoothly, even dipping slightly negative near $0$.

Practical approximation (used in BERT/GPT): $\text{GELU}(x)\approx 0.5x\Big(1+\tanh\big[\sqrt{\tfrac{2}{\pi}}(x+0.044715x^3)\big]\Big)$ because $\Phi$ has no elementary form.

Worked examples

Flashcards

Connections

• Vanishing and Exploding Gradients — the disease ReLU cures and dying-ReLU reintroduces.
• Sigmoid and Tanh Activations — the saturating functions ReLU replaced.
• Backpropagation — activation derivatives drive gradient flow.
• Weight Initialization (He vs Xavier) — He init is designed specifically for ReLU.
• Transformers and Attention — GELU is the default activation in BERT/GPT.
• Universal Approximation Theorem — why piecewise-linear kinks suffice.

Concept Map

Hinglish (regional understanding)