Exercises — ReLU and variants (Leaky ReLU, ELU, GELU)

ReLU zeroes anything $\le 0$ and passes anything $>0$ unchanged.

• $x=-3.2$: negative $\Rightarrow \text{ReLU}=0$.
• $x=0$: the boundary. By the definition $\max (0,x)$, at $x=0$ we get $\max (0,0)=0$. So output $=0$.
• $x=7$: positive $\Rightarrow \text{ReLU}=7$.

• ReLU: no — its smallest value is $0$.
• Leaky ReLU: yes — for $x<0$ it gives $\alpha x$, which is negative (small).
• ELU: yes — for $x<0$, $\alpha (e^x-1)$ is negative, bottoming out at $-\alpha$.
• GELU: yes — near small negative $x$, $x\Phi (x)$ dips slightly below $0$ (a small negative dip), though it returns to $0$ as $x\to -\infty$.

• $x=-4\le 0$: use $\alpha x=0.01\times (-4)=-0.04$.
• $x=5>0$: pass unchanged, $=5$.

For $x\le 0$, $\text{ELU}(x)=\alpha (e^x-1)$.

• $\alpha =1$: $1\cdot (e^{-1}-1)=0.367879-1=-0.632121$.
• $\alpha =2$: $2\cdot (e^{-1}-1)=2\times (-0.632121)=-1.264241$. WHAT this shows: $\alpha$ scales how deep the negative saturation goes — the floor is $-\alpha$.

$\text{GELU}(x)=x\Phi (x)$, so $\text{GELU}(1)=1\times 0.841345=0.841345$. Sanity check: at $x=1$ the gate is $\approx 84\%$ open, so we keep most of the input. Compare ReLU($1$)$=1$ (fully open). GELU is gentler.

The derivative is the local slope — how much a tiny change in $x$ changes the output, and therefore how much gradient flows back in Backpropagation.

• ReLU'$(-5)=0$ (flat on the negative side) → dead, no learning.
• Leaky'$(-5)=\alpha =0.01$ → tiny but alive.
• ELU'$(-5)=\alpha e^x=1\cdot e^{-5}=0.006738$ → alive, smoothly small.
• GELU'$(-5)$: near-zero but nonzero (the curve is smooth everywhere), $\approx -1.49\times 10^{-5}$; not exactly $0$, so not fully dead. Conclusion: only plain ReLU dies here. This is the dead-neuron failure the variants were built to cure.

Right side ($x>0$): both are $y=x$, slope $=1$. Left side derivative of ELU ($x\le 0$): $\frac{d}{dx}\alpha (e^x-1)=\alpha e^x$. At $x=0$: $\alpha e^0=\alpha \cdot 1=1$ (with $\alpha =1$). Left slope $=1=$ right slope → matched, no kink. ReLU left derivative at $x=0$ is $0$, right derivative is $1$: $0\ne 1$ → a corner. The exponential's job was precisely to glue the two slopes together at $\alpha =1$.

• ReLU$(0)=\max (0,0)=0$.
• Leaky$(0)=0$ (boundary, $\alpha \cdot 0=0$).
• ELU$(0)=\alpha (e^0-1)=\alpha (1-1)=0$.
• GELU$(0)=0\cdot \Phi (0)=0\cdot 0.5=0$. All equal $0$, and each piece meets its neighbour there → all four are continuous at $x=0$. (Continuity of value is separate from continuity of slope, which only ELU/GELU have — see L3.2.)

With $\alpha =1$: for $x>0$, output $=x$; for $x\le 0$, output $=1\cdot x=x$. Both pieces are $y=x$, so the function is the identity everywhere. An identity activation means $h_2=W_2\cdot \text{id}(W_{1}x)=W_2{W}_{1}x=Wx$: layers collapse into one matrix $W$. By the parent's [!intuition], you lose all curved decision boundaries — the whole reason activations exist (see Universal Approximation Theorem). Hence $\alpha$ must stay small.

Inner term: $\sqrt{2/\pi }=0.797885$. Cubic bump: $x+0.044715x^3=1+0.044715=1.044715$. Product: $0.797885\times 1.044715=0.833564$. $\tanh (0.833564)=0.682226$. $g(1)=0.5\times 1\times (1+0.682226)=0.5\times 1.682226=0.841113$. Exact was $0.841345$. Difference $=|0.841345-0.841113|=0.000232\approx 2.3\times 10^{-4}$ — under $10^{-3}$, exactly as the parent claimed. The two GELUs are effectively the same function; the tanh form just avoids computing $\Phi$.

Pick ELU.

• Dead-neuron cure: its negative-side derivative is $\alpha e^x>0$ everywhere, so no neuron freezes (ReLU fails this).
• Mean-centering: ELU can output negatives (down to $-\alpha$), pulling the average activation toward $0$; Leaky ReLU also goes negative but its output grows without bound as $x\to -\infty$, whereas ELU saturates to $-\alpha$, giving noise robustness. Leaky ReLU is a fine cheaper alternative if you don't need the smooth saturation; ReLU is ruled out because it dies. This connects to He initialization, which is separately tuned to keep ReLU-family variances stable.

Pre-activation: $x=wa+b=(-1.5)(2)+(-1)=-3-1=-4$. $\text{ReLU}(-4)=0$. Gradient to the weight (chain rule): $\frac{\partial \text{ReLU}}{\partial w}={\text{ReLU}}^{\prime }(x)\cdot \frac{\partial x}{\partial w}={\text{ReLU}}^{\prime }(-4)\cdot a=0\cdot 2=0$. The gradient is $0$, so for this input the neuron cannot update via $w$ — it is (locally) dead. If every training input drives $x<0$, it is permanently dead.

Chain rule again: $\frac{\partial }{\partial w}={\sigma }^{\prime }(x)\cdot a$ with $a=2$, $x=-4$.

• Leaky: ${\sigma }^{\prime }(-4)=\alpha =0.01$, so gradient $=0.01\times 2=0.02\ne 0$. Alive.
• ELU: ${\sigma }^{\prime }(-4)=\alpha e^x=1\cdot e^{-4}=0.018316$, so gradient $=0.018316\times 2=0.036631\ne 0$. Alive. Both deliver a nonzero nudge to $w$, so the neuron can climb back out of the dead region — the entire motivation for the variants.

Compute at $x=-4$:

By output (most negative first): ELU $(-0.98)$ → Leaky $(-0.04)$ → GELU $(-0.00013)$ → ReLU $(0)$. By derivative (largest first): ELU $(0.0183)$ → Leaky $(0.01)$ → GELU $(\approx 0)$ → ReLU $(0)$. Reveal: ELU wins on both here — largest surviving gradient and strongest mean-centering pull. GELU has nearly closed its gate this far into the negatives (near-zero output and gradient), behaving almost like ReLU far from the origin — its gentleness lives near $x=0$, not out at $x=-4$. So "which is best" depends on where your pre-activations sit.

The negative side is where all four differ ::: ReLU zeroes it, Leaky leaks slope $\alpha$, ELU saturates smoothly to $-\alpha$, GELU gates probabilistically. Dead neuron test ::: derivative $=0$ at the operating $x$ means no gradient flows to its weights. tanh-GELU vs exact GELU ::: agree to $\sim 2\times 10^{-4}$; same function, faster to compute.