ReLU and variants (Leaky ReLU, ELU, GELU) — Hinglish notes

3.1.5 · AI-ML › Neural Network Fundamentals

Activation functions network mein non-linearity inject karte hain. Inke bina, stacked linear layers ek single linear map mein collapse ho jaate hain. Yeh note ReLU family ko first principles se derive karta hai aur dikhata hai ki kyun har variant exist karta hai.

Non-linearity ki zaroorat kyun hai?

Isliye ek activation non-linear hona chahiye, ideally cheap ho, aur backprop ke dauran gradients ko destroy na kare.

ReLU — Rectified Linear Unit

YEH ITNA ACCHA KYUN KAAM KARTA HAI. Iska derivative yeh hai: $\frac{d}{d x} ReLU (x) = {10 x > 0 x < 0$

Active region mein gradient exactly $1$ hai. Sigmoid se compare karo, jiska derivative $σ (1 - σ)$ peak $0.25$ par karta hai aur bade $∣ x ∣$ ke liye $0$ ki taraf shrink hota hai. Aise chhote numbers ko ek deep network mein bahut baar multiply karne se vanishing gradient problem hoti hai. ReLU ka flat gradient of $1$ signals ko kai layers mein alive rakhta hai.

COMPUTATION MEIN HELP KAISE KARTA HAI. $max (0, x)$ ek single comparison hai — koi $exp$ nahi, koi division nahi. Forward aur backward passes saste hain.

Dying ReLU problem

Leaky ReLU — negatives ko ek chhoti slope do

KYUN. $x < 0$ ke liye ek tiny signal $α x$ leak hone dene se, negative side par derivative $α$ (na ki $0$ ) hai: $\frac{d}{d x} LReLU (x) = {1 α x > 0 x < 0$ Ek dead neuron ab recover kar sakta hai kyunki ek non-zero gradient ab bhi flow karta hai. (PReLU mein $α$ ek learned parameter hota hai.)

ELU — Exponential Linear Unit

EXPONENTIAL KYUN. $x \leq 0$ ke liye, $α (e^{x} - 1)$ smoothly $- α$ par saturate hota hai jab $x \to - \infty$ . Do fayde:

Smooth: differentiable-ish aur iska output negative ja sakta hai, mean activation ko $0$ ki taraf push karta hai (self-normalizing effect → faster training).
Bounded negatives: Leaky ReLU ke unlike, yeh bahut negative inputs ke liye blow up nahi karta; saturate karta hai, noise ke against robustness deta hai.

Derivative. $x \leq 0$ ke liye, $\frac{d}{d x} α (e^{x} - 1) = α e^{x}$ , jo $x = 0$ par continuously slope $1$ se match karta hai jab $α = 1$ (ReLU ke abrupt corner ke unlike).

GELU — Gaussian Error Linear Unit

$Φ (x)$ SE MULTIPLY KYUN? Ek stochastic gate socho: input $x$ ko probability $Φ (x)$ se rakhte hain (kitna likely hai ki ek standard Gaussian $x$ se neeche hai), warna drop kar do. Expected output $x \cdot Φ (x)$ hai. Isliye GELU, ReLU ke hard $0/1$ gate ka ek smooth, probabilistic version hai.

Chhota $x$ : $Φ (x) \approx 0.5$ , isliye output $\approx x /2$ — gentle.
Bada positive $x$ : $Φ \to 1$ , output $\to x$ (ReLU ki tarah).
Bada negative $x$ : $Φ \to 0$ , output $\to 0$ (ReLU ki tarah) lekin smoothly, $0$ ke paas thoda sa negative bhi dip karta hai.

Practical approximation (BERT/GPT mein use hoti hai): $GELU (x) \approx 0.5 x (1 + tanh [\frac{2}{π} (x + 0.044715 x^{3})])$ kyunki $Φ$ ka koi elementary form nahi hai.

Worked examples

Flashcards

Network sirf linear layers kyun use nahi kar sakta?

Yeh ek linear map

W_{2} W_{1} x = W x

mein collapse ho jaate hain; koi extra expressive power nahi milti, isliye non-linear activations zaroori hain.

ReLU aur uska derivative define karo.

ReLU (x) = max (0, x)

; derivative

x > 0

ke liye

1

hai,

x < 0

ke liye

0

hai.

ReLU sigmoid se vanishing gradients kyun better avoid karta hai?

Uska active-region gradient exactly

1

hai, jabki sigmoid ka derivative max

0.25

par karta hai aur shrink hota hai, isliye depth mein products decay karte hain.

Dying ReLU problem kya hai?

Negative pre-activation mein stuck neurons

0

output karte hain aur

0

gradient hota hai, isliye yeh kabhi update nahi hote — permanently dead.

Leaky ReLU dying neurons ko kaise fix karta hai?

Yeh

x < 0

ke liye slope

α \approx 0.01

use karta hai, non-zero gradient

α

deta hai taaki dead neurons recover kar sakein.

ELU likho aur batao ki exponential kyun use karta hai.

ELU (x) = x

x > 0

ke liye,

α (e^{x} - 1)

x \leq 0

ke liye; yeh smoothly

- α

par saturate karta hai, negative mean-centering allow karta hai, aur bahut negative inputs ke liye bounded hai.

GELU define karo aur uska probabilistic interpretation do.

GELU (x) = x Φ (x)

; input ko probability

Φ (x)

se rakho (chance ki ek standard Gaussian

x

se neeche hai), ek smooth stochastic gate.

Bade

\pm x

ke liye GELU ka value behaviour?

Bada positive

\to x

(ReLU ki tarah), bada negative

\to 0

, lekin origin ke paas smooth, thoda sa negative dip ke saath.

Leaky ReLU ka

α

chhota kyun rakhen?

Agar

α \to 1

toh function linear ho jaata hai (

y = x

), non-linearity destroy ho jaati hai.

x = - 2

α = 1

par ELU compute karo.

e^{- 2} - 1 \approx - 0.865

Recall Feynman: 12-saal ke bachche ko samjhao

Ek paani ka gate socho. ReLU ek one-way gate hai: agar paani aage push kare (positive) toh freely flow karta hai; agar peeche push kare (negative) toh gate band ho jaata hai aur kuch bhi nahi hota — woh pipe wahan "dead" hamesha ke liye. Leaky ReLU ek tiny crack chhodta hai taaki ek trickle hamesha nikle, pipe ko alive rakhte hue. ELU ek smooth curved gate hai jo thoda paani peeche flow karne deta hai lekin kabhi flood nahi hota. GELU ek smart gate hai jo zyada khulta hai jitna zyada paani push karta hai, narm tarike se decide karta hai all-or-nothing ke bajaye. Yeh sab network ko seedhi lines ki jagah bendy shapes seekhne dete hain.

Connections

Vanishing and Exploding Gradients — woh bimari jo ReLU theek karta hai aur dying-ReLU wapas laata hai.
Sigmoid and Tanh Activations — woh saturating functions jo ReLU ne replace kiye.
Backpropagation — activation derivatives gradient flow drive karte hain.
Weight Initialization (He vs Xavier) — He init specifically ReLU ke liye design ki gayi hai.
Transformers and Attention — GELU, BERT/GPT mein default activation hai.
Universal Approximation Theorem — kyun piecewise-linear kinks kaafi hain.