Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

5.6.6 · Coding › Machine Learning (Aerospace Applications)

The Artificial Neuron

WHY har component?

Weights $w_{i}$ : Seekhna matlab inhe adjust karna. High weight = "yeh input bahut matter karta hai."
Bias $b$ : Iske bina, jab sab inputs zero hote hain toh output bhi zero hota hai. Bias neuron ko bina kisi input ke bhi activate hone deta hai—ise neuron ka "resting potential" samjho.
Activation $f$ : Linear combination ko non-linear output mein convert karta hai. Yeh crucial hai—kai linear functions milke sirf ek aur linear function bante hain, jo XOR ya koi complex pattern nahi seekh sakta.

Activation Functions

1. ReLU (Rectified Linear Unit)

WHY ReLU?

Computationally cheap: Sirf ek max operation, koi exponentials nahi.
Vanishing gradient solve karta hai: Gradient $z > 0$ ke liye 1 hai, isliye backprop signals shrink nahi hote.
Sparse activation: Negative inputs zero output karte hain—network "switch off" karna seekhta hai irrelevant neurons ko.

WHEN to use: Deep networks mein hidden layers ke liye default choice (image recognition, control systems).

2. Sigmoid

Properties:

Range: $(0, 1)$ — naturally probability ki tarah interpret hota hai.
Smooth: Har jagah differentiable.
Saturates: $∣ z ∣ > 5$ ke liye, gradient $\approx 0$ (vanishing gradient problem).

WHEN to use: Binary classification output layer (jaise, "Kya yeh sensor reading anomalous hai?").

3. Tanh (Hyperbolic Tangent)

Properties:

Range: $(- 1, 1)$ — zero-centered (sigmoid ke unlike).
Stronger gradients zero ke paas sigmoid se zyada.
Extremes par still saturates karta hai.

WHEN to use: Hidden layers jab aap zero-centered activations chahte ho (convergence mein help karta hai), ya outputs jo signed quantities represent karte hain (jaise, control surface deflection: -1 = full down, +1 = full up).

Comparison Table

Function	Range	Zero-Centered?	Gradient Issues	Best For
ReLU	$[0, \infty)$	No	Dead neurons (agar $z < 0$ always)	Hidden layers, deep networks
Sigmoid	$(0, 1)$	No	Vanishing gradient	Binary output layer
Tanh	$(- 1, 1)$	Yes	Vanishing gradient	Hidden layers (sigmoid se better), signed outputs

Recall Ek 12-Saal ke Bachche ko Explain Karo

Socho tum ek robot pilot bana rahe ho. Usse decide karna hai: "Kya main control stick upar kheenchun ya neeche dhabaooon?"

Robot ka brain chhoti chhoti decision-makers se bana hai jinhe neurons kehte hain. Har neuron sensor readings (airspeed, altitude) dekhta hai, har ek ko ek number se multiply karta hai jise weight kehte hain (jaise bolna "airspeed altitude se 3× zyada important hai"), unhe add karta hai, phir ek bias add karta hai (ek starting nudge).

Lekin yahan trick hai: agar hum sirf add aur multiply karte rahein, toh robot sirf straight-line rules seekh sakta hai. Real flying curvy aur complex hoti hai! Isliye hum activation function use karte hain—ek magical math rule jo seedhi line ko curves mein mod deta hai.

ReLU ek darwaaze ki tarah hai: negative numbers zero ho jaate hain (darwaza band), positive numbers rehte hain (darwaza khula). Fast aur simple.
Sigmoid sab kuch 0 aur 1 ke beech squish kar deta hai, jaise ek confidence meter: "Mujhe 90% yakeen hai ki yeh emergency hai."
Tanh -1 aur +1 ke beech squish karta hai, perfect hai "stick aage dabao (-1) ya peeche kheecho (+1)" ke liye.

Agar yeh bending functions nahi hote, toh robot brain dumb hota—sirf fancy addition. Inke saath, yeh storms mein fly karna, obstacles dodge karna, kuch bhi seekh leta hai!

Connections

Backpropagation and Gradient Descent — gradients compute karne ke liye activation derivatives crucial hain
Vanishing and Exploding Gradients — deep networks mein sigmoid/tanh kyun problematic hain
Weight Initialization Strategies — ReLU ke liye He initialization, sigmoid/tanh ke liye Xavier
Binary Cross-Entropy Loss — sigmoid output ke saath naturally pair karta hai
Convolutional Neural Networks — hidden layers mein almost exclusively ReLU use karte hain
Overfitting and Regularization — dropout aur L2 regularization activation choices ke saath interact karte hain
Sensor Fusion in Aerospace — neural networks IMU, GPS, radar data combine karte hain; activation choice robustness affect karta hai

#flashcards/coding

Ek artificial neuron ke computation ke teen components kya hain? :: Inputs ka weighted sum ( $\sum w_{i} x_{i}$ ), bias ( $b$ ), aur activation function ( $f$ ).

Neural networks ko activation functions ki zaroorat kyun hai?

Bina non-linearity ke, layers stack karna single linear transformation (

y = A x + b

) mein collapse ho jaata hai, jo complex patterns jaise XOR ya curved decision boundaries nahi seekh sakta.

ReLU ka formula kya hai?

f (z) = max (0, z)

. Output

z

hai agar positive,

0

agar negative.

ReLU ka derivative kya hai?

f^{'} (z) = 1

agar

z > 0

, warna

0

. (

z = 0

par undefined, lekin practice mein 0 ya 1 treat kiya jaata hai.)

ReLU sigmoid ke comparison mein kaunsa problem solve karta hai?

Vanishing gradient problem. ReLU ka gradient positive inputs ke liye 1 hai, jo deep networks mein gradient shrinkage prevent karta hai.

Sigmoid function ka formula kya hai?

σ (z) = \frac{1}{1 + e ^{- z}}

Sigmoid ka range kya hai?

(0, 1)

—naturally probability ki tarah interpret hota hai.

Sigmoid ka derivative derive karo.

σ^{'} (z) = σ (z) (1 - σ (z))

. Proof:

\frac{1}{1 + e ^{- z}}

par chain rule apply karo,

σ (z) \cdot (1 - σ (z))

mein factor karo.

Tanh function ka formula kya hai?

tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}

ya equivalently

2 σ (2 z) - 1

Tanh ka range kya hai?

(- 1, 1)

—zero-centered, signed outputs ke liye useful.

Tanh ka derivative kya hai? :: $tanh^{'} (z) = 1 - tanh^{2} (z)$ .

ReLU kab use karna chahiye?

Deep networks mein hidden layers ke liye default (fast, vanishing gradient se bachata hai).

Sigmoid kab use karna chahiye?

Binary classification output layer jahan tumhe

[0, 1]

mein probabilities chahiyen.

Tanh kab use karna chahiye?

Zero-centered activations ke liye hidden layers mein, ya outputs jo signed quantities represent karte hain (jaise control deflections).

"Dead ReLU" problem kya hai?

Ek neuron ka weighted sum hamesha negative hota hai, isliye output hamesha 0 hai. Gradient 0 hai, weights kabhi update nahi hote—neuron permanently inactive ho jaata hai.

Dead ReLU neurons kaise fix karte hain?

Leaky ReLU use karo (

f (z) = max (0.01 z, z)

), proper weight initialization (He), ya lower learning rate.

Deep networks mein hidden layers ke liye sigmoid kyun bura hai?

Vanishing gradient:

σ^{'} (z) \leq 0.25

, isliye gradients kai layers mein multiply hokar near-zero ho jaate hain, learning ruk jaati hai.

Ek neuron mein bias

b

ka kya role hai?

Activation threshold shift karta hai, neuron ko tab bhi activate hone deta hai jab inputs zero hon.

ReLU non-linearity kaise introduce karta hai agar

z > 0

ke liye linear dikhta hai?

z = 0

par "kink" hi non-linearity hai. Function har jagah differentiable nahi hai, jo linear composition property ko tod deta hai.