Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Worked examples

5.6.6 · D3 · Coding › Machine Learning (Aerospace Applications) › Neural network fundamentals — neuron, activation functions (

Tumne parent note mein neuron aur uske teen activation functions dekhe hain. Is page ka ek hi kaam hai: har activation function ko har tarah ke number ke saath drag karke dikhana — bada positive, bada negative, bilkul zero, bahut chota, aur real aircraft data — taaki jab tum exam mein baitho ya network debug karo, tum woh case pehle se dekh chuke ho.

Shuru karne se pehle, sirf teen rules ka do-line recap — seedhe words mein, taaki koi symbol bina wajah na lage.

Scenario matrix

Har neuron input ek single number $z$ hai. Numbers ke sirf itne hi shapes hote hain, aur activations bhi sirf itne. Yeh grid poori duniya hai — neeche har worked example ek row fill karta hai.

Cell	Scenario class	Kya test kar rahe hain	Example
A	$z > 0$ large	ReLU pass karta hai, sigmoid → 1, tanh → +1 (saturation)	Ex. 1
B	$z < 0$ large	ReLU zero kar deta hai, sigmoid → 0, tanh → −1	Ex. 2
C	$z = 0$ exactly	Teeno ka pivot point	Ex. 3
D	$z$ small (near 0), dono signs	Jahan gradients sabse strong hote hain; symmetry	Ex. 4
E	Limiting behaviour	$z \to \pm \infty$ : kaunsi value approach hoti hai par kabhi reach nahi hoti?	Ex. 5
F	Gradient / vanishing	derivative values, kyun deep sigmoid mar jaata hai	Ex. 6
G	Real-world word problem	poora neuron: weights + bias + activation	Ex. 7
H	Exam twist	dead ReLU + Leaky ReLU rescue	Ex. 8

Pehle poora landscape ek saath dekho — har curve ki shape se answer milta hai, koi arithmetic se pehle.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Cell A — large positive $z$

Worked example Example 1:

$z = + 6$ teeno se Forecast: aage padhne se pehle har output guess karo. ReLU kya dega? Sigmoid kis value ko approach karta hai par reach nahi karta? Tanh ke baare mein kya?

Step 1 — ReLU. $f (6) = max (0, 6) = 6$ . Yeh step kyun? Jab $z > 0$ ho, ReLU identity hai — number seedha copy ho jaata hai. Koi squashing nahi.

Step 2 — Sigmoid. $σ (6) = \frac{1}{1 + e ^{- 6}} = \frac{1}{1 + 0.002479} \approx 0.99753$ . Yeh step kyun? $e^{- 6}$ bahut chota hai (bada positive $z$ matlab $- z$ bada negative, aur $e$ ki badi-negative power near zero hoti hai), toh denominator 1 se thoda upar hai, aur poora fraction 1 se thoda neeche baith jaata hai.

Step 3 — Tanh. $tanh (6) = \frac{e ^{6} - e ^{- 6}}{e ^{6} + e ^{- 6}} = \frac{403.43 - 0.00248}{403.43 + 0.00248} \approx 0.99999$ . Yeh step kyun? $e^{- 6}$ , $e^{6}$ ke samne negligible hai, toh upar aur neeche dono $\approx e^{6}$ hain aur ratio $\approx 1$ hai.

Verify: Teeno increasing functions hain, toh bada positive input bade outputs dena chahiye — deta hai. Sigmoid strictly 1 se neeche rehta hai, tanh strictly 1 se neeche (dono approach karte hain par kabhi touch nahi karte). Sanity ✔.

Cell B — large negative $z$

Worked example Example 2:

$z = - 6$ teeno se Forecast: negative number par ReLU kya karta hai? Aur sigmoid/tanh kis floor ki taraf jaate hain?

Step 1 — ReLU. $f (- 6) = max (0, - 6) = 0$ . Yeh step kyun? $0$ , $- 6$ se bada hai, toh max $0$ hai. ReLU "neuron ko switch off kar deta hai."

Step 2 — Sigmoid. $σ (- 6) = \frac{1}{1 + e ^{6}} = \frac{1}{1 + 403.43} \approx 0.002473$ . Yeh step kyun? Ab $- z = + 6$ hai, toh $e^{- z} = e^{6}$ bahut bada hai, denominator dominate kar raha hai; fraction tiny ho jaata hai → $0$ ke floor ke paas.

Step 3 — Tanh. $tanh (- 6) \approx - 0.99999$ . Yeh step kyun? Tanh ek odd function hai — input ka sign palto toh output ka sign bhi palat jaata hai: $tanh (- 6) = - tanh (6)$ .

Verify: Bada-negative input → sabse chhote outputs. ReLU apne floor par exactly pahuncha ( $0$ ); sigmoid apne floor ke paas pahuncha ( $0^{+}$ ); tanh apne floor ke paas pahuncha ( $- 1^{+}$ ). Note karo sigmoid $σ (- 6) + σ (6) = 0.002473 + 0.99753 \approx 1$ — yeh symmetry $σ (- z) = 1 - σ (z)$ ki nishaani hai. ✔

Cell C — exactly zero

Sabse important single point: teeno curves origin region se guzarti hain, aur yahan unke derivatives (steepness) peak karte hain.

Worked example Example 3:

$z = 0$ — pivot Forecast: $max (0, 0)$ kya hai? Aur $\frac{1}{1 + e ^{0}}$ ? Yaad karo $e^{0} = 1$ .

Step 1 — ReLU. $f (0) = max (0, 0) = 0$ . Yeh step kyun? Dono arguments $0$ hain; max $0$ hai. Hinge bilkul axis par baitha hai. (Iska derivative yahan technically undefined hai — graph mein ek corner hai — lekin code bas $0$ pick kar leta hai.)

Step 2 — Sigmoid. $σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{1 + 1} = 0.5$ . Yeh step kyun? $e^{0} = 1$ exactly. Sigmoid ka centre $0.5$ hai — "maximum uncertainty," uski 0-to-1 strip ka dead-centre.

Step 3 — Tanh. $tanh (0) = \frac{1 - 1}{1 + 1} = 0$ . Yeh step kyun? Upar $e^{0} - e^{0} = 0$ hai, toh tanh ka centre $0$ hai — yahi reason hai ki tanh ko zero-centered kehte hain, aur sigmoid ko nahi.

Verify: Figure mein $z = 0$ read karo: magenta origin ko touch karta hai, violet height $0.5$ par cross karta hai, orange height $0$ par cross karta hai. Match karta hai. ✔

Cell D — small $z$ near the origin (dono signs)

Worked example Example 4:

$z = + 0.5$ aur $z = - 0.5$ — near-linear zone aur uski symmetry Forecast: zero ke paas curves almost straight lines hain. Guess karo ki tanh ya sigmoid mein se kaun zyada move karta hai same small input ke liye — aur kya $- 0.5$ daalna $+ 0.5$ daalne ka mirror hai.

Step 1 — ReLU at $+ 0.5$ . $f (0.5) = max (0, 0.5) = 0.5$ . Yeh step kyun? Positive input → identity, toh ReLU bas copy kar deta hai.

Step 2 — Sigmoid at $+ 0.5$ . $σ (0.5) = \frac{1}{1 + e ^{- 0.5}} = \frac{1}{1 + 0.60653} \approx 0.62246$ . Yeh step kyun? Ek chota positive input output ko $0.5$ centre se thoda upar push karta hai.

Step 3 — Tanh at $+ 0.5$ . $tanh (0.5) = \frac{e ^{0.5} - e ^{- 0.5}}{e ^{0.5} + e ^{- 0.5}} = \frac{1.64872 - 0.60653}{1.64872 + 0.60653} \approx 0.46212$ . Yeh step kyun? Same input, lekin tanh $0$ se climb karta hai aur origin ke paas steeper hai, toh woh apne centre se zyada dur jaata hai.

Step 4 — Ab negative twin, $z = - 0.5$ . ReLU: $f (- 0.5) = max (0, - 0.5) = 0$ . Sigmoid: $σ (- 0.5) = \frac{1}{1 + e ^{0.5}} \approx 0.37754$ . Tanh: $tanh (- 0.5) \approx - 0.46212$ . Yeh step kyun? Yeh symmetry test karta hai jo har function apne centre ke around rakhta hai. ReLU symmetric nahi hai — woh negative side ko flat zero kar deta hai. Sigmoid apne centre $0.5$ ke around symmetric hai: dekho $0.37754 = 1 - 0.62246$ , yaani $σ (- z) = 1 - σ (z)$ . Tanh $0$ ke around anti-symmetric hai: $tanh (- 0.5) = - tanh (0.5)$ exactly.

Verify: Centre se movement ki distance (magnitude), $+ 0.5$ ke liye: sigmoid $0.62246 - 0.5 = 0.122$ ; tanh $0.46212 - 0 = 0.462$ . $- 0.5$ ke liye: sigmoid $0.5 - 0.37754 = 0.122$ (same magnitude, opposite side); tanh $0 - (- 0.46212) = 0.462$ (same magnitude, opposite side). Toh near-origin gradient $\pm 0.5$ ke liye identical hai — curves locally symmetric hain — aur tanh sigmoid se dono taraf ~ $3.8 \times$ zyada move karta hai. ✔

Cell E — limiting behaviour ( $z \to \pm \infty$ )

Worked example Example 5: Kaunsi value approach hoti hai par kabhi reach nahi hoti?

Forecast: woh chaar horizontal lines naam batao jinhein curves flatten karke approach karte hain.

Step 1 — Sigmoid limits. Jab $z \to + \infty$ , $e^{- z} \to 0$ , toh $σ \to \frac{1}{1 + 0} = 1$ . Jab $z \to - \infty$ , $e^{- z} \to + \infty$ , toh $σ \to \frac{1}{\infty} = 0$ . Yeh step kyun? Limits asymptotes describe karte hain — woh ceiling aur floor jo curve hamesha ke liye creep karta hai bina touch kiye. Sigmoid strictly $(0, 1)$ ke andar trapped hai.

Step 2 — Tanh limits. Jab $z \to + \infty$ , $tanh \to \frac{e ^{z}}{e ^{z}} = + 1$ ; jab $z \to - \infty$ , $tanh \to - 1$ . Toh tanh strictly $(- 1, 1)$ ke andar rehta hai. Yeh step kyun? Bade $+ z$ par $e^{- z}$ terms vanish ho jaate hain aur upar aur neeche dono $e^{z}$ ban jaate hain, giving $+ 1$ ; odd symmetry phir dum-left par $- 1$ de deti hai. Yeh tanh ke do asymptotes hain.

Step 3 — ReLU limit. Jab $z \to + \infty$ , $f \to + \infty$ (koi ceiling nahi — yahi reason hai ReLU positive side par kabhi saturate nahi hota). Jab $z \to - \infty$ , $f = 0$ (ek hard floor). Yeh step kyun? Positive side par ReLU identity $f (z) = z$ hai, jo bina bound ke grow karta hai, toh koi upper asymptote nahi hai; negative side par $max (0, z)$ bilkul $0$ par pinned hai — yeh approached limit ki jagah ek genuine floor hai. Yahi asymmetry exactly reason hai ki ReLU active side par saturation dodge karta hai.

Verify (numeric proxy). Bada $z = 20$ plug karo: $σ (20) = 0.99999999794 \approx 1$ lekin $< 1$ ; $tanh (20) \approx 1$ lekin $< 1$ . Kabhi equal nahi, hamesha thoda short. ✔

Yahi saturation ki poori kahani hai: ek baar flat tails mein pahunch gaye, $z$ move karne par output barely change hota hai, toh derivative $\approx 0$ — dekho Vanishing and Exploding Gradients.

Cell F — gradients aur vanishing problem

Derivative curve ki ek point par steepness hai. Training mein (dekho Backpropagation and Gradient Descent) yeh steepness values layer by layer multiply hoti hain, toh hamen jaanna zaroori hai ki yeh kitni chhoti ho sakti hain.

Worked example Example 6: Sigmoid ka sabse bada possible gradient, aur uske 10 layers

Forecast: sigmoid derivative hai $σ^{'} (z) = σ (z) (1 - σ (z))$ . Yeh sabse bada kahan hai, aur kitna bada?

Step 1 — Peak dhundho. Product $p (1 - p)$ (jahan $p = σ (z) \in (0, 1)$ ) sabse bada hota hai jab $p = 0.5$ , yaani $z = 0$ par. Yeh step kyun? Ek number times "one minus itself" ek downward parabola hai jo $0.5$ par peak karta hai; yeh basic hai aur yaad rakhne layak hai.

Step 2 — Peak par value. $σ^{'} (0) = 0.5 \times (1 - 0.5) = 0.25$ . Yeh step kyun? Yeh sigmoid ka maximum slope hai — toh har sigmoid gradient satisfy karta hai $σ^{'} (z) \leq 0.25$ .

Step 3 — 10 layers chain karo. Backprop ek factor per layer multiply karta hai. Best case: $0.2 5^{10}$ . $0.2 5^{10} = 9.5367 \times 1 0^{- 7} \approx 1 0^{- 6} .$ Yeh step kyun? Best case mein bhi gradient apne starting value ka ek millionth hai — deep layers barely learn karte hain. Yahi vanishing gradient hai.

Step 4 — ReLU contrast. Active neurons ke liye $f^{'} (z) = 1$ , toh $1^{10} = 1$ — koi shrinkage nahi. Compare karo tanh ka peak: $tanh^{'} (0) = 1 - tanh^{2} (0) = 1 - 0 = 1$ , sigmoid ke peak se chaar guna steeper.

Verify: $0.2 5^{10} = 9.5367431640625 \times 1 0^{- 7}$ ✔; $tanh^{'} (0) = 1$ ✔; $σ^{'} (0) = 0.25$ ✔. Yahi exactly reason hai ki hidden layers mein ReLU use karte hain (parent ka mistake callout dekho).

Cell G — real-world word problem (poora neuron)

Worked example Example 7: Ek pitch-hold neuron real sensor data ke saath

Ek autopilot mein ek neuron "climb tendency" score estimate karta hai. Uske do inputs hain — normalised airspeed $x_{1} = 0.7$ aur normalised altitude-rate $x_{2} = - 0.4$ — weights $w_{1} = 1.5$ , $w_{2} = 2.0$ aur bias $b = - 0.3$ ke saath. Yeh tanh use karta hai kyunki output ek signed control tendency hai (climb +, dive −).

Forecast: kya score positive hoga (climb) ya negative (dive)? Compute karne se pehle sign guess karo.

Step 1 — Weighted sum. $z = w_{1} x_{1} + w_{2} x_{2} + b = 1.5 (0.7) + 2.0 (- 0.4) + (- 0.3)$ . Yeh step kyun? Yeh neuron ka core hai: importance-weighted evidence plus resting nudge. $z = 1.05 - 0.8 - 0.3 = - 0.05.$

Step 2 — Activation. $tanh (- 0.05) = \frac{e ^{- 0.05} - e ^{0.05}}{e ^{- 0.05} + e ^{0.05}} \approx - 0.049958$ . Yeh step kyun? Zero ke paas tanh almost linear hai, toh ek chota negative $z$ ek almost-equal chota negative output deta hai.

Step 3 — Interpret karo. Output $\approx - 0.05$ : ek bahut halki dive tendency, essentially neutral. Full down authority ka kareeb $5%$ . Yeh step kyun? Tanh ki $[- 1, 1]$ range directly "full dive … full climb" se map hoti hai, toh magnitude = kitna hard, sign = kaun sa direction.

Verify: Units/range check — output $(- 1, 1)$ ke andar land karta hai ✔ aur uska sign $z$ ke sign se match karta hai (dono negative) kyunki tanh sign preserve karta hai ✔. Agar humne (galti se) yahan sigmoid use kiya hota, toh milta $σ (- 0.05) \approx 0.4875$ — ek positive number jo "dive" express nahi kar sakta. Sahi tool choose kiya. ✔

Cell H — exam twist: dead ReLU rescued by Leaky ReLU

Worked example Example 8: Ek neuron har data par

$z = - 3$ par stuck hai Ek hidden neuron har training sample par $z = - 3$ produce karta hai. Dikhaao ki woh ReLU ke under dead hai, aur Leaky ReLU ( $f (z) = max (0.01 z, z)$ ) se use rescue karo.

Forecast: yahan ReLU ka output aur uska gradient kya hai? "Gradient = 0" ka matlab kyun hai ki neuron kabhi recover nahi kar sakta?

Step 1 — ReLU output. $f (- 3) = max (0, - 3) = 0$ . Yeh step kyun? $0$ , $- 3$ se bada hai, toh max $0$ hai — ReLU har negative pre-activation ko same flat floor par clamp kar deta hai, bilkul koi output signal nahi deta.

Step 2 — ReLU gradient. $f^{'} (- 3) = 0$ (flat zero region). Yeh step kyun? Training mein weight updates is gradient ke proportional hote hain. Exactly $0$ gradient ka matlab hai update $0$ hai — weights kabhi move nahi karte, $z$ hamesha $- 3$ rehta hai. Dead neuron.

Step 3 — Leaky ReLU output. $f (- 3) = max (0.01 \times (- 3), - 3) = max (- 0.03, - 3) = - 0.03$ . Yeh step kyun? $- 0.03$ , $- 3$ se bada hai, toh leak jeet jaata hai: neuron kuch nahi ki jagah ek tiny signal emit karta hai.

Step 4 — Leaky ReLU gradient. $z < 0$ ke liye slope $0.01$ hai, toh $f^{'} (- 3) = 0.01 \neq = 0$ . Yeh step kyun? Nonzero gradient ka matlab hai updates abhi bhi flow kar rahe hain — neuron positive territory ki taraf crawl back kar sakta hai aur revive ho sakta hai. (Pehle se prevent karne ke liye Weight Initialization Strategies dekho.)

Verify: $z = - 3$ par dono compare karo: ReLU $(0, 0)$ vs Leaky ReLU $(- 0.03, 0.01)$ . Rescue exactly nonzero derivative hai. Aur positive input ke liye, jaise $z = + 3$ , dono agree karte hain: $max (0.03, 3) = 3$ — Leaky ReLU good side par kuch nahi badalta. ✔

Recall Self-check: pehle cell name karo, phir answer

$max (0, - 6)$ ::: $0$ (Cell B — large negative par ReLU) $σ (0)$ ::: $0.5$ (Cell C — pivot) $tanh (0)$ ::: $0$ (Cell C — zero-centered) Sigmoid jis value ko $z \to + \infty$ par approach karta hai ::: $1$ , lekin kabhi reach nahi karta (Cell E) $σ^{'} (z)$ ki maximum possible value ::: $0.25$ , $z = 0$ par (Cell F) $z < 0$ ke liye Leaky ReLU gradient ::: $0.01$ (nonzero → neuron alive rehta hai, Cell H) 10 sigmoid layers mein best-case gradient ::: kareeb $1 0^{- 6}$ — vanishing gradient (Cell F) $σ (- 0.5)$ ko $σ (0.5)$ ke terms mein ::: $1 - σ (0.5) \approx 0.37754$ (Cell D — sigmoid symmetry)

Related: Vanishing and Exploding Gradients · Backpropagation and Gradient Descent · Binary Cross-Entropy Loss (sigmoid outputs yahan feed hote hain) · Sensor Fusion in Aerospace (jahan yeh neurons real telemetry read karte hain).