Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Exercises

5.6.6 · D4 · Coding › Machine Learning (Aerospace Applications) › Neural network fundamentals — neuron, activation functions (

Shuru karne se pehle, char objects ka ek reminder jo hum baar baar use karenge — yeh padho chahe tumhe lagta ho ki tum jaante ho, kyunki har solution isi pe lean karta hai.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Upar ke blueprint ko dekho (har curve legend mein naam se labeled hai, sirf colour se nahi): ReLU ek flat floor hai phir ek $45°$ ramp. Sigmoid ek gentle S hai jo $0$ aur $1$ ke beech trapped hai. Tanh ek steeper S hai jo $- 1$ aur $1$ ke beech trapped hai. Figure ke arrows har curve ko naam se point out karte hain. Yeh picture apne dimag mein rakho — zyaadatar L1/L2 answers sirf yeh hain ki "main kaunsi curve pe khada hoon, aur kahan?"

Level 1 — Recognition

Problem 1.1

Kaunsa activation function value $- 0.7$ output kar sakta hai? (ReLU, sigmoid, tanh mein se choose karo.)

Recall Solution 1.1

HUM KYA DEKHTE HAIN: har function ka range (possible outputs ka set).

ReLU output range hai $[0, \infty)$ — yeh kabhi negative nahi ho sakta (iska floor $0$ hai).
Sigmoid range hai $(0, 1)$ — strictly positive, kabhi $0$ ya $1$ nahi reach karta.
Tanh range hai $(- 1, 1)$ — sirf yahi zero ke neeche jaata hai.

$- 0.7$ , $- 1$ aur $0$ ke beech hai, isliye yeh sirf tanh se reachable hai.

YESH KAISE DIKHTA HAI: figure mein, sirf tanh curve (legend mein labeled) kabhi horizontal axis ke neeche jaati hai.

Problem 1.2

Ek 12-layer network ki hidden layer mein ek neuron ko aisi activation chahiye jiska derivative positive inputs ke liye signals ko shrink na kare. Use naam do aur uska poora derivative do (negative branch aur $z = 0$ pe convention including).

Recall Solution 1.2

ReLU. $z > 0$ ke liye, $ReLU (z) = z$ , toh uski slope exactly $1$ hai. $z < 0$ ke liye, $ReLU (z) = 0$ , ek flat line jiska slope $0$ hai. Exactly $z = 0$ pe dono branches ek corner pe milti hain jahan derivative truly defined nahi hai; convention se hum ise simply $0$ assign karte hain ( $z \leq 0$ side ke saath group karke). Poori cheez likhte hain: $ReLU^{'} (z) = {10 z > 0 z \leq 0$ Active branch pe $1$ ki slope ka matlab hai backpropagation $1$ se multiply karta hai — signal bina dimmed hue pass hota hai, isliye ReLU deep hidden layers ke liye default hai. Contrast: Vanishing and Exploding Gradients tab hota hai jab $1$ se chhoti slopes kaafi saari layers mein ek saath multiply hoti hain.

Level 2 — Application

Problem 2.1

Ek neuron ke weights $w = (0.4, - 0.6)$ , bias $b = 1.0$ , inputs $x = (2.0, 3.0)$ hain. $z$ compute karo, phir ReLU apply karo.

Recall Solution 2.1

Step 1 — weighted sum (KYA & KYUN): hum har input ko uske weight se multiply karte hain uske contribution ko score karne ke liye, phir bias nudge add karte hain. Yahan $n = 2$ . $z = (0.4) (2.0) + (- 0.6) (3.0) + 1.0 = 0.8 - 1.8 + 1.0 = 0.0$ Step 2 — activation: exactly $0$ pe ReLU: $ReLU (0) = max (0, 0) = 0.$ Output $y = 0$ . Edge case note karo: $z = 0$ ReLU curve ke corner pe baithta hai. Hamari definition se $f (z) = 0$ for $z \leq 0$ , toh darwaza abhi abhi band hai.

Problem 2.2

$z = 2$ ke liye, $σ (2)$ 4 decimals tak compute karo, phir identity $σ^{'} (z) = σ (z) (1 - σ (z))$ use karke wahan derivative nikalo.

Recall Solution 2.2

Step 1: $σ (2) = \frac{1}{1 + e ^{- 2}} = \frac{1}{1 + 0.135335} = \frac{1}{1.135335} \approx 0.8808$ .

Step 2 (KYUN yeh identity): hum har baar scratch se differentiate nahi karna chahte; yeh identity humein abhi compute ki gayi value reuse karne deti hai. $σ^{'} (2) = 0.8808 (1 - 0.8808) = 0.8808 \times 0.1192 \approx 0.1050.$ Padhna: lagbhag $0.105$ ki slope — pehle hi maximum $0.25$ se kaafi neeche. Sigmoid $z = 2$ pe flatten (saturate) hona shuru ho rahi hai.

Problem 2.3

$z = - 1.2$ ke liye, $tanh (- 1.2)$ compute karo aur phir $tanh^{'} (- 1.2)$ compute karo $tanh^{'} (z) = 1 - tanh^{2} (z)$ use karke.

Recall Solution 2.3

Step 1: $tanh (- 1.2) = \frac{e ^{- 1.2} - e ^{1.2}}{e ^{- 1.2} + e ^{1.2}} = \frac{0.301194 - 3.320117}{0.301194 + 3.320117} = \frac{- 3.018922}{3.621311} \approx - 0.8337$ .

Step 2 (KYUN value reuse karein): sigmoid jaisa hi trick — output ko square karo aur $1$ se subtract karo. $tanh^{'} (- 1.2) = 1 - (- 0.8337)^{2} = 1 - 0.6951 \approx 0.3049.$ Padhna: $z = - 1.2$ pe tanh ka ek healthier slope ( $\approx 0.30$ ) hai sigmoid ke $z = 2$ pe wale se — yahi tanh ka "middle ke paas stronger gradient" wala advantage hai.

Level 3 — Analysis

Problem 3.1

Ek student ek "two-layer linear network" propose karta hai bina kisi activation ke: $h = W_{1} x + b_{1}, y = W_{2} h + b_{2} .$ Algebraically dikhao ki yeh ek single-layer linear model ke equivalent hai, aur ek sentence mein bolo ki yeh problem kyun hai.

Recall Solution 3.1

$h$ ko $y$ mein substitute karo: $y = W_{2} (W_{1} x + b_{1}) + b_{2} = (W_{2} W_{1}) x + (W_{2} b_{1} + b_{2}) .$ $W_{eff} = W_{2} W_{1}$ aur $b_{eff} = W_{2} b_{1} + b_{2}$ maano. Tab $y = W_{eff} x + b_{eff},$ jo ek single linear layer hai. Ek sentence mein problem: chahe tum kitne bhi linear layers stack karo, woh ek straight-line map mein collapse ho jaate hain, toh network sirf straight decision boundaries draw kar sakta hai aur kabhi XOR jaisa curved pattern nahi seekh sakta. Ilaaj hai layers ke beech ek non-linear $f$ insert karna.

Problem 3.2

Sigmoid ki maximum slope $0.25$ hai, $z = 0$ pe hoti hai. $σ^{'} (z) = σ (z) (1 - σ (z))$ use karke yeh maximum value prove karo.

Recall Solution 3.2

KYUN yeh method: $σ^{'}$ do cheezoon ka product hai jinka sum $1$ hai: $p = σ (z)$ maano, toh $σ^{'} = p (1 - p)$ jahan $p \in (0, 1)$ .

$p (1 - p)$ maximize karo: ise $p$ mein ek downward parabola ki tarah treat karo. W.r.t. $p$ iska derivative $1 - 2 p$ hai, $p = 0.5$ pe zero. Toh peak $p = σ (z) = 0.5$ pe hai, jo $z = 0$ pe hota hai. $σ_{m a x}^{'} = 0.5 (1 - 0.5) = 0.25.$ Consequence: har sigmoid slope $\leq 0.25$ hai. 10-layer stack mein slopes ka product $\leq 0.2 5^{10} \approx 9.5 \times 1 0^{- 7}$ hai — yahi exactly vanishing gradient problem hai, aur isliye gradient descent ruk jaata hai.

Problem 3.3

Identity $tanh (z) = 2 σ (2 z) - 1$ ko $z = 0.5$ pe numerically verify karo (4 decimals), confirm karo ki dono forms agree karte hain.

Recall Solution 3.3

Left side: $tanh (0.5) = \frac{e ^{0.5} - e ^{- 0.5}}{e ^{0.5} + e ^{- 0.5}} = \frac{1.648721 - 0.606531}{1.648721 + 0.606531} = \frac{1.042190}{2.255252} \approx 0.4621$ .

Right side: $σ (2 \cdot 0.5) = σ (1) = \frac{1}{1 + e ^{- 1}} = \frac{1}{1.367879} \approx 0.7311$ ; phir $2 (0.7311) - 1 = 0.4621$ .

Dono $\approx 0.4621$ dete hain — identity holds. Isliye tanh "ek stretched, shifted sigmoid" hai: input ko $2$ se scale karo, output ko $2$ se scale karo, aur zero pe recenter karne ke liye $1$ se neeche shift karo.

Level 4 — Synthesis

Problem 4.1

Ek control-output neuron elevator deflection ke liye weights $w = (0.5, - 0.25)$ , bias $b = 0.1$ , inputs $x = (1.0, 4.0)$ ke saath hai. Output $[- 1, 1]$ mein hona chahiye jahan $- 1$ = full down, $+ 1$ = full up. (a) Sahi activation choose karo aur justify karo. (b) Deflection compute karo.

Recall Solution 4.1

(a) Choice — KYUN tanh: output signed aur $[- 1, 1]$ tak bounded hona chahiye. ReLU negative nahi ja sakta (toh koi "down" nahi); sigmoid $(0, 1)$ mein stuck hai (woh bhi koi "down" nahi). Sirf tanh $(- 1, 1)$ ko zero ke aas paas symmetrically cover karta hai — ek control surface ke liye perfect jo neutral se dono taraf move karti hai.

(b) Compute karo (yahan $n = 2$ ): $z = (0.5) (1.0) + (- 0.25) (4.0) + 0.1 = 0.5 - 1.0 + 0.1 = - 0.4.$ $tanh (- 0.4) = \frac{e ^{- 0.4} - e ^{0.4}}{e ^{- 0.4} + e ^{0.4}} = \frac{0.670320 - 1.491825}{0.670320 + 1.491825} = \frac{- 0.821505}{2.162145} \approx - 0.3799.$ Padhna: elevator maximum ke lagbhag $38.0%$ downward deflect karta hai (negative = down). Sensor Fusion in Aerospace se compare karo, jahan aisi signed outputs flight controller ko feed karti hain.

Problem 4.2

Ek anomaly detector ke output neuron ka $z = 3.5$ hai. (a) Kaunsa activation output ko probability banata hai, aur kyun? (b) Woh probability compute karo, aur (c) ek truly-anomalous sample (label $= 1$ ) ke liye Binary Cross-Entropy Loss $L = - ln (p)$ use karke loss compute karo.

Recall Solution 4.2

(a) Humein $[0, 1]$ mein output chahiye jo "anomaly ki chance" ki tarah readable ho, toh sigmoid. ReLU $3.5$ output kar sakta hai (probability nahi); tanh negative ja sakta hai (negative "chance" nonsense hai).

(b) $σ (3.5) = \frac{1}{1 + e ^{- 3.5}} = \frac{1}{1 + 0.030197} = \frac{1}{1.030197} \approx 0.9707$ . → lagbhag $97%$ confident hai ki yeh anomalous hai.

(c) Ek true anomaly (label $1$ ) ke liye, binary cross-entropy hai $L = - ln (p) = - ln (0.9707) \approx 0.02975$ . Padhna: loss tiny hai kyunki network confident aur correct tha — exactly yahi hum chahte hain. Agar usne $0.03$ predict kiya hota, loss $- ln (0.03) \approx 3.51$ tak balloon ho jaata.

Level 5 — Mastery

Problem 5.1 (Dead ReLU + ek fix)

Ek neuron ReLU use karta hai. Poore dataset pe uska pre-activation hamesha $z \leq 0$ hai, toh $y = 0$ har baar. (a) Explain karo kyun gradient descent ise kabhi revive nahi kar sakta (ek "dead neuron"). (b) ReLU ko Leaky ReLU $f (z) = max (0.01 z, z)$ se replace karo aur $z = - 5$ pe $f (z)$ aur $f^{'} (z)$ compute karo. (c) Ek sentence mein explain karo ki yeh neuron ko kyun rescue karta hai.

Recall Solution 5.1

(a) $z \leq 0$ ke liye ReLU ki slope $0$ hai. Backprop ke dauran, weight update us slope ke proportional hota hai, toh $0$ ka update matlab weights kabhi nahi badalte — neuron hamesha ke liye "off" pe frozen hai. Weight Initialization Strategies bhi dekho, kyunki buri init ek common cause hai.

(b) $z = - 5$ pe (negative branch, Leaky ReLU $0.01 z$ use karta hai): $f (- 5) = max (0.01 \cdot (- 5), - 5) = max (- 0.05, - 5) = - 0.05.$ $f^{'} (- 5) = 0.01 (leaky branch ki slope) .$ (c) Kyunki slope ab $0$ ki jagah $0.01$ hai, ek small non-zero gradient flow karta hai back, toh weights abhi bhi nudge ho sakte hain aur neuron recover ho sakta hai.

Problem 5.2 (Vanishing-gradient budget)

Tum $L$ sigmoid layers stack karte ho, aur derivatives ka best-case product $0.2 5^{L}$ hai. Sabse bada $L$ dhundho jiske liye yeh product $1 0^{- 3}$ se upar rahe (taaki learning abhi bhi hopeless na ho). Logarithms se solve karo.

Recall Solution 5.2

KYUN logs: humein ek exponent ko unstack karna hai, aur logarithm woh tool hai jo "power of" ko "multiply by" mein convert karta hai. $0.2 5^{L} > 1 0^{- 3} ⟹ L ln (0.25) > ln (1 0^{- 3}) .$ Ab $ln (0.25) = - 1.386294$ (negative) hai, toh divide karne se inequality flip ho jaati hai: $L < \frac{l n ( 1 0 ^{- 3} )}{l n ( 0.25 )} = \frac{- 6.907755}{- 1.386294} \approx 4.982.$ Kyunki $L$ poora number hona chahiye, sabse bada allowed hai $L = 4$ . Padhna: best case mein bhi, sirf lagbhag chaar stacked sigmoids gradients ko $1 0^{- 3}$ se upar rakhte hain — solid proof ki deep sigmoid stacks doomed hain, aur isliye hum ReLU ya Vanishing and Exploding Gradients mein techniques tak pahunchte hain. Yahi reasoning Convolutional Neural Networks design karte waqt aur Overfitting and Regularization control karte waqt aati hai.

Recall Self-test checklist

ReLU / sigmoid / tanh ka range? ::: $[0, \infty)$ / $(0, 1)$ / $(- 1, 1)$ . Linear layers ka stack kyun collapse hota hai? ::: Linear maps compose karke ek aur linear map $W_{eff} x + b_{eff}$ deta hai. Maximum sigmoid slope aur kahan? ::: $0.25$ , $z = 0$ pe hoti hai (jahan $σ = 0.5$ ). Signed, bounded control output ke liye kaunsi activation? ::: tanh. Probability output ke liye kaunsi? ::: sigmoid. Leaky ReLU dead neuron ko kaise revive karta hai? ::: Yeh $z < 0$ ke liye ek small non-zero slope ( $0.01$ ) deta hai toh gradients abhi bhi flow karte hain.