Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Question bank

5.6.6 · D5 · Coding › Machine Learning (Aerospace Applications) › Neural network fundamentals — neuron, activation functions (

Shuru karne se pehle, ek term samajh lo taaki neeche kuch bhi mysterious na lage: pre-activation $z$ woh plain number hai jo tumhe $z = \sum_{i} w_{i} x_{i} + b$ se milta hai kisi bhi activation function ke touch karne se pehle. Activation $y = f (z)$ woh hai jo baad mein nikalta hai. Neeche sab kuch inhe alag-alag rakhne par hinge karta hai.

True or false — justify karo

100 linear layers ko bina kisi activation ke stack karna ek curved decision boundary seekh sakta hai.

False — poora stack ek single

W_{eff} x + b_{eff}

mein collapse ho jaata hai, jo sirf straight lines/planes khenchta hai. Layers ki number koi farak nahi daalti jab tak unke beech non-linearity na ho.

Ek neuron jiske saare weights zero aur bias zero hai woh hamesha ke liye useless hai.

Uss waqt True hai (output

f (0)

hai har input ke liye), lekin training gradients ke zariye weights ko move kar sakti hai — jab tak activation gradient ko kill na kare, isliye initialization matter karta hai (Weight Initialization Strategies).

ReLU ek non-linear function hai chahe uske dono pieces straight lines hain.

True —

z = 0

par "kink" hi non-linearity hai. Ek function linear tabhi hota hai jab woh origin se guzarne wali ek single straight line ho; zero par ka bend woh tod deta hai.

Sigmoid hamesha ek valid probability output karta hai, isliye yeh kisi bhi output layer ke liye safe choice hai.

False — iska range

(0, 1)

sirf binary probability ke liye suit karta hai, lekin yeh signed quantities (control deflection) ya multi-class distributions represent nahi kar sakta, aur hidden layers mein yeh vanishing gradients cause karta hai.

Tanh bas sigmoid shifted aur stretched hai.

True — exactly

tanh (z) = 2 σ (2 z) - 1

. Iska wahi S-shape hai lekin re-centered hai taaki

(0, 1)

ki jagah

(- 1, 1)

span kare.

Bias term optional hai kyunki tum hamesha inputs ko rescale kar sakte ho.

False — inputs ko rescale karna slope change karta hai, vertical shift nahi. Bina

b

ke activation ek fixed point se guzarne ke liye forced hota hai, isliye neuron apna threshold move nahi kar sakta.

ReLU kabhi vanishing gradients se suffer nahi karta.

Iske positive side par mostly true hai (derivative exactly 1 hai), lekin negative side par gradient 0 hai, jo dead neuron problem cause karta hai — yeh ek alag failure hai, vanishing nahi.

Ek bada weight hamesha matlab hai woh input zyada important hai.

Generally False — importance weight magnitude par input scale ke relative depend karti hai. Ek input par bada weight jo hamesha tiny ho woh ek large input par chhote weight se kam matter kar sakta hai; isliye input normalization matter karta hai.

Error dhundho

"Hum output layer mein ReLU use karte hain taaki network ek probability de."

Error — ReLU

[0, \infty)

output karta hai,

[0, 1]

nahi. Probabilities ke liye sigmoid (binary) ya softmax (multi-class) chahiye; ReLU hidden layers mein hota hai.

"Sigmoid ki derivative 1 jitni badi ho sakti hai, isliye gradients deep networks mein survive karte hain."

Error —

σ^{'} (z) = σ (z) (1 - σ (z))

z = 0

par peak karta hai value

0.25

ke saath, 1 nahi.

\leq 0.25

ke kai factors ko multiply karne se gradients fast shrink hote hain (Vanishing and Exploding Gradients).

"tanh zero-centered hai, isliye yeh kabhi saturate nahi ho sakta."

Error — zero-centering aur saturation unrelated hain. tanh large

∣ z ∣

ke liye phir bhi

\pm 1

tak flatten hota hai, jahan iska derivative

1 - tanh^{2} (z)

0 tak jaata hai.

"Ek dead ReLU neuron kuch epochs baad khud recover kar lega."

Error — agar saare data ke liye

z < 0

hai, toh us neuron ke through gradient 0 hai, isliye uske weights kabhi update nahi hote. Yeh dead rehta hai jab tak Leaky ReLU use na karo ya re-initialize na karo.

"Non-linearity weighted sum se aati hai, isliye jitne zyada weights utna zyada non-linear neuron."

Error — weighted sum

\sum w_{i} x_{i} + b

kitne bhi terms hon, perfectly linear hai. Non-linearity sirf activation function

f

se aati hai.

"Kyunki tanh negative values deta hai, iska gradient negative ho sakta hai, jo learning rok deta hai."

Error — output negative ho sakta hai, lekin iska derivative

1 - tanh^{2} (z)

hamesha

(0, 1]

mein hai, yaani non-negative. Learning ki direction loss se set hoti hai, activation output ke sign se nahi.

"Consistency ke liye hamesha har layer mein same activation use karni chahiye."

Error — best practice inhe mix karti hai: hidden layers mein ReLU (ya variants) healthy gradients ke liye, aur output par sigmoid/tanh/softmax jo task ke hisaab se choose karo (probability vs. signed vs. class).

Why questions

Do linear layers ke beech

f (z)

insert karna expressiveness kyun bachaata hai?

Kyunki

f

algebraic collapse ko tod deta hai —

W_{2} f (W_{1} x + b_{1}) + b_{2}

ab ek single

A x + b

mein fold nahi ho sakta, isliye composition input space ko curves mein bend aur fold kar sakta hai.

ReLU hidden layers ke liye default kyun hai jaante hue bhi yeh

z = 0

par non-differentiable hai?

z = 0

par ek akela kink real data par almost kabhi nahi aata, aur iska constant slope-1 gradient vanishing problem ko avoid karta hai — sasti computation plus healthy gradients ek undefined point se zyada outweigh karte hain.

Sigmoid deep networks mein learning stall kyun karti hai?

Backprop mein, per-layer derivatives multiply hoti hain. Kyunki har

σ^{'} (z) \leq 0.25

hai, das layers ek factor

\leq 0.2 5^{10} \approx 1 0^{- 6}

deti hain, isliye sabse pehle layers ko near-zero updates milte hain (Backpropagation and Gradient Descent).

Jab S-shape use karni ho to hidden layers ke liye sigmoid ke bajaay tanh prefer kyun karein?

tanh zero-centered hai, isliye activations sab-positive nahi hote; yeh gradient directions balanced rakhta hai aur typically convergence speed karta hai. Zero ke paas iska slope (1 tak) sigmoid ke (0.25 tak) se bhi steeper hai.

Bias ek neuron ke "resting potential" ki tarah kyun kaam karta hai?

Yeh output set karta hai jab saare inputs zero hon:

y = f (b)

. Isse ek neuron bina kisi input signal ke fire (ya silent reh) sakta hai, activation threshold ko left ya right shift karta hai.

Careful weight initialization iss baat se kyun juda hai ki network learning start bhi karega ya nahi?

Poor scaling har

z

ko ek saturated region mein deep push kar sakta hai (sigmoid/tanh) ya zero ke neeche (ReLU), pehle step se gradients zero kar ke. He-initialization

z

ko sensitive range mein rakhta hai (Weight Initialization Strategies).

Edge cases

z = 0

par exactly ReLU kya output karta hai?

Zero, convention

max (0, z)

ke hisaab se jahan boundary "off" branch ko assign ki gayi hai. Wahan subderivative ambiguous hai, lekin frameworks

0

(ya

1

) pick karte hain aur yeh practically rarely matter karta hai.

z \to + \infty

aur

z \to - \infty

par

σ (z)

kya hai?

Yeh respectively

1

aur

0

ke paas jaata hai lekin kisi ko bhi reach nahi karta — range open interval

(0, 1)

hai, isliye sigmoid output kabhi exactly 0 ya 1 nahi hota.

tanh (0)

kya hai aur isse tanh "zero-centered" kyun kehlata hai?

tanh (0) = 0

, isliye zero pre-activation zero output deta hai. Origin ke baare mein symmetry ka matlab hai positive aur negative inputs symmetrically map hote hain, mean activation near zero rakhte hain.

Agar har input

x_{i} = 0

ho, toh neuron kya output karta hai?

y = f (b)

— sirf bias activation se pass hota hai. Exactly isliye bias exist karta hai: iske bina answer sab-zero inputs ke liye

f (0)

par forced hota.

Sigmoid use karne wale ek saturated neuron (

∣ z ∣

bahut large) ke liye, backprop mein iska gradient contribution kya hota hai?

Yeh near zero hota hai, isliye upstream weights is path ke through almost koi update nahi paate — neuron effectively frozen hai chahe woh abhi bhi output produce kar raha hai.

Kya ek Leaky ReLU neuron plain ReLU ki tarah permanently dead ho sakta hai?

Nahi —

z < 0

ke liye yeh phir bhi ek small slope pass karta hai (jaise

0.01

), isliye ek nonzero gradient hamesha flow karta hai aur weights active region ki taraf update hote reh sakte hain.

Recall Pakad ke rakhne ke liye ek-line summary

Linearity weighted sum mein rehti hai; saari expressive power aur saari gradient trouble activation function mein rehti hai — $f$ choose karo iss hisaab se ki layer ko kya chahiye (hidden ke liye gradient health, last ke liye output meaning).