3.1.4
Intuition Woh 20% jo 80% result deta haiEk neural network bina nonlinear activation ke sirf ek glorified matrix multiply hai — sau linear layers stack karo aur phir bhi ek hi linear function milega. Activation functions woh nonlinearity hain jo ek network ko bend, curve, aur aisa data separate karne deti hain jo ek seedhi line kabhi nahi kar sakti. Sigmoid aur tanh do classic "S-shaped" (sigmoidal) squashers hain.
Socho har layer z = W x + b W 2 z + b 2 = W 2 ( W 1 x + b 1 ) + b 2 ( W 2 W 1 ) x + ( … ) ek bada linear map . Chahe kitna bhi deep ho, sab collapse ho jaata hai. Expressive power paane ke liye hum layers ke beech ek nonlinear function σ a = σ ( W x + b ) collapse nahi karti.
WHAT ek activation karta hai: ek raw pre-activation number z
σ ( z ) = 1 + e − z 1 R → ( 0 , 1 ) probability ki tarah padha ja sakta hai (kabhi exactly 0 ya 1 nahi hota).
Yeh step kyun? Backpropagation ke liye hume σ ′
σ ( z ) = ( 1 + e − z ) − 1 σ ′ ( z ) = − ( 1 + e − z ) − 2 ⋅ d z d ( 1 + e − z ) = − ( 1 + e − z ) − 2 ⋅ ( − e − z ) = ( 1 + e − z ) 2 e − z
Yeh step kyun? e − z = ( 1 + e − z ) − 1 σ σ ′ ( z ) = ( 1 + e − z ) 2 ( 1 + e − z ) − 1 = 1 + e − z 1 − ( 1 + e − z ) 2 1 = σ ( z ) − σ ( z ) 2 = σ ( z ) ( 1 − σ ( z ) )
tanh ( z ) = e z + e − z e z − e − z R → ( − 1 , 1 ) zero-centered hai (output average 0 ke paas hota hai).
Yeh step kyun? Yeh dikhana ki tanh rescaled, shifted sigmoid hai, explain karta hai ki dono similar kyun behave karte hain lekin tanh centered kyun hai.
σ ( 2 z ) = 1 + e − 2 z 1 = e z + e − z e z 2 σ ( 2 z ) − 1 = e z + e − z 2 e z − e z + e − z e z + e − z = e z + e − z e z − e − z = tanh ( z )
tanh ′ tanh = c o s h s i n h c o s h 2 c o s h 2 − s i n h 2 = c o s h 2 1 = 1 − tanh 2 cosh 2 − sinh 2 = 1
Worked example Example 1 — Sigmoid neuron ka forward passNeuron: weights w = [ 1 , − 2 ] b = 0.5 x = [ 2 , 1 ] Step 1: z = w ⋅ x + b = ( 1 ) ( 2 ) + ( − 2 ) ( 1 ) + 0.5 = 0.5 Kyun? Pre-activation weighted sum hota hai.
Step 2: σ ( 0.5 ) = 1 + e − 0.5 1 = 1 + 0.6065 1 = 0.622 Kyun? Squasher apply karo.
Interpretation: neuron ~62% confidence ke saath "fire" kar raha hai.
Worked example Example 2 — Backprop ke dauran local gradientSame neuron, output a = σ ( z ) = 0.622 = ∂ a ∂ L = 0.8 Step 1: σ ′ ( z ) = a ( 1 − a ) = 0.622 ( 0.378 ) = 0.235 Kyun? Self-referential form use karo — exponentials dobara compute karne ki zaroorat nahi!
Step 2: ∂ z ∂ L = 0.8 × 0.235 = 0.188 Kyun? Chain rule incoming gradient ko local slope se multiply karta hai.
Worked example Example 3 — Saturated tail mein vanishing gradientz = 6 σ ( 6 ) = 0.9975 σ ′ ( 6 ) = 0.9975 ( 0.0025 ) ≈ 0.0025 Yeh kyun important hai: local slope bahut chhota hai, toh backprop upstream gradient ko ~0.0025 se multiply karta hai. Aisi kai layers chain karo aur gradients 0 ki taraf shrink ho jaate hain → learning ruk jaati hai . Yahi vanishing gradient problem hai.
Common mistake "Sigmoid outputs probabilities hain jo neurons mein sum karke 1 banti hain."Kyun sahi lagta hai: har output ( 0 , 1 ) Fix: sigmoids har neuron ke liye independent hote hain — yeh sum karke 1 NAHI bante. Classes par probability distribution ke liye softmax chahiye. Sigmoid independent binary decisions ke liye hai.
Common mistake "Tanh aur sigmoid basically same hain, koi bhi choose kar lo."Kyun sahi lagta hai: dono S-shaped saturating functions hain. Fix: tanh zero-centered (( − 1 , 1 ) ( 0 , 1 ) binary output neuron ke liye.
Common mistake "Deeper hamesha better hota hai, activations depth ko limit nahi karte."Kyun sahi lagta hai: zyaada layers = zyaada capacity, intuitively. Fix: sigmoid/tanh ke saath, gradients saturate ho jaate hain (Example 3), toh bahut deep stacks vanishing gradients suffer karte hain. Yahi historical pain hai jis wajah se ReLU popular hua.
Recall Ise ek 12-saal ke bacche ko explain karo (hidden)Socho ek light dimmer switch. Raw signal z Sigmoid use 0% (off) aur 100% (fully on) ke beech ki brightness mein convert karta hai — lekin smoothly, toh beech mein thodi si push se brightness bahut badlti hai, jabki pehle se bright light ko push karne se almost kuch nahi badalta. Tanh same idea hai lekin dial −100% se +100% tak jaata hai, toh yeh "bilkul nahi", "meh", ya "bilkul haan" bhi bol sakta hai. Kyunki change smooth hai (koi achanak jumps nahi), network sahi direction mein knobs nudge karke seekh sakta hai. Lekin jab koi knob extreme par push hota hai, toh use hilane se almost kuch nahi hota — yahi reason hai ki bahut deep networks "seekhna band kar dete hain".
Network ko nonlinear activation kyun chahiye? Iske bina, stacked linear layers ek linear map mein collapse ho jaati hain; nonlinearity network ko real expressive power deti hai.
Sigmoid formula? σ ( z ) = 1 + e − z 1 ( 0 , 1 )
Sigmoid derivative khud ke terms mein? σ ′ ( z ) = σ ( z ) ( 1 − σ ( z ))
Sigmoid ki maximum slope aur woh kahan hai? 0.25 z = 0
Tanh formula aur range? tanh ( z ) = e z + e − z e z − e − z ( − 1 , 1 )
Tanh aur sigmoid ka relationship? tanh ( z ) = 2 σ ( 2 z ) − 1
Tanh derivative? tanh ′ ( z ) = 1 − tanh 2 ( z ) 1.0 z = 0
Tanh ka sigmoid ke upar key advantage? Tanh zero-centered hai, jisse all-positive-activation zig-zag gradients se bacha ja sakta hai.
Yahan vanishing gradient problem kya hai? Saturated tails mein σ ′ ≈ 0
Aaj sigmoid ka best use case kya hai? Binary classifier ka output neuron (probability interpretation).
Collapses to one linear map
Output 0 to 1 as probability
