Noesis
3.1.4Neural Network Fundamentals

Activation functions - sigmoid, tanh

1,541 words7 min readdifficulty · medium6 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHY do we even need an activation function?

WHAT an activation does: takes a raw pre-activation number zz (any real value) and squashes it into a bounded, smooth output — a "soft" version of "is this neuron firing?".

Sigmoid (logistic function)

HOW to derive its derivative from scratch

Why this step? We need σ′\sigma' for backpropagation, and it has a beautiful self-referential form.

Start with σ(z)=(1+e−z)−1\sigma(z) = (1+e^{-z})^{-1}. Use the chain rule: σ′(z)=−(1+e−z)−2⋅ddz(1+e−z)=−(1+e−z)−2⋅(−e−z)\sigma'(z) = -(1+e^{-z})^{-2}\cdot \frac{d}{dz}(1+e^{-z}) = -(1+e^{-z})^{-2}\cdot(-e^{-z}) =e−z(1+e−z)2= \frac{e^{-z}}{(1+e^{-z})^2}

Why this step? Rewrite e−z=(1+e−z)−1e^{-z} = (1+e^{-z}) - 1 to factor out σ\sigma: σ′(z)=(1+e−z)−1(1+e−z)2=11+e−z−1(1+e−z)2\sigma'(z) = \frac{(1+e^{-z}) - 1}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}} - \frac{1}{(1+e^{-z})^2} =σ(z)−σ(z)2=σ(z)(1−σ(z))= \sigma(z) - \sigma(z)^2 = \sigma(z)\big(1-\sigma(z)\big)

Tanh (hyperbolic tangent)

Why this step? Showing tanh⁡\tanh is a rescaled, shifted sigmoid explains why they behave similarly but tanh is centered.

σ(2z)=11+e−2z=ezez+e−z\sigma(2z)=\frac{1}{1+e^{-2z}}=\frac{e^{z}}{e^{z}+e^{-z}} Then 2σ(2z)−1=2ezez+e−z−ez+e−zez+e−z=ez−e−zez+e−z=tanh⁡(z)2\sigma(2z)-1 = \frac{2e^{z}}{e^{z}+e^{-z}} - \frac{e^{z}+e^{-z}}{e^{z}+e^{-z}} = \frac{e^{z}-e^{-z}}{e^{z}+e^{-z}} = \tanh(z)

Derive tanh⁡′\tanh': with tanh⁡=sinh⁡cosh⁡\tanh = \frac{\sinh}{\cosh}, quotient rule gives cosh⁡2−sinh⁡2cosh⁡2=1cosh⁡2=1−tanh⁡2\frac{\cosh^2 - \sinh^2}{\cosh^2} = \frac{1}{\cosh^2} = 1-\tanh^2 (using cosh⁡2−sinh⁡2=1\cosh^2-\sinh^2=1).

Figure — Activation functions - sigmoid, tanh

Worked Examples

Common Mistakes

Recall Explain it to a 12-year-old (hidden)

Think of a light dimmer switch. The raw signal zz is how hard you push the knob. The sigmoid turns that into a brightness between 0% (off) and 100% (fully on) — but softly, so a small push near the middle changes brightness a lot, while pushing an already-bright light barely changes it. Tanh is the same idea but the dial goes from −100% to +100%, so it can say "definitely no", "meh", or "definitely yes". Because the change is smooth (no sudden jumps), the network can learn by nudging knobs in the right direction. But when a knob is pushed to an extreme, wiggling it does almost nothing — that's why very deep networks "stop learning".

Connections

  • ReLU and Leaky ReLU — the modern fix for vanishing gradients.
  • Softmax — the normalized multi-class cousin of sigmoid.
  • Backpropagation — where σ′(z)=σ(1−σ)\sigma'(z)=\sigma(1-\sigma) is used every step.
  • Vanishing and Exploding Gradients — caused by saturating activations.
  • Logistic Regression — literally a single sigmoid neuron.
  • Weight Initialization — chosen to keep pre-activations off the saturated tails.

Flashcards

Why does a network need a nonlinear activation?
Without it, stacked linear layers collapse into one linear map; nonlinearity gives the network real expressive power.
Sigmoid formula?
σ(z)=11+e−z\sigma(z)=\dfrac{1}{1+e^{-z}}, range (0,1)(0,1).
Sigmoid derivative in terms of itself?
σ′(z)=σ(z)(1−σ(z))\sigma'(z)=\sigma(z)(1-\sigma(z)).
Maximum slope of sigmoid and where?
0.250.25 at z=0z=0.
Tanh formula and range?
tanh⁡(z)=ez−e−zez+e−z\tanh(z)=\dfrac{e^z-e^{-z}}{e^z+e^{-z}}, range (−1,1)(-1,1).
Relationship between tanh and sigmoid?
tanh⁡(z)=2σ(2z)−1\tanh(z)=2\sigma(2z)-1.
Tanh derivative?
tanh⁡′(z)=1−tanh⁡2(z)\tanh'(z)=1-\tanh^2(z); max 1.01.0 at z=0z=0.
Key advantage of tanh over sigmoid?
Tanh is zero-centered, avoiding all-positive-activation zig-zag gradients.
What is the vanishing gradient problem here?
In saturated tails σ′≈0\sigma'\approx 0, so chained gradients shrink toward zero and learning stalls.
Best use case for sigmoid today?
The output neuron of a binary classifier (probability interpretation).

Concept Map

stacking

fixed by

provides

enables

enables

maps to

maps to

derived

derived

equals 2*sigma 2z -1

used in

used in

Linear layers only

Collapses to one linear map

Activation function

Nonlinearity

Sigmoid

Tanh

sigma' = sigma*1-sigma

tanh' = 1 - tanh^2

Output 0 to 1 as probability

Zero-centered -1 to 1

Backpropagation

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, neural network ka har neuron pehle ek weighted sum banata hai: z=Wx+bz = Wx+b. Agar hum ispe koi nonlinear function na lagayein, to poora deep network sirf ek badi si linear line ban jaata hai — chahe 100 layers ho ya 1, farak nahi padta. Isliye humein activation function chahiye jo curve/bend laa sake. Sigmoid aur tanh dono "S-shape" wale squasher hain jo kisi bhi bade ya chhote number ko ek chhote range me daba dete hain.

Sigmoid number ko 00 se 11 ke beech laata hai — isko probability ki tarah padh sakte ho ("neuron kitna fire kar raha hai"). Iska derivative bahut pyaara hai: σ′=σ(1−σ)\sigma'=\sigma(1-\sigma), matlab backprop me dubara exponential calculate karne ki zarurat nahi. Tanh wahi cheez hai par −1-1 se +1+1 tak — yeh zero-centered hota hai, isliye hidden layers me aksar behtar train karta hai. Ek important trick: tanh⁡(z)=2σ(2z)−1\tanh(z)=2\sigma(2z)-1, yaani tanh basically sigmoid ka scaled-shifted version hai (scaling by 2), aur z=0z=0 pe 4 guna zyada steep (slope 1.0 vs 0.25).

Sabse bada problem: jab zz bahut bada ya bahut chhota ho jaata hai, curve flat ho jaata hai aur slope (derivative) almost zero. Backprop me chain rule ke through yeh chhoti slopes multiply hoti rehti hain, to gradient "vanish" ho jaata hai aur network seekhna band kar deta hai. Isi dard ki wajah se aaj hidden layers me log zyada tar ReLU use karte hain, aur sigmoid ko sirf binary output neuron ke liye rakhte hain. Yaad rakho: Sig = single-sided probability (slope 0.25), Tanh = two-sided, four-times-as-steep.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 1,915 wordsD2Visual walkthroughThe derivation in pictures · 2,379 wordsD3Worked examplesEvery scenario covered · 2,036 wordsD4ExercisesGraded, full solutions · 2,695 wordsD5Question bankConcept traps, quick reveals · 1,601 words

Test yourself — Neural Network Fundamentals

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Backpropagation — chain rule, gradient computationCoding · 5.6.8Vanishing and exploding gradientsAI-ML · 3.1.11Logistic regression — sigmoid, cross-entropy lossCoding · 5.6.2Weight initialization (Xavier, He)AI-ML · 3.1.12Functions, limits, and continuityAI-ML · 1.2.1Derivatives and rules (product, quotient, chain)AI-ML · 1.2.2Chain rule for multivariate functions (backprop foundation)AI-ML · 1.2.14The perceptron model and historyAI-ML · 3.1.1Forward propagation computationAI-ML · 3.1.3Softmax for output layersAI-ML · 3.1.6