Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Foundations

5.6.6 · D1 · Coding › Machine Learning (Aerospace Applications) › Neural network fundamentals — neuron, activation functions (

Parent note padhne se pehle, tumhe har symbol ko dekh kar yeh sochne ki zaroorat hai ki woh bol kya raha hai. Yeh page ek neuron ka poora alphabet walk karta hai — humble subscript se lekar exponential $e^{- z}$ tak — har ek ko uske pehle waale ke upar build karta hua. Yahan kuch bhi assume nahi kiya gaya ki tumne "multiply aur add" se aage algebra dekhi hai.

1. Ek number jo kuch represent karta hai: variable

Ek cockpit gauge imagine karo. Airspeed labelled gauge ek moment mein $150$ dikhata hai aur ek minute baad $210$ . Label $x$ hai; needle reading andar wala number hai. Hum letters isliye use karte hain taaki hum reading ke baare mein baat kar sakein bina use ek value pe freeze kiye.

Parent note $x$ ko ek sensor reading ke liye use karta hai — airspeed, altitude, temperature. Bas yahi hai $x$ ka matlab: ek input number.

2. Ek jaisi bahut saari cheezein: subscript

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Figure dekho. Hamare paas gauges ki ek row hai, ek har input ke liye. Har ek ke liye nayi letter invent karne ki bajay ( $a$ airspeed ke liye, $h$ altitude ke liye, …) hum ek hi letter $x$ reuse karte hain aur sirf neecha wala label badlate hain. Subscripts sirf isi liye exist karte hain: yeh kehne ke liye ki "same type ki cheez, alag slot."

3. Counting ka shorthand: $i$ , $n$ , aur $\sum$

Ab woh scary symbol. Parent note likhta hai $\sum_{i = 1}^{n} w_{i} x_{i}$ .

Figure mein loop unrolled dikhaya gaya hai. $\sum_{i = 1}^{3} w_{i} x_{i}$ literally waisa hi hai jaise likhna: $w_{1} x_{1} + w_{2} x_{3} ? no — w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} .$ Dono letters ke neechey same counter hai, toh slot $1$ slot $1$ ke saath pair karta hai. $\sum$ sirf isliye hai taaki jab hundred inputs hon toh hume sao + signs na likhne padein. Is topic ko kyun chahiye: ek real sensor bank mein dozens of readings hoti hain; loop notation neuron formula ko ek line mein rakhta hai chahe kitne bhi inputs hon.

4. Multiply-then-add: weighted sum $z$

Ab hum neuron ka dil padh sakte hain. Do words:

Ek mixing desk imagine karo (figure). Har sensor ek slider $w_{i}$ ko feed karta hai. Ise upar slide karo, final knob $z$ mein us sensor ka contribution badhta hai. Single output number $z$ woh "master volume" hai jis par neuron react karega. Kyun chahiye: learning ka poora point yahi hai ki yeh dials ghuma dein — network data se discover karta hai ki pitch predict karne ya fault spot karne ke liye kaun se sensors matter karte hain.

5. Resting nudge: bias $b$

Ek bathroom scale imagine karo jo kuch na hone par bhi $2 kg$ dikhata hai. Woh constant $+ 2$ ek bias hai. Ek neuron mein, agar har input $0$ ho, toh weighted sum bhi $0$ hoga — bias ke bina neuron us case mein "nothing" output karne par majboor hoga. Bias use ek resting value deta hai jis par woh lean kar sake, taaki woh tab bhi fire kar sake jab inputs quiet hon. Is topic ko kyun chahiye: yeh har neuron ko yeh choose karne deta hai ki uski decision line kahan baithe, sirf kitni steep hai yahi nahi.

6. Bending machine: function $f (\cdot)$ kya hota hai

Parent note ka neuron hai $y = f (z)$ : master number $z$ lo, machine $f$ se guzaro, result ko $y$ kaho. Teen machines jo parent introduce karta hai — ReLU, sigmoid, tanh — teen alag bending rules hain. Inhe padhne ke liye hume do aakhri tools chahiye: exponential aur steepness ka idea.

7. Curves ke liye zaroorat: exponential $e^{- z}$

Sigmoid aur tanh dono mein $e^{- z}$ hai. Aao ise samjhein.

Figure mein dono curves follow karo. Sigmoid $e^{- z}$ kyun use karta hai: hum ek aisi machine chahte hain jiska output smoothly $0$ se $1$ tak slide kare. Trick $\frac{1}{1 + e ^{- z}}$ bilkul yahi karta hai:

Jab $z$ bahut negative ho, $e^{- z}$ bahut bada hai, toh $\frac{1}{1 + huge} \to 0$ .
Jab $z$ bahut positive ho, $e^{- z} \to 0$ , toh $\frac{1}{1 + 0} \to 1$ .
$z = 0$ par, $e^{0} = 1$ , toh $\frac{1}{1 + 1} = \frac{1}{2}$ — dead centre.

Toh $e^{- z}$ woh ingredient hai jo sigmoid ko $0$ aur $1$ ke beech uski smooth S-shape deta hai. Hum yeh tool use karte hain aur plain straight line nahi, precisely isliye kyunki hum ek gentle, always-differentiable squash chahte hain na ki hard cut.

8. Curve yahan kitni steep hai: derivative $f^{'} (z)$

Parent note baar baar $f^{'} (z)$ , $σ^{'} (z)$ , $tanh^{'} (z)$ likhta hai. Dash ko out loud "prime" padho.

Curve par khade ho kar socho "agar main ek baal daayein kadam rakhun, toh kitna upar jaaunga?" Woh rise-per-step hi derivative hai. Is topic ko kyun chahiye: learning (dekho Backpropagation and Gradient Descent) weights ko slope ki direction mein nudge karke kaam karta hai. Agar koi machine wide range par flat ho ( $f^{'} \approx 0$ ) — jaise sigmoid bade $∣ z ∣$ ke liye hoti hai — toh nudge signal mar jaata hai. Wahi flat-region death exactly woh vanishing gradient hai jisके baare mein parent warn karta hai.

Recall ReLU gradients kyun alive rakhta hai jabki sigmoid unhe kill karta hai?

ReLU ka slope positive $z$ ke liye $1$ hai (steady climb, signal pass hota hai), jabki sigmoid ka slope zyada se zyada $0.25$ hai aur saturation par $0$ ke kareeb — aaise bahut saare chote slopes ko multiply karo toh learning signal kuch nahi reh jaata.

9. Sab kuch ek saath — neuron ek saanthe mein

Parent ka master formula ab symbol by symbol padho: $y = f (\sum_{i = 1}^{n} w_{i} x_{i} + b)$

$x_{i}$ — $i$ -th input gauge (§1–3)
$w_{i}$ — uska importance dial (§4)
$\sum_{i = 1}^{n}$ — loop jo saare $n$ ko multiply-add karta hai (§3–4)
$+ b$ — resting nudge (§5)
$f (\dots)$ — bending machine, ReLU / sigmoid / tanh mein se koi ek (§6–7)
$y$ — single output number

Yahi poora neuron hai. Parent page par baaki sab kuch yeh choose karna hai ki kaun si bending machine $f$ use karni hai, aur learning ke liye yeh kitni steep ( $f^{'}$ ) hai.

Prerequisite map

Har foundation agले ko feed karta hai; do streams (forward blend-and-bend, aur learning ke liye zaroorat ka slope) neuron par milte hain, jo parent topic ka darwaza hai.

Equipment checklist

Khud test karo — kya tum reveal karne se pehle har jawab bol sakte ho?

x_{3}

mein neecha number tumhe kya batata hai?

Yeh ek subscript hai — label meaning "input number three," power nahi.

\sum_{i = 1}^{n} w_{i} x_{i}

ko plain English mein padho.

i

ko 1 se

n

tak loop karo, har baar input

x_{i}

ko uske weight

w_{i}

se multiply karo, aur saare results add karo.

Weight kya hai, ek sentence mein?

Ek dial jo kehta hai ki woh input neuron ke decision ke liye kitna matter karta hai.

Bias

b

kyun add karte hain?

Taaki neuron tab bhi non-zero output produce kar sake jab har input zero ho — ek resting value jise woh shift kar sake.

f (z)

kya hai?

Ek machine jo ek number

z

leta hai aur ek fixed bending rule se ek output return karta hai.

e^{- z}

bada hai ya chota jab

z

bahut positive ho?

Chota — yeh zero ki taraf jaata hai, jo sigmoid ka output 1 ki taraf push karta hai.

f^{'} (z)

mein prime kya measure karta hai?

z

par

f

ka slope (steepness) — input mein thodi si nudge se output kitna change hota hai.

Vanishing gradients kyun matter karte hain?

Agar slope wide range par zero ke kareeb ho, toh learning signal shrink ho jaata hai aur weights update hona band ho jaate hain.