Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Visual walkthrough

5.6.6 · D2 · Coding › Machine Learning (Aerospace Applications) › Neural network fundamentals — neuron, activation functions (

Neeche jo bhi hai woh ek aise bachche se shuru hota hai jisne sirf multiplication aur addition dekhi hai aur kuch nahi. Har symbol use hone se pehle earn kiya jaata hai.

Step 1 — Ek input, ek weight, ek bias: seedhi line

KYA HAI. Sabse chota possible neuron ek hi number leta hai andar — ise $x$ bolein — aur ek hi number bahar nikalta hai. Kisi bhi fancy function se pehle, neuron sirf do kaam karta hai: woh $x$ ko ek number $w$ se scale karta hai, phir result mein ek number $b$ shift karta hai. Is scaled-and-shifted number ko hum $z$ kehte hain:

z = how steep w \cdot the input x + where it crosses b

$x$ — input number (maan lo, airspeed reading).
$w$ — weight: yeh $x$ ko stretch ya squash karta hai. Bada $w$ = steep line, $x$ bahut zyaada matter karta hai.
$b$ — bias: yeh poori line ko upar ya neeche uthata hai bina steepness ko chhuye. Yeh $+ b$ hi woh cheez hai jo map ko strictly linear ki jagah affine banata hai.
$z$ — weighted sum, neuron ka raw score kisi bhi bending se pehle.

YEH DO OPERATIONS HI KYUN, AUR KOI NAHI? Scaling ( $w \cdot x$ ) kehne ka ek hi tarika hai "yeh input $3 \times$ zyaada important hai." Shifting ( $+ b$ ) neuron ko $x = 0$ par bhi kuch non-zero output karne deta hai. Saath milkar yeh sabse general straight line hain jo tum likh sakte ho — ek aisi line jo origin se guzarne ki zaroorat nahi, yaani affine line. Yahi exactly point hai — aur yahi exactly woh limitation bhi hai jisse hum aage takrayenge.

PICTURE. $z = w x + b$ ka graph ek ruler-straight line hai. $w$ iska tilt hai; $b$ woh height hai jahan yeh vertical axis cross karta hai.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Step 2 — Bahut saare inputs: line ban jaati hai weighted sum

KYA HAI. Real neurons ek saath bahut saare numbers dekhte hain: airspeed, altitude, angle-of-attack. Inhe $x_{1}, x_{2}, \dots, x_{n}$ bolein. Har ek ko apna khud ka weight $w_{1}, \dots, w_{n}$ milta hai. Neuron har pair ko multiply karta hai aur sab add karta hai, phir ek shared bias add karta hai:

z = every input, scaled by its importance w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n} + one shared shift b

"Saare $w_{i} x_{i}$ add karo" ka shuddh shorthand hai sum symbol $\sum$ (capital Greek "S", Sum ke liye):

z = i = 1 \sum n w_{i} x_{i} + b

$\sum_{i = 1}^{n}$ — " $i$ ko $1$ se $n$ tak chalao aur har term add karo." Kuch mysterious nahi: yeh sirf ek compact for loop hai jiska matlab hai $w_{1} x_{1} + \dots + w_{n} x_{n}$ .
Baaki sab exactly Step 1 jaisa hai, har input ke liye ek baar repeat.

ABHI BHI SIRF ADDITION AUR SCALING KYUN? Kyunki hum ne abhi tak kuch aur add nahi kiya hai. Chahe hazaar inputs hon, $z$ abhi bhi chhupe hue flat (affine) plane jaisa hai — straight line ka multi-input twin. Yeh thought pakad ke rakho; yeh aane waala hai humein bite karne.

PICTURE. Do inputs ke saath "line" ek tilted flat sheet ban jaati hai jo $(x_{1}, x_{2})$ floor ke upar float karti hai. Kisi bhi point par uski height $z$ hai.

Step 3 — Do neurons stack karo aur dekho line kaise bend karne se mana kar deti hai

KYA HAI. Deep learning ka promise hai stacking: ek layer ka output doosre mein dalo. Pehle humein apna shorthand single numbers (scalars) se upgrade karna hoga numbers ki lists (vectors) aur numbers ke grids (matrices) mein, kyunki ek real layer mein bahut saare neurons hote hain.

Definition Vectors, matrices, aur

$W x + b$ ka matlab

Vector numbers ka ek column hota hai, jaise input $x = [x_{1} x_{2}]$ jisme $2$ entries hain. Hum kehte hain $x$ ki dimension $2$ hai.
Matrix $W$ numbers ka ek grid hota hai. Agar layer 1 mein $3$ neurons hain jo woh $2$ inputs padh rahe hain, toh $W_{1}$ ek $3 \times 2$ grid hai (3 rows, 2 columns) — har neuron ke liye ek row of weights.
Matrix times vector, $W_{1} x$ : $W_{1}$ ki har row lo, use $x$ ke saath entry-by-entry multiply karo, aur sum karo — yeh exactly Step 2 ka weighted sum $\sum w_{i} x_{i}$ hai, har neuron ke liye ek baar. Result ek vector hai jisme $3$ entries hain (ek per neuron).
Bias vector $b_{1}$ (bhi $3$ entries) add karna har neuron ka score shift karta hai. Toh $h = W_{1} x + b_{1}$ matlab hai "saare $3$ neurons ke liye Step 2 ek saath chalao," jisse $3$ -entry vector $h$ milta hai.
Layer 2 mein (maan lo) $1$ neuron hai jo woh $3$ padh raha hai, uska $W_{2}$ shape $1 \times 3$ ka hai, bias $b_{2}$ ek single number, output $y$ ek single number.

Us vocabulary ke saath, layer 1 input vector $x$ ko hidden vector $h$ mein convert karti hai, aur layer 2 $h$ ko $y$ mein:

Layer 1: h = W_{1} x + b_{1}, Layer 2: y = W_{2} h + b_{2}

Pehle ko doosre mein substitute karo:

y = W_{2} (this is h W_{1} x + b_{1}) + b_{2} = one new weight box, call it W_{eff} (W_{2} W_{1}) x + one new bias, call it b_{eff} (W_{2} b_{1} + b_{2})

Yahan $W_{2} W_{1}$ matrix times matrix hai ( $1 \times 3$ times $3 \times 2$ se milta hai $1 \times 2$ grid) — ek combined weight box — aur $W_{2} b_{1} + b_{2}$ ek combined bias number hai.

YEH KYUN MATTER KARTA HAI? Final line dekho: iska exact shape hai $y = W_{eff} x + b_{eff}$ — phir se ek single affine map. Do layers ek mein collapse ho gayi. Do affine functions multiply karke shifts add karne se straightness nahi chhutti. Sau stack karo; phir bhi ek straight (affine) map.

PICTURE. Do straight ramps chain karne par ek straight ramp milta hai — woh fold jo tumne socha tha kabhi nahi aata.

Step 4 — Woh problem jo ek straight line solve nahi kar sakti: XOR

KYA HAI. Yahan ek aisa task hai jo curve maangta hai. Chaar points ek floor par:

$x_{1}$	$x_{2}$	chahiya label
0	0	0 (blue)
0	1	1 (orange)
1	0	1 (orange)
1	1	0 (blue)

Yeh hai XOR: output 1 jab inputs alag hon. Hum ek aisi boundary kheenchna chahte hain jo do orange points ko ek taraf aur do blue points ko doosri taraf rakhe.

YEH STEP 3 KI MACHINE SE KYUN IMPOSSIBLE HAI. Blue points opposite corners par baithe hain; orange bhi. Koi ek straight line nahi hai jo do diagonal pairs ko separate kar sake — try karo aur koi na koi point hamesha galat side par hoga. Kyunki stacked affine layers = ek straight boundary (Step 3), poora affine network XOR nahi seekh sakta.

PICTURE. Har straight cut mein ek point phansa reh jaata hai.

Step 5 — Ek bend insert karo: activation function

KYA HAI. Do layers ke beech hum ek function $f$ slip karte hain jo straight line nahi hai — ek activation function. Ab:

y = W_{2} \cdot apply the bend before layer 2 f (W_{1} x + b_{1}) + b_{2}

Sabse simple possible bend hai ReLU (Rectified Linear Unit) — "positives ko through jaane do, negatives ko zero par flatten karo":

f (z) = max (0, z) = {z 0 z > 0 z \leq 0

$z > 0$ ke liye: output equals input — ek $4 5^{\circ}$ ramp.
$z \leq 0$ ke liye: output flat zero hai — "darwaza band hai."
$z = 0$ par corner asli cheez hai: yeh ek kink hai, wahi non-straightness jo Step 3 mein nahi tha.

SIRF EK KINK HUMEIN KYUN BACHATA HAI? Ek concrete 1-D check. Usi input $x$ ke do affine pieces lo: $a = w_{1} x + b_{1}$ aur $c = w_{2} x + b_{2}$ . Inhe add karo, $a + c = (w_{1} + w_{2}) x + (b_{1} + b_{2})$ — abhi bhi ek straight line, koi escape nahi. Lekin pehle har ek ko ReLU se guzaro aur phir add karo:

g (x) = max (0, w_{1} x + b_{1}) + max (0, w_{2} x + b_{2}) .

Numbers try karo $w_{1} = 1, b_{1} = 0$ aur $w_{2} = - 1, b_{2} = 0$ , yaani $g (x) = max (0, x) + max (0, - x) = ∣ x ∣$ . Yeh hai absolute-value "V" — provably us form mein nahi $(constant) \cdot x + constant$ , kyunki iska slope left par $- 1$ aur right par $+ 1$ hai. $max$ sum ke across distribute karne se mana karta hai: $max (0, a) + max (0, c) \neq = max (0, a + c)$ generally (at $x = - 1$ : $1 + 0 = 1$ left side par, lekin $max (0, - 1 - 1 \cdot (- 1))$ -style bookkeeping $0$ deta hai). Ek bend ne humein ek genuine corner dilaya — Step 3 ka collapse toot gaya.

PICTURE. ReLU: left par flat, right par rising ramp, origin par ek sharp elbow ke saath.

Step 6 — Dekho bends space ko fold karte hain aur XOR solve karte hain

KYA HAI. Chote network ko do hidden ReLU neurons do. Har ReLU ek fold contribute karta hai. Do folds combine karke, network ek aisi boundary banata hai jo bent line ki tarah hoti hai — ek "V" ya corner — straight cut ki jagah.

YEH KAHAN STEP 4 FAIL HUA THA WAAHAN KYUN KAAM KARTA HAI. Ek corner do blue points ko ek region mein hug kar sakta hai aur do orange points ko bahar dhakela sakta hai. Impossible straight-line separation ek easy folded-region separation ban jaati hai. Yahi Step 5 ke kink ka payoff hai, visible banaya gaya.

PICTURE. Learned boundary blue diagonal ke around bend karti hai; orange aur blue ab saaf alag hain.

Blue region ← dono points $0$ label kiye.
Orange region ← dono points $1$ label kiye.
Boundary piecewise-straight hai (ReLU segments se bani) lekin globally curved — exactly jo ek affine layer kabhi produce nahi kar sakti thi.

Step 7 — Har bend barabar nahi hota: teen activations side by side

KYA HAI. ReLU ek bend hai; smoother bhi hain. Parent note ke teeno, ek axis par:

ReLU: max (0, z), σ (z) = \frac{1}{1 + e ^{- z}}, tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}

Har symbol ko jahan hai wahin padhna:

$e^{- z}$ — number $e \approx 2.718$ jo $- z$ ki power tak raised hai; $z$ badhne ke saath yeh fast shrink hota hai, isliye $σ$ $1$ ki taraf climb karta hai.
$σ (z)$ — sigmoid, kisi bhi input ko open interval $(0, 1)$ mein squash karta hai: ek confidence meter.
$tanh (z)$ — same S-shape lekin $(- 1, 1)$ mein squash hai, isliye yeh zero-centred hai (negatives allowed).

KAUN SA CHOOSE KAREIN, YEH KYUN MATTER KARTA HAI? Unki steepness alag hai, aur steepness wahi hai jis par gradient descent ride karta hai:

ReLU ka slope exactly $1$ hai $z > 0$ ke liye — learning signals bina kisi kharche ke pass hote hain.
Sigmoid ka slope kabhi $0.25$ se zyaada nahi hota (uska steepest point, $z = 0$ par); tanh ka kabhi $1$ se zyaada nahi lekin dono extremes par almost $0$ tak flatten ho jaate hain ("saturation").

PICTURE. ReLU ka hard elbow, sigmoid ka gentle $(0, 1)$ S apna $0.25$ max-slope tangent marked kiye, tanh ka symmetric $(- 1, 1)$ S apna slope- $1$ tangent aur $\pm 1$ asymptotes ke saath — sab overlaid.

Step 8 — Degenerate case: flat activation matlab koi activation nahi

KYA HAI. Agar koi "simplify" karke ek linear activation choose kare, $f (z) = z$ ? Tab

y = W_{2} (W_{1} x + b_{1}) + b_{2} = W_{eff} x + b_{eff}

— hum wapas Step 3 ke collapse par aa jaate hain. Usi tarah ek constant activation $f (z) = c$ saari input dependency khatam kar deta hai ( $y$ $x$ ko ignore karta hai).

YEH KYUN DIKHAAYEN. Yeh poore idea ki boundary nail karta hai: bend genuinely non-linear aur input-dependent hona chahiye. Straight ya flat $f$ silently sab kuch undo kar deta hai. Yahi reason hai ki ek ReLU neuron jiska $z$ saare data ke liye negative ho dead hai — locally woh flat, useless case hai, aur (parent ke Dead-ReLU note dekho) uska gradient $0$ hai isliye woh kabhi recover nahi karta.

PICTURE. Do "bad bends": identity line (affine mein collapse) aur flat constant (input ignore karta hai) — koi bhi space fold nahi kar sakta.

Ek-picture summary

Chaar panels left-to-right, top-to-bottom padhein — yeh poora page replay karte hain:

Top-left "linear = straight": ek affine layer $z = w x + b$ ek ruler line hai (Steps 1–3). Aur stack karne se kabhi bend nahi aata.
Top-right "line fails XOR": chaar XOR dots; red dashed line koi bhi straight cut hai, aur woh hamesha ek point stranded chhod deta hai (Step 4).
Bottom-left "add a bend (ReLU)": $max (0, z)$ insert karo — green ramp jisme origin par red kink hai woh naya non-linear ingredient hai (Step 5).
Bottom-right "folds solve XOR": do ReLU folds ek corner-shaped boundary banate hain jo finally blue ko orange se alag karti hai (Step 6).

The through-line: stretch (weights) → slide (bias, affine) → BEND (activation). Sirf bend straight-boundary trap se bahar nikalti hai.

Recall Feynman retelling — plain words mein wapas bolo

Ek aisi machine se shuru karo jo sirf do tricks jaanti hai: ek number ko stretch karo aur use slide karo. Stretch-and-slide wahi hai jo grown-ups affine map kehte hain — ek straight ruler line jo zero se guzarne ki zaroorat nahi. Ek sau aisi machines chain karo aur — surprise — phir bhi ek straight line milti hai, kyunki stretch ka stretch sirf bada stretch hai aur slide ka slide sirf bada slide. Ab yahan ek puzzle hai jo chaar dots banate hain, jise XOR kehte hain: do dots "yes" chahte hain, do "no", aur woh opposite corners par baithe hain. Ruler se inhe separate karne ki koshish karo — tum kar hi nahi sakte, kabhi bhi; koi na koi hamesha galat side par hoga. Toh hum ek nai trick add karte hain: stretch karne ke baad, number ko bend karo — sabse simple bend sirf saare negatives ko zero par flatten kar deta hai (woh hai ReLU, ek darwaza jo zero ke neeche band ho jaata hai). Woh single kink matlab hai ki chained machines ab ek straight ruler nahi rahi; woh paper fold kar sakti hain. (Proof jo feel ho sake: $max (0, x) + max (0, - x)$ hai V-shape $∣ x ∣$ — koi ruler V nahi banata.) Do folds ek corner banate hain, aur ek corner finally do "no" dots ke around wrap kar sakta hai aur do "yes" dots ko bahar chhod sakta hai. Yahi neural networks ka poora secret hai: stretch, slide, aur — sabse zaroori — bend. Koi genuinely curved bend choose karo (ReLU, sigmoid, tanh); koi flat ya straight choose karo aur tumne magic throw away kar di. Aur ReLU ke bottom mein woh ek sharp point ka koi proper slope nahi hai, toh computer wahin koi bhi value $0$ aur $1$ ke beech choose karta hai aur aage chal deta hai.

Recall Quick self-test

Stacked affine layers XOR kyun nahi seekh sakti? ::: Kyunki stacking affine layers ek single affine layer mein collapse ho jaati hai ( $W_{2} (W_{1} x + b_{1}) + b_{2} = W_{eff} x + b_{eff}$ ), jo sirf ek straight boundary kheench sakti hai — aur koi straight line XOR ke diagonal pairs separate nahi kar sakti. ReLU mein kink tumhe kya deta hai? ::: Ek genuine non-linearity: $max (0, z)$ ko straight line ki tarah nahi likha ja sakta (jaise $max (0, x) + max (0, - x) = ∣ x ∣$ ), isliye layers collapse karna band kar deti hain aur network apna input space curved boundaries mein fold kar sakta hai. Linear aur affine map mein kya fark hai? ::: Linear pure scaling hai $w x$ (origin se guzarta hua); affine ek shift add karta hai, $w x + b$ . "Linear layer" $W x + b$ actually affine hai jab bhi $b \neq = 0$ . ReLU ke liye $z = 0$ par kya hota hai? ::: Yeh non-differentiable hai (left slope $0$ , right slope $1$ ); software subgradient use karta hai — $[0, 1]$ mein koi bhi value, usually $0$ . Kaun sa activation zero-centred hai, aur yeh achha kyun hai? ::: $tanh$ , range $(- 1, 1)$ ; zero ke around symmetric hone se yeh signed values output kar sakta hai aur generally gradient descent ko converge karne mein help karta hai.

Yeh aage kahan jaata hai: har bend ki steepness control karti hai ki learning signals backward kaise flow karti hain — yeh subject hai Backpropagation and Gradient Descent aur uska failure mode Vanishing and Exploding Gradients. Bend aur starting weights saath milke choose karna hai Weight Initialization Strategies.