Feedforward network — forward pass: Worked examples

5.6.7 · D3 · Coding › Machine Learning (Aerospace Applications) › Feedforward network — forward pass

Shuru karne se pehle, plain-word reminders (neeche koi bhi symbol undefined nahi chhoda gaya):

Definition Machine ke parts (saral shabdon mein)

Ek weight matrix $W$ numbers ki ek rectangular grid hai. Row $i$ , column $j$ mein $w_{ij}$ hota hai: input $j$ output $i$ par kitna strongly push karta hai. Boldface hamesha poore vector ya matrix ka matlab hai; ek plain italic letter jaise $b^{(2)}$ ek single number hai.
Ek pre-activation $z$ har neuron ka raw score hai kisi bhi squashing se pehle: $z = Wa + b$ . Yahan $a$ incoming activations ka column hai, $b$ bias column hai (har neuron mein add hone wala constant).
Ek activation vector $a$ woh column hai jo ek layer se squashing ke baad nikalta hai: $a = σ (z)$ , matlab "curve $σ$ ko $z$ ki har entry par apply karo". Input layer ka activation bas input khud hota hai, $a^{(0)} = x$ ; yahi woh $a$ hai jo pehle $z = Wa + b$ ko feed karta hai.
Output $y$ (ya column $y$ ) last layer ka activation hai — network ki actual prediction. Neeche har example is $y$ ko name deke khatam hota hai.

Hum chaar activation curves use karenge. Har ek alag sawaal ka jawaab deta hai, isliye hum batate hain kyun hum use uthate hain — aur har ek ko plot kiya gaya hai taaki tum uska shape dekh sako computation se pehle:

Formula Chaar curves aur har ek kis liye hai

ReLU: $ReLU (z) = max (0, z)$ . Positive scores rakhta hai, negatives delete karta hai. Ise use karo jab neurons "fire karein ya chup rahein" aur hidden values non-negative aur sasta rakhna ho.
Sigmoid: $σ (z) = \frac{1}{1 + e ^{- z}}$ . Kisi bhi real number ko $(0, 1)$ mein squash karta hai. Ise use karo jab output ko ek single probability ki tarah padhna ho.
tanh: $tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$ . $(- 1, 1)$ mein squash karta hai, $0$ par centred. Ise hidden layers mein use karo jab tum chahte ho ki positive aur negative dono activations zero ke around balanced hon.
Softmax: poore vector $z$ ke liye, entry $k$ ban jaata hai $\frac{e ^{z_{k}}}{\sum _{m} e ^{z_{m}}}$ . Ise final layer par use karo taaki kai scores ko aise probabilities mein convert kiya ja sake jo $1$ tak sum hon — ek probability per class.

Neeche ki figure sab chaar ko draw karti hai taaki unka behaviour algebra se pehle visual ho — notice karo ReLU ka hard elbow $0$ par, sigmoid ki smooth climb $0$ se $1$ tak, aur tanh ki S-shape $- 1$ se $+ 1$ tak:

Figure — Feedforward network — forward pass

Aage badhne se pehle plot padho: yellow ReLU flat (dead) hai $z < 0$ ke liye phir $4 5^{\circ}$ ka ramp; blue sigmoid left mein $0$ ke paas chipka hai aur right mein $1$ ke paas; pink tanh bhi wahi karta hai lekin $- 1$ aur $+ 1$ ke beech. Ye teen shapes is page ke har "kyun ye neuron mar gaya / saturate ho gaya / negative ho gaya" moment ko explain karti hain.

$e$ yahaan aata hai: $e \approx 2.71828$ woh number hai jiska exponential curve $e^{z}$ apni khud ki height ke barabar rate se badhta hai. Hum ise sigmoid/tanh/softmax mein uthate hain kyunki yeh "smooth squashing" ki natural currency hai — yeh baad mein backprop ke dauran derivatives ko clean banata hai. Haath ke kaam ke liye bas itna chahiye: $e^{0} = 1$ , $e^{1} \approx 2.718$ , $e^{- 1} \approx 0.368$ .

The scenario matrix

Neeche har cell ek alag cheez hai jo forward pass se puchhi ja sakti hai. Jo examples follow karte hain woh inhe sab cover karte hain.

Cell	Scenario	Covered by
A	Positive-only inputs, ReLU hidden, linear output (plain regression)	Ex 1
B	$z$ mein sign mix — kuch neurons fire karte hain, kuch mar jaate hain (ReLU zeroing)	Ex 2
C	Zero input vector $x = 0$ (degenerate) — bias-only output	Ex 3
D	Huge input → squashing curve ka saturation (limiting behaviour)	Ex 4
E	Sigmoid output ko probability ke roop mein padhna + decision threshold	Ex 5
F	tanh hidden layer jo signed activations produce karta hai	Ex 6
G	Softmax multi-class: raw scores → probabilities jo 1 tak sum hoon	Ex 7
H	Real-world word problem (stall-warning autopilot) end-to-end	Ex 8
I	Exam twist: "collapse" — koi nonlinearity nahi wala network sirf ek affine map hai	Ex 9

Forecast: Guess karo ki kya dono hidden neurons ReLU se bachenge, phir aage padhne se pehle $y$ guess karo.

Pre-activation, layer 1. $z^{(1)} = W^{(1)} a^{(0)} + b^{(1)}$ jahan $a^{(0)} = x$ , deta hai $[1 \cdot 2 + 0 \cdot 3 0 \cdot 2 + 2 \cdot 3] + [- 1 - 1] = [15]$ . Yeh step kyun? Har neuron apni weight row ka incoming activation ke saath dot product hota hai, phir bias add hota hai — scores machine mein daakhil hone ka yahi ek raasta hai.
Activate. $a^{(1)} = σ (z^{(1)})$ jahan $σ = ReLU$ . Dono scores positive hain, isliye ReLU unhe chhod deta hai: $a^{(1)} = [15]$ . Yeh step kyun? ReLU sirf negatives delete karta hai; positives unchanged pass hote hain, isliye koi neuron yahaan nahi marata — intro figure mein yellow curve par waapas dekho, dono points uske rising ramp par baithe hain.
Output layer. $z^{(2)} = 1 \cdot 1 + 1 \cdot 5 + 0 = 6$ . Linear output ⇒ last activation $y = a^{(2)} = z^{(2)} = 6$ hai. Yeh step kyun? Linear output prediction ko unbounded rakhta hai — ek regression number ke liye sahi.

Figure kya dikhata hai: do blue input nodes dono yellow hidden nodes ko feed karte hain (dono alive, values $1$ aur $5$ ), jo single pink output node ko feed karte hain jisme $y = 6$ hai — "dot, add, squash" do baar hone ki picture.

Verify: Full sum se recompute karo: $y = (2 - 1) + (6 - 1) = 1 + 5 = 6$ . ✓ Units: jo bhi target hai (jaise N of thrust) — ek single real number, jaise ek regression head dena chahiye.

Forecast: Ek neuron ReLU se pehle negative hoga. Kaun sa, aur kya output mein uska weight abhi bhi matter karta hai?

Pre-activation. $z_{1}^{(1)} = 1 \cdot 2 + 0 \cdot 7 = 2$ , $z_{2}^{(1)} = - 1 \cdot 2 + 0 \cdot 7 = - 2$ . Toh $z^{(1)} = [2 - 2]$ . Yeh step kyun? Doosri row mein feature 1 par negative weight hai, isliye woh negative score produce karta hai — ReLU ke liye ise zero karne ka classic setup.
Activate. $a^{(1)} = ReLU (z^{(1)})$ : $ReLU (2) = 2$ , $ReLU (- 2) = 0$ , toh $a^{(1)} = [20]$ . Neuron 2 is input ke liye dead hai. Yeh step kyun? ReLU har negative ko exactly $0$ set karta hai — ek hard cliff, yellow curve ke flat left part ke roop mein dikhta hai.
Output. $y = 3 \cdot 2 + 3 \cdot 0 + 1 = 7$ . Dead neuron ka weight $3$ , $0$ se multiply hota hai, kuch contribute nahi karta. Yeh step kyun? Ek silent neuron aage koi information carry nahi karta chahe uska outgoing weight kitna bhi bada ho — isliye ReLU sparse activity create karta hai.

Figure kya dikhata hai: same 2-2-1 graph, lekin doosra hidden node value $0$ par greyed out hai — uska outgoing edge faint draw kiya gaya hai yeh signal karne ke liye "koi signal pass nahi hota", isliye $y$ sirf neuron 1 aur bias se banta hai.

Verify: $y = 6 + 0 + 1 = 7$ . ✓ Sanity: agar hum neuron 2 ko poori tarah hata dein toh answer same hoga — confirm karta hai ki yeh yahaan inactive hai.

Forecast: Sab inputs zero hone par, kya bachta hai — aur output kya read karta hai?

Pre-activation. $W0 = 0$ kisi bhi weights ke liye, toh $z = 0 + b = b$ ; pehle do entries hain $[0.5 - 0.5]$ . Yeh step kyun? Kisi grid ko zero column se multiply karne par zero column milta hai — input kuch contribute nahi karta, machine sirf bias par depend karti hai. Yahi bias ka the matlab hai: output jab input kuch nahi kehta.
Activate (sigmoid). $a = σ (z)$ : $σ (0.5) = \frac{1}{1 + e ^{- 0.5}}$ , $σ (- 0.5) = \frac{1}{1 + e ^{0.5}}$ . Numerically $e^{- 0.5} \approx 0.6065$ toh $σ (0.5) \approx 0.6225$ ; $e^{0.5} \approx 1.6487$ toh $σ (- 0.5) \approx 0.3775$ . Kyunki yeh last layer hai, ye outputs $y_{1}, y_{2}$ hain. Yeh step kyun? Sigmoid har bias ko $(0, 1)$ mein squash karta hai; positive bias $0.5$ se upar land karta hai, negative bias $0.5$ se neeche — intro figure mein blue curve se $z = \pm 0.5$ par padho.

Verify: Note karo $σ (0.5) + σ (- 0.5) = 0.6225 + 0.3775 = 1.000$ kyunki sigmoid symmetric hai: $σ (z) + σ (- z) = 1$ . ✓

Forecast: Compute karne se pehle $σ (20)$ aur $σ (- 20)$ do decimals tak guess karo.

$σ (20)$ compute karo. $e^{- 20} \approx 2 \times 1 0^{- 9}$ , toh $y = σ (20) = \frac{1}{1 + 2 \times 1 0 ^{- 9}} \approx 0.999999998$ . Yeh step kyun? Bada positive $z$ , $e^{- z}$ ko vanish kar deta hai, output ko top rail $1$ par pin kar deta hai.
$σ (- 20)$ compute karo. $e^{20} \approx 4.85 \times 1 0^{8}$ , toh $y = σ (- 20) = \frac{1}{1 + 4.85 \times 1 0 ^{8}} \approx 2 \times 1 0^{- 9}$ . Yeh step kyun? Bada negative $z$ , $e^{- z}$ ko explode kar deta hai, output ko bottom rail $0$ par pin kar deta hai.

Figure kya dikhata hai: blue sigmoid jisme $z = \pm 20$ par do marked dots hain jo dashed rails $y = 1$ aur $y = 0$ ke flat against baithe hain — curve wahan itna flat hai ki $z$ move karne se $y$ barely move karta hai, jo exactly saturation ka danger hai.

Limiting behaviour: jaise $z \to + \infty$ , $σ \to 1$ ; jaise $z \to - \infty$ , $σ \to 0$ . Flat rails mein curve barely change karta hai, isliye ek baar inputs huge hone ke baad network insensitive ho jaata hai — isliye hum inputs normalize karte hain.

Verify: $σ (20) \approx 1$ aur $σ (- 20) \approx 0$ 6 decimals tak; aur $σ (20) + σ (- 20) \approx 1$ . ✓

Forecast: $z$ compute karo; kya yeh $0$ se upar hai ya neeche (sigmoid ki $0.5$ line)?

Pre-activation. $z = 0.4 \cdot 1 + (- 0.1) \cdot 2 + 0.3 = 0.4 - 0.2 + 0.3 = 0.5$ . Yeh step kyun? Ek dot product plus bias single decision score deta hai.
Sigmoid. $y = σ (0.5) = \frac{1}{1 + e ^{- 0.5}} \approx 0.6225$ . Yeh step kyun? Sigmoid score ko probability mein convert karta hai taaki hum use threshold kar sakein.
Threshold. $0.6225 > 0.5 \Rightarrow$ predict fault. Yeh step kyun? $σ (z) > 0.5$ exactly tab hota hai jab $z > 0$ ; yahaan $z = 0.5 > 0$ , consistent hai.

Verify: Kyunki $z = 0.5 > 0$ hai, sigmoid $0.5$ se upar hona chahiye; wakai $0.6225 > 0.5$ . ✓ Output $y$ probability ki tarah $(0, 1)$ mein hai.

Forecast: Ek neuron ka score $0$ hoga, doosra nonzero. Har ek ka tanh predict karo.

Pre-activation. $z_{1} = 1 \cdot 1 + 1 \cdot (- 1) = 0$ ; $z_{2} = 1 \cdot 1 + (- 1) \cdot (- 1) = 2$ . Yeh step kyun? Neuron 1 ek cancelling pair add karta hai → exactly $0$ ; neuron 2 reinforce karta hai → $2$ .
tanh. $a^{(1)} = tanh (z^{(1)})$ : $tanh (0) = 0$ ; $tanh (2) = \frac{e ^{2} - e ^{- 2}}{e ^{2} + e ^{- 2}} \approx \frac{7.389 - 0.135}{7.389 + 0.135} \approx 0.964$ . Yeh step kyun? tanh zero ko seedha pass karta hai aur positive score ke liye signed value near $+ 1$ deta hai — uska output range $(- 1, 1)$ hidden units ko negative hone deta hai, unlike ReLU.

Figure kya dikhata hai: pink tanh curve jisme $z = 0$ par dot hai (0 par land karta hai) aur $z = 2$ par (near $0.964$ par land karta hai), dashed $\pm 1$ rails ke beech — proof ki tanh negative values output kar sakta hai, woh feature jo ReLU mein nahi hai.

Verify: $tanh (0) = 0$ exactly; $tanh (2) \approx 0.9640$ . ✓ Dono $(- 1, 1)$ mein hain.

Forecast: Kaun si class jeetegi, aur kya teenon probabilities exactly $1$ tak add honge?

Har score ko exponentiate karo. $e^{2} \approx 7.389$ , $e^{1} \approx 2.718$ , $e^{- 1} \approx 0.368$ . Yeh step kyun? $e^{z}$ hamesha positive hota hai aur bade scores ko bahut bade numbers mein convert karta hai — softmax ka "soft argmax" banana ka tarika.
Sum. $S = 7.389 + 2.718 + 0.368 = 10.475$ . Yeh step kyun? Total se divide karne par outputs $1$ tak sum karte hain, toh woh probabilities ki tarah read hote hain.
Divide karo. $y_{1} = 7.389/10.475 \approx 0.7054$ , $y_{2} = 2.718/10.475 \approx 0.2595$ , $y_{3} = 0.368/10.475 \approx 0.0351$ . Yeh step kyun? Har ratio us class ka total exponential mass mein share hai.

Figure kya dikhata hai: teen chalk bars, Nominal ke liye sabse bada ( $0.7054$ ), jo largest raw score $2$ se match karta hai — softmax ke ranked scores ko ek probability bar chart mein convert karne ki picture jo $1$ tak sum hoti hai.

Verify: $0.7054 + 0.2595 + 0.0351 = 1.0000$ . ✓ Argmax class 1 (Nominal) hai, jo largest raw score $2$ se match karta hai. ✓

Forecast: High $α$ stall warning ki taraf push karna chahiye. Kya output $0.5$ se upar jayega?

Hidden pre-activation. $z_{1} = 0.1 \cdot 12 + 0.5 \cdot (- 1) + 0.2 \cdot 0.5 - 1 = 1.2 - 0.5 + 0.1 - 1 = - 0.2$ . $z_{2} = - 0.2 \cdot 12 + 0.1 \cdot (- 1) + 0.3 \cdot 0.5 + 0 = - 2.4 - 0.1 + 0.15 = - 2.35$ . Yeh step kyun? Do dot products sensors se hidden scores banate hain.
ReLU. $a^{(1)} = ReLU (z^{(1)})$ : $ReLU (- 0.2) = 0$ , $ReLU (- 2.35) = 0$ , toh $a^{(1)} = [00]$ — dono hidden neurons mar jaate hain. Yeh step kyun? Dono scores negative hain; ReLU dono ko zero karta hai, ek fully degenerate hidden layer.
Output. $z^{(2)} = 2 \cdot 0 + 1 \cdot 0 - 0.5 = - 0.5$ ; $y = σ (- 0.5) \approx 0.3775$ . Yeh step kyun? Hidden layer silent hone se, sirf output bias bolta hai — aur woh kehta hai "no warning".
Decision. $0.3775 < 0.5 \Rightarrow$ no stall warning.

Verify: $σ (- 0.5) \approx 0.3775 < 0.5$ . ✓ Sanity: kyunki $z^{(2)} = - 0.5 < 0$ hai, sigmoid $0.5$ se neeche girna chahiye — consistent. (Lesson: raw $α = 12$ yahaan theek tha; yahi net bade $α$ ke liye fire karta hai, dikhata hai ki dead-neuron case "hamesha no" nahi hota.)

Forecast: Do stacked linear maps — single equivalent slope aur intercept guess karo.

Layer 1. $a^{(1)} = 2 x + 1$ (identity ise linear chhod deta hai). Yeh step kyun? Koi squashing nahi hone se, activation pre-activation ke barabar hoti hai.
Layer 2. $y = 3 a^{(1)} - 4 = 3 (2 x + 1) - 4 = 6 x + 3 - 4 = 6 x - 1$ . Yeh step kyun? Substitute karne par dikhta hai ki pura network single affine map $y = 6 x - 1$ hai — prove karta hai ki stacked linear layers kuch haasil nahi karte. Yahi wajah hai ki real networks ko nonlinear $σ$ chahiye.
Evaluate karo. $x = 5$ par: $y = 6 \cdot 5 - 1 = 30 - 1 = 29$ . Yeh step kyun? Collapsed line mein plug karne par wahi answer milta hai jo dono layers chalane par milta — neeche ki check confirm karti hai.

Verify: Direct two-step: $a^{(1)} = 2 \cdot 5 + 1 = 11$ , phir $y = 3 \cdot 11 - 4 = 33 - 4 = 29$ . ✓ Collapsed line $6 x - 1$ se $x = 5$ par match karta hai. ✓

Recall Quick self-check

Negative pre-activation ka ReLU kya hota hai? ::: Exactly $0$ — neuron silent hai. Activation vector $a$ , $z$ ke terms mein kaise define hota hai? ::: $a = σ (z)$ — curve ko $z$ ki har entry par apply karo. Input $x = 0$ hone par, pre-activation kya hota hai? ::: Bias $b$ (kyunki $W0 = 0$ ). Softmax outputs hamesha kaun si ek property satisfy karte hain? ::: Woh positive hain aur exactly $1$ tak sum karte hain. Do stacked linear layers kis cheez ke equivalent hain? ::: Ek single affine map $W_{eff} x + b_{eff}$ .