Feedforward network — forward pass

5.6.7 · Coding › Machine Learning (Aerospace Applications)

"Forward" kyun? Kyunki hum input se output ki taraf jaate hain (prediction direction). Baad mein, backpropagation gradients compute karne ke liye backward jayega. Forward pass woh tarika hai jisse tumhara trained model actually production mein inference run karta hai—yeh waise hi hai jaise airplane ka autopilot sensor readings se control commands compute karta hai.

The Architecture — Hum Kya Compute Kar Rahe Hain?

Layer $\ell -1$ ke neuron $j$ se layer $\ell$ ke neuron $i$ tak har connection ka ek weight $w_{ij}^{(\ell )}$ hota hai. Layer $\ell$ ke har neuron $i$ ka ek bias $b_i^{(\ell )}$ hota hai.

Key insight: Ek neuron apne inputs ka weighted sum compute karta hai, bias add karta hai, phir ek nonlinear activation function apply karta hai. Nonlinearity ke bina, layers ko stack karna ek single linear transformation mein collapse ho jaata (complex aerospace dynamics ke liye useless).

The Forward Pass Algorithm — Step by Step Derivation

Starting Point: Hamare Paas Kya Hai?

Inputs:

• Input vector $\mathbf{x}\in {\mathbb{R}}^{n_0}$ (sensor readings)
• Weight matrices ${\mathbf{W}}^{(1)},{\mathbf{W}}^{(2)},\dots ,{\mathbf{W}}^{(L)}$ jahan ${\mathbf{W}}^{(\ell )}\in {\mathbb{R}}^{n_{\ell }\times n_{\ell -1}}$
• Bias vectors ${\mathbf{b}}^{(1)},{\mathbf{b}}^{(2)},\dots ,{\mathbf{b}}^{(L)}$ jahan ${\mathbf{b}}^{(\ell )}\in {\mathbb{R}}^{n_{\ell }}$
• Activation functions ${\sigma }^{(1)},{\sigma }^{(2)},\dots ,{\sigma }^{(L)}$ (aksar ReLU, sigmoid, tanh, ya linear)

Goal: Output $\mathbf{y}={\mathbf{a}}^{(L)}$ compute karna.

Layer-by-Layer Propagation

Step 1: Initialize ${\mathbf{a}}^{(0)}=\mathbf{x}$

Kyun? Layer 0 ki "activation" bas input data hi hai. Hum ise ${\mathbf{a}}^{(0)}$ denote karte hain taaki recursion uniform rahe.

Step 2: Har layer $\ell =1,2,\dots ,L$ ke liye:

(a) Pre-activation compute karo (weighted sum + bias)* ${\mathbf{z}}^{(\ell )}={\mathbf{W}}^{(\ell )}{\mathbf{a}}^{(\ell -1)}+{\mathbf{b}}^{(\ell )}$

Yeh form kyun?

• Layer $\ell$ ka har neuron $i$ compute karta hai $z_i^{(\ell )}={\sum }_{j=1}^{n_{\ell -1}}{w}_{ij}^{(\ell )}{a}_j^{(\ell -1)}+b_i^{(\ell )}$
• Matrix form mein: ${\mathbf{W}}^{(\ell )}$ ka shape $(n_{\ell },n_{\ell -1})$ hai, ${\mathbf{a}}^{(\ell -1)}$ $(n_{\ell -1},1)$ hai, toh product $(n_{\ell },1)$ hai
• Yeh previous layer ke output ka ek linear transformation hai

(b) Activation function element-wise apply karo ${\mathbf{a}}^{(\ell )}={\sigma }^{(\ell )}({\mathbf{z}}^{(\ell )})$

Activation kyun? $\sigma$ ke bina, poora network yeh ho jaata: ${\mathbf{a}}^{(L)}={\mathbf{W}}^{(L)}({\mathbf{W}}^{(L-1)}(\cdots ({\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)})+\cdots )+{\mathbf{b}}^{(L-1)})+{\mathbf{b}}^{(L)}$ jo collapse hokar ${\mathbf{a}}^{(L)}={\mathbf{W}}_{\text{eff}}\mathbf{x}+{\mathbf{b}}_{\text{eff}}$, ek single affine map, ban jaata. Aerospace systems nonlinear hote hain (stall, shock waves, compressibility)—hume unhe approximate karne ke liye nonlinearity chahiye.

Step 3: Output $\mathbf{y}={\mathbf{a}}^{(L)}$

Yahan kyun rokein? Final layer ki activation network ki prediction hai. Regression ke liye (continuous control surfaces predict karna), aksar ${\sigma }^{(L)}$ linear (identity) hoti hai. Classification ke liye (fault detection: nominal/critical), ${\sigma }^{(L)}$ softmax ho sakti hai.

Worked Examples — Concrete Calculations

Setup: 2 sensors se thrust adjustment predict karo (Mach number, altitude).

• Input: $\mathbf{x}=\left[\begin{array}{c}0.8\\ 10000\end{array}\right]$ (Mach 0.8, 10k ft)
• Hidden layer: 3 neurons, ReLU activation
• Output layer: 1 neuron, linear activation

Weights aur biases (pretrained): ${\mathbf{W}}^{(1)}=\left[\begin{array}{cc}0.5& 0.0001\\ -0.3& 0.00005\\ 0.2& -0.0001\end{array}\right],{\mathbf{b}}^{(1)}=\left[\begin{array}{c}0.1\\ -0.2\\ 0.0\end{array}\right]$ ${\mathbf{W}}^{(2)}=\left[\begin{array}{ccc}1.0& -0.5& 2.0\end{array}\right],{\mathbf{b}}^{(2)}=\left[\begin{array}{c}0.3\end{array}\right]$

Second column ke weights itne tiny kyun hain? Altitude feature (10000) Mach (0.8) ke mukable bahut bada hai. Altitude column par chhote weights contributions ko balanced rakhte hain—isliye practice mein training se pehle hum inputs normalize karte hain.

Forward Pass Computation:

Layer 1 (Hidden):

Step 2a: Pre-activation ${\mathbf{z}}^{(1)}={\mathbf{W}}^{(1)}{\mathbf{a}}^{(0)}+{\mathbf{b}}^{(1)}$ $=\left[\begin{array}{cc}0.5& 0.0001\\ -0.3& 0.00005\\ 0.2& -0.0001\end{array}\right]\left[\begin{array}{c}0.8\\ 10000\end{array}\right]+\left[\begin{array}{c}0.1\\ -0.2\\ 0.0\end{array}\right]$ $=\left[\begin{array}{c}0.5\cdot 0.8+0.0001\cdot 10000\\ -0.3\cdot 0.8+0.00005\cdot 10000\\ 0.2\cdot 0.8-0.0001\cdot 10000\end{array}\right]+\left[\begin{array}{c}0.1\\ -0.2\\ 0.0\end{array}\right]$ $=\left[\begin{array}{c}0.4+1.0\\ -0.24+0.5\\ 0.16-1.0\end{array}\right]+\left[\begin{array}{c}0.1\\ -0.2\\ 0.0\end{array}\right]=\left[\begin{array}{c}1.5\\ 0.06\\ -0.84\end{array}\right]$

Yeh step kyun? Har neuron input features se evidence aggregate karta hai. Neuron 1 ko strong signal milta hai (1.5), neuron 2 ko weak (0.06), neuron 3 negative (-0.84).

Step 2b: Activation (ReLU: $\max (0,z)$) ${\mathbf{a}}^{(1)}=\text{ReLU}({\mathbf{z}}^{(1)})=\left[\begin{array}{c}\max (0,1.5)\\ \max (0,0.06)\\ \max (0,-0.84)\end{array}\right]=\left[\begin{array}{c}1.5\\ 0.06\\ 0.0\end{array}\right]$

ReLU yahan kyun? Negative pre-activation ka matlab hai "yeh feature combination is neuron ko fire nahi karta." ReLU use zero kar deta hai, sparse representations create karta hai (bahut saare neurons off). Neuron 3 is input ke liye "dead" hai—usne ek aisa pattern detect kiya jo absent hai.

Layer 2 (Output):

Step 2a: Pre-activation ${\mathbf{z}}^{(2)}={\mathbf{W}}^{(2)}{\mathbf{a}}^{(1)}+{\mathbf{b}}^{(2)}$ $=\left[\begin{array}{ccc}1.0& -0.5& 2.0\end{array}\right]\left[\begin{array}{c}1.5\\ 0.06\\ 0.0\end{array}\right]+\left[\begin{array}{c}0.3\end{array}\right]$ $=[1.0\cdot 1.5-0.5\cdot 0.06+2.0\cdot 0]+0.3=1.5-0.03+0.3=1.77$

Yeh step kyun? Output neuron hidden features ko combine karta hai. Positive weights (1.0, 2.0) neurons 1 aur 3 ko amplify karte hain; negative weight (-0.5) neuron 2 ko suppress karta hai. Neuron 3 off hai, toh uska connection (2.0) kuch contribute nahi karta.

Step 2b: Activation (Linear: $\sigma (z)=z$) ${\mathbf{a}}^{(2)}={\mathbf{z}}^{(2)}=1.77$

Linear output kyun? Regression ke liye, hume unbounded predictions chahiye. Network thrust adjustment 1.77 N predict karta hai (ya jo bhi units training mein use hue the).

Setup: Fault detection—flight state ko 3 categories mein classify karo: 0=Nominal, 1=Degraded, 2=Critical.

• Input: $\mathbf{x}=\left[\begin{array}{c}250\\ 0.95\\ 35\end{array}\right]$ (airspeed m/s, throttle fraction, angle-of-attack deg)
• Hidden: 4 neurons, tanh activation
• Output: 3 neurons, softmax activation

Weights: ${\mathbf{W}}^{(1)}=\left[\begin{array}{ccc}0.01& 0.5& 0.02\\ -0.01& -0.3& 0.03\\ 0.005& 0.1& -0.05\\ 0.02& 0.2& 0.01\end{array}\right],{\mathbf{b}}^{(1)}=\left[\begin{array}{c}-2\\ 1\\ 0\\ -1\end{array}\right]$ ${\mathbf{W}}^{(2)}=\left[\begin{array}{cccc}1.5& -0.8& 0.3& 1.0\\ -1.0& 0.5& -0.2& -0.9\\ 0.2& 1.2& 0.9& -0.5\end{array}\right],{\mathbf{b}}^{(2)}=\left[\begin{array}{c}0\\ 0\\ 0\end{array}\right]$

Forward Pass:

Layer 1 — Pre-activation (${\mathbf{z}}^{(1)}={\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)}$ row by row compute karte hain):

• $z_1=0.01(250)+0.5(0.95)+0.02(35)-2=2.5+0.475+0.7-2=1.675$
• $z_2=-0.01(250)-0.3(0.95)+0.03(35)+1=-2.5-0.285+1.05+1=-0.735$
• $z_3=0.005(250)+0.1(0.95)-0.05(35)+0=1.25+0.095-1.75=-0.405$
• $z_4=0.02(250)+0.2(0.95)+0.01(35)-1=5.0+0.19+0.35-1=4.54$

${\mathbf{z}}^{(1)}=\left[\begin{array}{c}1.675\\ -0.735\\ -0.405\\ 4.54\end{array}\right]$

Har row check kyun karein? Matrix–vector product bas ek row per dot product hai. Inhe explicitly karne se sign errors pakad mein aate hain—jo bugs ka ek common source hai.

Step 2b: Activation (tanh) ${\mathbf{a}}^{(1)}=\tanh ({\mathbf{z}}^{(1)})\approx \left[\begin{array}{c}0.933\\ -0.626\\ -0.384\\ 0.9997\end{array}\right]$

Tanh kyun? Output range $(-1,1)$ activations ko zero ke around center karti hai, gradient flow mein help karta hai. Aerospace ke liye, negative activations "opposite conditions" encode kar sakti hain (e.g., pitch-up vs pitch-down). Notice karo neuron 4 $+1$ ke paas saturate ho jaata hai.

Layer 2 — Pre-activation (${\mathbf{z}}^{(2)}={\mathbf{W}}^{(2)}{\mathbf{a}}^{(1)}+{\mathbf{b}}^{(2)}$):

• $z_1=1.5(0.933)-0.8(-0.626)+0.3(-0.384)+1.0(0.9997)\approx 1.400+0.501-0.115+1.000=2.786$
• $z_2=-1.0(0.933)+0.5(-0.626)-0.2(-0.384)-0.9(0.9997)\approx -0.933-0.313+0.077-0.900=-2.069$
• $z_3=0.2(0.933)+1.2(-0.626)+0.9(-0.384)-0.5(0.9997)\approx 0.187-0.751-0.346-0.500=-1.410$

${\mathbf{z}}^{(2)}\approx \left[\begin{array}{c}2.786\\ -2.069\\ -1.410\end{array}\right]$

Softmax activation: ${\mathbf{a}}_i^{(2)}=\frac{e^{z_i^{(2)}}}{{\sum }_{j=1}^3{e}^{z_j^{(2)}}}$

$e^{{\mathbf{z}}^{(2)}}\approx \left[\begin{array}{c}{e}^{2.786}\\ e^{-2.069}\\ e^{-1.410}\end{array}\right]=\left[\begin{array}{c}16.22\\ 0.126\\ 0.244\end{array}\right]$

Sum $\approx 16.59$, toh: ${\mathbf{a}}^{(2)}\approx \left[\begin{array}{c}0.978\\ 0.008\\ 0.015\end{array}\right]$

Interpretation: Network predict karta hai 97.8% probability of Nominal flight state (class 0), 1.5% Critical, 0.8% Degraded. Argmax class 0 hai.

Softmax kyun? Yeh arbitrary scores (logits) ko ek valid probability distribution mein convert karta hai (sab milake 1, sab positive). Classification ke liye critical hai—pilot ko raw scores nahi, confidence levels chahiye.

Computational Complexity — Real-Time Systems Mein Kyun Matters Hai

Flops per layer: Layer $\ell$ ke liye:

• Matrix multiply ${\mathbf{W}}^{(\ell )}{\mathbf{a}}^{(\ell -1)}$: $n_{\ell }\times n_{\ell -1}$ multiplications + $n_{\ell }\times (n_{\ell -1}-1)$ additions $\approx 2n_{\ell }{n}_{\ell -1}$ flops
• Bias add: $n_{\ell }$ additions
• Activation: $n_{\ell }$ function evaluations (ReLU: 1 flop each; tanh: ~10 flops)

Total forward pass: ${\sum }_{\ell =1}^L(2n_{\ell }{n}_{\ell -1}+n_{\ell }\cdot c_{\sigma })$ flops, jahan $c_{\sigma }$ activation cost hai.

Example: Ek 10-100-100-5 network ke liye (10 inputs, DO hidden layers of 100, 5 outputs):

• Layer 1: $2\cdot 100\cdot 10+100=2100$ flops
• Layer 2: $2\cdot 100\cdot 100+100=20100$ flops
• Layer 3: $2\cdot 5\cdot 100+5=1005$ flops
• Total: ~23k flops

Aerospace mein yeh kyun matters hai: Flight control computers high frequency par run karte hain (100-1000 Hz). Ek 1 GFlop processor par 23k-flop network 23 microseconds leta hai—1 ms control loop ke andar aaram se. Lekin ek 1000-1000-1000-5 network (2M flops) deadlines miss kar sakta hai. Model size embedded systems ke liye ek design constraint hai.

Common Mistakes — Confusion Ko Steel-Manning Karna

Galat approach: Hamesha output par ReLU apply karo.

Yeh sahi kyun lagta hai: "Nonlinearity ke liye har layer ko activation chahiye."

Fix: Output activation task par depend karta hai:

• Regression (continuous values): Linear (identity) ya kabhi kabhi ReLU agar outputs non-negative hone chahiye (e.g., thrust ≥ 0)
• Binary classification: Sigmoid (outputs probability in [0,1])
• Multi-class classification: Softmax (outputs probability distribution)

Regression output par ReLU apply karna negative predictions ko zero tak clip kar dega—yeh catastrophic hai agar tumhe negative elevator deflection predict karni ho (nose-down command).

Galat approach: ${\mathbf{W}}^{(\ell )}$ ka shape $(n_{\ell -1},n_{\ell })$ hai (transposed).

Yeh sahi kyun lagta hai: "Weights previous layer (rows) se current layer (columns) tak connect karte hain."

Fix: Standard convention mein, ${\mathbf{W}}^{(\ell )}$ $(n_{\ell }\times n_{\ell -1})$ hai taaki ${\mathbf{W}}^{(\ell )}{\mathbf{a}}^{(\ell -1)}$ size $n_{\ell }$ ka column vector de. ${\mathbf{W}}^{(\ell )}$ ki har row mein layer $\ell$ ke ek neuron ke incoming weights hote hain.

Check: Agar ${\mathbf{a}}^{(\ell -1)}$ $(n_{\ell -1},1)$ hai aur ${\mathbf{z}}^{(\ell )}$ $(n_{\ell },1)$ hona chahiye, toh product valid hone ke liye ${\mathbf{W}}^{(\ell )}$ $(n_{\ell },n_{\ell -1})$ hona zaroori hai.

Galat approach: Backpropagation ke dauran, activations paane ke liye forward pass scratch se recompute karo.

Yeh sahi kyun lagta hai: "Mujhe gradient calculations ke liye ${\mathbf{a}}^{(\ell )}$ values chahiye."

Fix: Forward pass ke dauran ${\mathbf{z}}^{(\ell )}$ aur ${\mathbf{a}}^{(\ell )}$ cache karo. Backprop ko yeh chahiye hote hain (khaaskar ${\mathbf{z}}^{(\ell )}$ activation derivatives compute karne ke liye), aur recomputing cost double kar deta hai. Resource-constrained hardware par run karne wale aerospace applications mein, yeh memory-vs-compute tradeoff critical hai.

Implementation tip:

# Store in a dictionary during forward pass
cache = {}
cache[f'z{ell}'] = z
cache[f'a{ell}'] = a
# Access during backprop
z = cache[f'z{ell}']

Mnemonic & Recall

Yaad rakho: Har layer ke liye W·A → Z → σ → A. Yeh cycle $L$ baar repeat hoti hai.

Ek feedforward network ek row mein baithne wale decision-makers ki chain jaisi hai. Pehla aadmi (input layer) instruments ke raw numbers dekhta hai. Woh kuch simple math karta hai (weights se multiply karo, bias add karo) aur apne answers agli aadmi ko pass karta hai. Woh aadmi un answers par AUR math karta hai aur aage pass karta hai. Yeh kai logo (hidden layers) se hota rehta hai jab tak aakhri aadmi (output layer) final decision nahi chillata: "5 degrees upar jhukao!"

"Activation function" ek rule ki tarah hai: "Agar tumhara answer negative hai, toh bas zero keh do" (woh ReLU hai). Yeh math ko zyada interesting banata hai kyunki iske bina, chain mein woh saare log bas ek bada sa multiplication kar rahe hote, aur tumhe sirf ek hi aadmi ki zaroorat hoti.

"Forward pass" ka matlab hai hum shuruaat se

Concept Map