Exercises — Feedforward network — forward pass

${\mathbf{W}}^{(\ell )}$ pichli layer ke activation (length $n_{\ell -1}$) ko is layer (length $n_{\ell }$) mein map karta hai. Toh iske $n_{\ell }$ rows aur $n_{\ell -1}$ columns hain.

• ${\mathbf{W}}^{(1)}$: layer 1 mein $n_1=6$ neurons hain, layer 0 mein $n_0=4$ inputs hain $\Rightarrow$ shape $6\times 4$.
• ${\mathbf{b}}^{(1)}$: layer 1 mein har neuron ke liye ek bias $\Rightarrow$ length $6$.

Multiplication check karo: $(6\times 4)\cdot (4\times 1)=(6\times 1)$, aur ${\mathbf{b}}^{(1)}$ jis ki length $6$ hai use add karna kaam karta hai. ✓

ReLU positives ko unchanged pass karta hai aur har cheez $\le 0$ ko $0$ par clamp karta hai — Figure 1 mein orange curve trace karo: kink se left flat hai, uske right ek straight $45°$ ramp hai.

• $\text{ReLU}(-3.2)=\max (0,-3.2)=0$
• $\text{ReLU}(0)=\max (0,0)=0$
• $\text{ReLU}(4.5)=\max (0,4.5)=4.5$

Step (a) pre-activation — ek single dot product (row · column) plus bias: $z=0.5(2)+1.0(-1)+(-0.5)(3)+0.4=1.0-1.0-1.5+0.4=-1.1$ Step (b) activation: $a=\text{ReLU}(-1.1)=\max (0,-1.1)=0$ $0$ kyun? Weighted evidence net negative nikla, aur kink se left (Figure 1) ReLU ka output zero par flat hai. Neuron "fire nahi kiya" — iska matlab hai ki usne ek aise pattern ko detect kiya jo is input ke liye absent hai.

Sigmoid element-by-element apply hota hai: $\sigma (z)=\frac{1}{1+e^{-z}}$.

• $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.500$
• $\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.13534}=0.881$

${\mathbf{a}}^{(1)}=\left[\begin{array}{c}0.500\\ 0.881\end{array}\right]$ Yeh values kyun? $z=0$ par term $e^{-z}=1$ hai, toh top aur bottom balance hote hain aur hum S-curve ke midpoint par exactly baith jaate hain, $0.5$ — yahi wajah hai ki sigmoid $z=0$ par centred hota hai. Jaise $z$ badhta hai, $e^{-z}$ shrink hokar $0$ ki taraf jata hai, toh fraction $1$ ki taraf climb karta hai (par kabhi reach nahi karta); isliye $\sigma (2)=0.881$ high hai lekin ceiling se neeche hai. Dono outputs $(0,1)$ ke andar aate hain — woh "squashing" jo Figure 1 mein blue curve karte dikhai deta hai.

Layer 1 pre-activation (${\mathbf{W}}^{(1)}$ ki har row ko $\mathbf{x}$ se dot karo): ${\mathbf{z}}^{(1)}=\left[\begin{array}{c}1(1)+0(2)\\ -1(1)+1(2)\end{array}\right]=\left[\begin{array}{c}1\\ 1\end{array}\right]$ Layer 1 activation (ReLU): dono entries positive hain, toh hum kink ke ramp side par hain aur ReLU unhe unchanged chhod deta hai: ${\mathbf{a}}^{(1)}=\left[\begin{array}{c}1\\ 1\end{array}\right]$. Layer 2 pre-activation: $z^{(2)}=2(1)+3(1)+1=6$ Layer 2 activation (linear): $y=z^{(2)}=6$. End mein linear kyun? Ek regression output ko koi bhi real value lene ki freedom honi chahiye; identity (Figure 1 mein gray dashed line) koi squashing apply nahi karta, toh raw score $6$ seedha prediction ke roop mein pass ho jaata hai.

${\mathbf{a}}^{(1)}$ ko $\mathbf{y}$ mein substitute karo: $\mathbf{y}={\mathbf{W}}^{(2)}({\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)})+{\mathbf{b}}^{(2)}=\underset{{\mathbf{W}}_{\text{eff}}}{\underbrace{{\mathbf{W}}^{(2)}{\mathbf{W}}^{(1)}}}\mathbf{x}+\underset{{\mathbf{b}}_{\text{eff}}}{\underbrace{{\mathbf{W}}^{(2)}{\mathbf{b}}^{(1)}+{\mathbf{b}}^{(2)}}}$ Toh ${\mathbf{W}}_{\text{eff}}={\mathbf{W}}^{(2)}{\mathbf{W}}^{(1)}$ aur ${\mathbf{b}}_{\text{eff}}={\mathbf{W}}^{(2)}{\mathbf{b}}^{(1)}+{\mathbf{b}}^{(2)}$. Matlab: linear maps stack karne se ek linear map milta hai — extra layer ne hume kuch nahi diya. Yahi wajah hai ki nonlinearity $\sigma$ essential hai.

Layer by layer: ${\mathbf{a}}^{(1)}=\left[\begin{array}{c}1(1)+2(1)+1\\ 0(1)+1(1)+0\end{array}\right]=\left[\begin{array}{c}4\\ 1\end{array}\right]$, phir $y=1(4)+1(1)+0=5$. Collapsed: ${\mathbf{W}}_{\text{eff}}={\mathbf{W}}^{(2)}{\mathbf{W}}^{(1)}=\left[\begin{array}{cc}1& 1\end{array}\right]\left[\begin{array}{cc}1& 2\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}1& 3\end{array}\right]$; $b_{\text{eff}}=\left[\begin{array}{cc}1& 1\end{array}\right]\left[\begin{array}{c}1\\ 0\end{array}\right]+0=1$. Phir $y=1(1)+3(1)+1=5$. ✓ Same answer.

Neuron $\text{ReLU}(2x-4)$ output karta hai. Yeh dead hota hai jab bhi $2x-4\le 0$, yaani $x\le 2$.

• Dead region: $x\le 2$ (output exactly $0$).
• Active region: $x>2$, jahan output $=2x-4$ slope $2$ se badhta hai.
• Kink (corner) $x=2$ par hai, jahan $z=0$.

Yeh wahi corner hai jo Figure 1 mein orange curve par dikhti hai, bas shifted: ReLU ke andar linear part kink ko $z=0$ se $x=2$ par shift kar deta hai. Woh hinge hi woh tarika hai jisse networks curved aerospace dynamics ke piecewise-linear approximations banate hain.

Step 1 — har logit ko exponentiate karo: $e^2=7.389,e^0=1.000,e^{-1}=0.368$ Step 2 — sum: $S=7.389+1.000+0.368=8.757$. Step 3 — divide karo: $\mathbf{p}=\left[\begin{array}{c}7.389/8.757\\ 1.000/8.757\\ 0.368/8.757\end{array}\right]=\left[\begin{array}{c}0.844\\ 0.114\\ 0.042\end{array}\right]$ Pehle exponentiate kyun? Raw logits negative ho sakte hain, lekin probabilities nahi ho sakti — $e^z$ har score ko ek positive number mein map karta hai aur saath mein order preserve karta hai (bada logit → bada $e^z$). Sum se divide karne par total $1.000$ ho jaata hai ✓. Sabse bada class 0 hai, jise index→name table Nominal (probability $0.844$) padhta hai.

Layer 1 pre-activation (teen dot products): ${\mathbf{z}}^{(1)}=\left[\begin{array}{c}1(1)+1(-2)+0\\ 0(1)+(-1)(-2)+1\\ 2(1)+0(-2)-1\end{array}\right]=\left[\begin{array}{c}-1\\ 3\\ 1\end{array}\right]$ Layer 1 activation (tanh, element-wise): ${\mathbf{a}}^{(1)}=\left[\begin{array}{c}\tanh (-1)\\ \tanh (3)\\ \tanh (1)\end{array}\right]=\left[\begin{array}{c}-0.762\\ 0.995\\ 0.762\end{array}\right]$ $\tanh (3)$ almost $1$ kyun hai? tanh saturate karta hai: jab $z$ zero se kuch units dur ho jaata hai tab Figure 1 mein green curve apni $+1$ ceiling ke against already flatten ho chuki hoti hai, toh $\tanh (3)=0.995$ nearly maxed out hai, jabki $\tanh (-1)$ aur $\tanh (1)$ abhi bhi sloped middle par hain aur symmetric rehte hain ($\mp 0.762$). Layer 2 pre-activation: $z_1^{(2)}=1(-0.762)+0(0.995)+1(0.762)=0.000$ $z_2^{(2)}=0(-0.762)+1(0.995)-1(0.762)=0.233$ Softmax: $e^0=1.000,e^{0.233}=1.263$; sum $=2.263$. $\mathbf{p}=\left[\begin{array}{c}1.000/2.263\\ 1.263/2.263\end{array}\right]=\left[\begin{array}{c}0.442\\ 0.558\end{array}\right]$ Coin-flip ke itna kareeb kyun? Dono logits sirf $0.233$ se differ karte hain; kyunki $e^z$ chhote gaps ke liye ek gentle multiplier hai, ek chhota sa logit lead sirf ek modest probability lead ban jaata hai. Predicted class: 1 = Degraded (probability $0.558$).

Pura point yahi hai: ek matrix multiply poore batch ko handle kar leti hai. $\mathbf{Z}=\mathbf{WX}+\mathbf{b}$ compute karo (bias har column mein broadcast hota hai). $\mathbf{WX}=\left[\begin{array}{cc}1& -1\\ 1& 1\end{array}\right]\left[\begin{array}{cc}1& 0\\ 0& 2\end{array}\right]=\left[\begin{array}{cc}1& -2\\ 1& 2\end{array}\right]$ $\mathbf{b}=\left[\begin{array}{c}1\\ -1\end{array}\right]$ har column mein add karo: $\mathbf{Z}=\left[\begin{array}{cc}1+1& -2+1\\ 1-1& 2-1\end{array}\right]=\left[\begin{array}{cc}2& -1\\ 0& 1\end{array}\right]$ ReLU element-wise apply karo: $\mathbf{A}=\left[\begin{array}{cc}2& 0\\ 0& 1\end{array}\right]$ Dono samples ke liye ek multiply kyun? Har column usi $\mathbf{W}$ se independently process hota hai, toh inputs ko side-by-side stack karne se hardware dono passes ek single matrix product mein compute kar leta hai — exactly yahi tarika hai jisse real inference ek saath sensor frames ka poora batch run karta hai. Column 1 sample 1 ka activation hai $\left[\begin{array}{c}2\\ 0\end{array}\right]$; column 2 sample 2 ka $\left[\begin{array}{c}0\\ 1\end{array}\right]$ hai.

Claim: softmax invariant hai jab har logit mein constant $c$ add karo, kyunki $\frac{e^{z_i+c}}{{\sum }_j{e}^{z_j+c}}=\frac{e^c{e}^{z_i}}{e^c{\sum }_j{e}^{z_j}}=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}.$ $e^c$ top aur bottom cancel ho jaata hai. Toh hum $c=-{\max }_j{z}_j=-1002$ pick kar sakte hain taaki exponents $\le 0$ rahein (koi overflow nahi). Shifted logits: $\left[\begin{array}{c}-2\\ -1\\ 0\end{array}\right]$. Phir $e^{-2}=0.135,e^{-1}=0.368,e^0=1.000$; sum $=1.503$. $\mathbf{p}=\left[\begin{array}{c}0.135/1.503\\ 0.368/1.503\\ 1.000/1.503\end{array}\right]=\left[\begin{array}{c}0.090\\ 0.245\\ 0.665\end{array}\right]$ Yeh safe aur exact kyun hai: upar wala cancellation algebraic hai, approximation nahi — probabilities literally unchanged hain, lekin har exponent ab $\le 0$ hai toh $e^z\in (0,1]$ hai aur kabhi overflow nahi kar sakta.

Step 1 — hidden pre-activations (${\mathbf{W}}^{(1)}$ ki har row ko $\mathbf{x}$ se dot karo, plus bias): $z_1=-1(1)-1(2)-1=-4,z_2=-2(1)+0(2)-3=-5$ Step 2 — activation (ReLU): dono $\le 0$ hain, toh hum orange curve ke flat left side par hain (Figure 1) aur har ek zero par clamp hota hai: ${\mathbf{a}}^{(1)}=\left[\begin{array}{c}\max (0,-4)\\ \max (0,-5)\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$ Step 3 — output. Jab har hidden activation zero ho, toh output pre-activation hai $z^{(2)}={\mathbf{W}}^{(2)}\left[\begin{array}{c}0\\ 0\end{array}\right]+b^{(2)}={\mathbf{W}}^{(2)}0+b^{(2)}=b^{(2)}$ Weighted-sum term ${\mathbf{W}}^{(2)}0$ zero vector hai chahe ${\mathbf{W}}^{(2)}$ mein koi bhi numbers hon, toh prediction sirf output bias ke barabar hoti hai. Yeh kyun matter karta hai ("dying ReLU" failure): zyada negative biases har neuron ko uske kink se neeche push kar dete hain; jab ek neuron saare realistic inputs ke liye $0$ output karta hai toh woh stuck ho jaata hai (uske weights output ko influence karna band kar dete hain), aur agar puri layer mar jaaye toh network ek constant $b^{(2)}$ par freeze ho jaata hai aur apne sensors ko bilkul ignore karta hai. Fix in practice: careful bias initialisation, input normalisation, ya leaky-ReLU jo negative side par ek small slope rakhta hai.