Multi-layer perceptron architecture

3.1.2 · AI-ML › Neural Network Fundamentals

What is an MLP?

Har layer mein multiple neurons (nodes) hote hain, aur layer $l$ ka har neuron layer $l+1$ ke har neuron se connected hota hai (fully connected). Information sirf forward direction mein flow karti hai—koi cycles nahi.

Why Multiple Layers?

Single-layer perceptrons sirf linearly separable functions hi seekh sakte hain. XOR problem ek famous example hai jo single layer se solve nahi ho sakti. Hidden layers add karne se network non-linear decision boundaries seekh sakta hai, simple transformations ko compose karke.

$z_j^{(l)}={\sum }_i{w}_{ji}^{(l)}{a}_i^{(l-1)}+b_j^{(l)}$

$a_j^{(l)}=\sigma (z_j^{(l)})$

Jahan:

• $a_i^{(l-1)}$ = previous layer se activations (is neuron ke inputs)
• $w_{ji}^{(l)}$ = layer $l-1$ ke neuron $i$ se layer $l$ ke neuron $j$ tak ka weight
• $b_j^{(l)}$ = neuron $j$ ke liye bias term
• $z_j^{(l)}$ = pre-activation (weighted sum)
• $a_j^{(l)}$ = activation (non-linearity apply karne ke baad)
• $\sigma$ = activation function (sigmoid, ReLU, tanh, etc.)

Yeh formula kyun? Har neuron apne inputs ka ek weighted vote karta hai, ek bias add karta hai (decision boundary ko shift karta hai), phir ek non-linear function apply karta hai. $\sigma$ ke bina, layers ko stack karna ek single linear transformation mein collapse ho jaata: $W_3(W_2(W_{1}x))=(W_3{W}_2{W}_1)x=W_{combined}x$. Non-linearity ise break karti hai aur complex patterns seekhna enable karti hai.

Architecture Components

Input Layer

• Purpose: Raw feature vector $\mathbf{x}=[x_1,x_2,\dots ,x_n{]}^T$ ko hold karta hai
• Koi computation nahi: Yeh "neurons" sirf values ko forward pass karte hain
• Size: Tumhare data se determine hoti hai (e.g., 28×28 pixel images ke liye 784, 100 features ke liye 100)

Hidden Layers

Har layer mein kitne neurons? Common heuristics:

• Input size ke 2-3x se shuru karo
• Computational efficiency ke liye powers of 2 use karo (32, 64, 128, 256)
• Zyada neurons = zyada capacity lekin overfitting ka risk
• Deeper (zyada layers) aksar same parameter count ke liye wider (zyada neurons per layer) se behtar hota hai

Layer output ki derivation (vectorized form):

$m$ neurons wali poori layer $l$ ke liye:

${\mathbf{z}}^{(l)}=W(l){\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$

${\mathbf{a}}^{(l)}=\sigma ({\mathbf{z}}^{(l)})$

Jahan $W^{(l)}$ ek $m\times n$ matrix hai ($n$ = layer $l-1$ mein neurons ki sankhya), aur $\sigma$ element-wise apply hota hai.

Matrix form kyun? Saare neurons ko efficiently parallel mein compute karta hai. Modern libraries (NumPy, PyTorch) matrix operations ke liye optimize ki gayi hain, jo individual neurons par loop karne se 100-1000x faster hai.

Output Layer

• Size: Task par depend karta hai
  • Binary classification: sigmoid ke saath 1 neuron ($p\in [0,1]$)
  • Multi-class: softmax ke saath $K$ neurons (probabilities ka sum 1 hota hai)
  • Regression: 1 neuron, koi activation nahi ya linear activation

$\text{softmax}(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$

Exponentiation kyun?

1. Saare outputs positive banata hai
2. Bade logits bahut badi probabilities ban jaate hain (differences amplify hote hain)
3. Outputs ka sum 1 hota hai (valid probability distribution)

Activation Functions

Common Choices

ReLU (Rectified Linear Unit): $\sigma (z)=\max (0,z)$

• Kyun? Compute karna fast hai, positive values ke liye saturate nahi karta, deep networks train karna enable karta hai
• Drawback: "Dying ReLU" problem—neurons 0 par stuck ho sakte hain agar weights $z$ ko hamesha negative push karein

Sigmoid: $\sigma (z)=\frac{1}{1+e^{-z}}$

• Kyun? $(0,1)$ mein output deta hai, probability ki tarah interpretable hai
• Drawback: Saturate karta hai (bade $|z|$ ke liye gradient 0 ke paas), vanishing gradients cause karta hai

Tanh: $\sigma (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$

• Kyun? Zero-centered (mean output = 0), symmetric
• Drawback: Yeh bhi saturate karta hai, lekin sigmoid se kam problematic

Leaky ReLU: $\sigma (z)=\max (0.01z,z)$

• Kyun? Small negative gradients allow karke dying ReLU ko fix karta hai

Task: 10 features se classify karo ki tumor malignant (1) hai ya benign (0).

Architecture:

• Input layer: 10 neurons (features: radius, texture, perimeter, etc.)
• Hidden layer 1: 16 neurons, ReLU activation
• Hidden layer 2: 8 neurons, ReLU activation
• Output layer: 1 neuron, sigmoid activation

Ek sample ke liye forward pass $\mathbf{x}=[x_1,\dots ,x_{10}{]}^T$:

1. Hidden layer 1: ${\mathbf{z}}^{(1)}=W^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)}\text{(16×10 matrix times 10×1 vector = 16×1)}$ ${\mathbf{a}}^{(1)}=\text{ReLU}({\mathbf{z}}^{(1)})\text{(element-wise)}$

   Yeh step kyun? Hum 10D input ko ek 16D space mein project kar rahe hain jahan patterns zyada separable ho sakti hain.

2. Hidden layer 2: ${\mathbf{z}}^{(2)}=W^{(2)}{\mathbf{a}}^{(1)}+{\mathbf{b}}^{(2)}\text{(8×16 times 16×1 = 8×1)}$ ${\mathbf{a}}^{(2)}=\text{ReLU}({\mathbf{z}}^{(2)})$

   Yeh step kyun? Representation ko aur refine karke 8D space mein le jaana jo higher-level tumor characteristics capture karta hai.

3. Output layer: $z^{(3)}=W^{(3)}{\mathbf{a}}^{(2)}+b^{(3)}\text{(1×8 times 8×1 = scalar)}$ $\hat{y}=\sigma (z^{(3)})=\frac{1}{1+e^{-z^{(3)}}}$ Sigmoid kyun? Output ko $[0,1]$ mein squash karta hai taaki hum ise probability ki tarah interpret kar sakein: $P(\text{malignant}\mid \mathbf{x})$.

Agar $\hat{y}=0.89$ hai, toh hum malignant predict karte hain (typically threshold 0.5 par).

Parameter count: $(10\times 16+16)+(16\times 8+8)+(8\times 1+1)=160+16+128+8+8+1=321$ parameters.

Task: 28×28 grayscale images ko digits 0-9 mein classify karo.

Architecture:

• Input: 784 neurons (28×28 flattened pixels)
• Hidden 1: 128 neurons, ReLU
• Hidden 2: 64 neurons, ReLU
• Output: 10 neurons (ek per class), softmax

Forward pass:

1. Image flatten karo: $\mathbf{x}\in {\mathbb{R}}^{784}$

2. Hidden 1: ${\mathbf{z}}^{(1)}=W^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)},{\mathbf{a}}^{(1)}=\text{ReLU}({\mathbf{z}}^{(1)})\in {\mathbb{R}}^{128}$

   128 neurons kyun? Stroke patterns, curves, edges capture karta hai.

3. Hidden 2: ${\mathbf{z}}^{(2)}=W^{(2)}{\mathbf{a}}^{(1)}+{\mathbf{b}}^{(2)},{\mathbf{a}}^{(2)}=\text{ReLU}({\mathbf{z}}^{(2)})\in {\mathbb{R}}^{64}$

   64 neurons kyun? Stroke patterns ko digit-jaise features (loops, lines, intersections) mein combine karta hai.

4. Output: ${\mathbf{z}}^{(3)}=W^{(3)}{\mathbf{a}}^{(2)}+{\mathbf{b}}^{(3)}\in {\mathbb{R}}^{10}$ $\hat{\mathbf{y}}=\text{softmax}({\mathbf{z}}^{(3)})$

   Softmax kyun? 10 scores (logits) ko probabilities mein convert karta hai: $[{\hat{y}}_0,{\hat{y}}_1,\dots ,{\hat{y}}_9]$ jahan ${\sum }_i{\hat{y}}_i=1$.

Agar output $[0.01,0.02,0.05,0.78,0.03,0.02,0.04,0.01,0.02,0.02]$ hai, toh hum class 3 predict karte hain (highest probability).

Parameter count: $(784\times 128+128)+(128\times 64+64)+(64\times 10+10)=100,352+8,256+650=109,258$ parameters.

Yeh step kyun? Network training ke through seekhta hai ki kaun se pixel combinations kaun se digits se correspond karte hain (backpropagation + gradient descent).

Universal Approximation Theorem

Iska matlab kya hai? Enough neurons ke saath, ek MLP theoretically koi bhi pattern seekh sakta hai. Lekin: yeh nahi bataata ki "enough" kitne neurons hain, ya kya hum actually sahi weights find kar sakte hain (training fail ho sakti hai).

Practice mein, deeper networks (zyada layers, har layer mein kam neurons) bahut wide shallow networks se behtar train karte hain aur better generalize karte hain.

Design Considerations

Width vs. Depth

Width = neurons per layer Depth = number of layers

Modern trend: Deeper is better (ResNets mein 50-200+ layers hoti hain).

Deeper kyun?

• Har layer ek level of abstraction seekhti hai (layer 1: edges, layer 2: textures, layer 3: parts, layer 4: objects)
• Same capacity ke liye kam parameters (zyada efficient)
• Better generalization (deeper networks mein implicit regularization hoti hai)

Trade-off: Bahut deep networks train karna mushkil hota hai (vanishing/exploding gradients) → batch normalization, skip connections (ResNets) jaise techniques ki zarurat padti hai.

Hyperparameters to Choose

1. Hidden layers ki sankhya: 2-3 se shuru karo, underfitting ho toh badhao
2. Neurons per layer: Common: 32, 64, 128, 256
3. Activation functions: Hidden layers ke liye ReLU (default), output ke liye sigmoid/softmax
4. Initialization: Xavier/He initialization (shuru mein vanishing/exploding activations rokta hai)
5. Regularization: Overfitting rokne ke liye Dropout, L2 penalty

Kyun sahi lagta hai: Zyada capacity zyada complex patterns seekhni chahiye.

Steel-man: Agar tumhare paas infinite data aur perfect optimization ho, toh haan. Reality mein, zyada capacity overfitting ki taraf le jaati hai—network general patterns seekhne ki jagah training data memorize kar leta hai. Training error kam hote hue bhi test error badh jaata hai.

Fix: Regularization use karo (dropout, L2), early stopping karo, ya capacity ghataao. Goal hai sweet spot: itni capacity ki true patterns capture ho sakein, lekin itni zyada nahi ki noise fit ho jaaye.

Kyun sahi lagta hai: Activations sirf functions hain, kahin bhi kaam karne chahiye.

Steel-man: Output activation ko tumhare loss function aur task se match karna padta hai. Binary classification mein BCE loss ke saath sigmoid chahiye (outputs $[0,1]$ mein). Multi-class mein cross-entropy ke saath softmax chahiye (valid probability distribution). Classification ke liye output mein ReLU use karne se negative values ya $>1$ aa sakti hain, jo probability interpretation aur loss computation dono todh deta hai.

Fix: Output activation ko task se match karo:

• Binary classification → sigmoid
• Multi-class → softmax
• Regression → linear (koi activation nahi) ya ReLU agar outputs positive hone chahiye

Kyun sahi lagta hai: Weights already transformation control karte hain, bias redundant lagta hai.

Steel-man: Bias ke bina, har decision boundary/hyperplane origin se guzarni chahiye. Ek single neuron $z=wx$ ke liye, activation hamesha $x=0$ par switch karega. Bias $z=wx+b$ ke saath, yeh $x=-b/w$ par switch karega (shiftable threshold). Bias network ko functions ko vertically shift karne ki freedom deta hai, jo un data ke liye crucial hai jo zero par center nahi hota.

Fix: Hamesha bias include karo jab tak tumne inputs ko zero mean par standardize na kiya ho AUR koi specific reason na ho (bahut rare hai).

Connections

• 3.1.01-SinglePerceptron-and-Linear-Separability – MLPs us XOR problem ko solve karte hain jo single perceptrons nahi kar sakte
• 3.1.03-Backpropagation-Algorithm – MLPs actually kaise seekhte hain (gradient-based weight updates)
• 3.1.04-Activation-Functions – sigmoid, ReLU, tanh aur doosron ka deep dive
• 3.2.01-Gradient-Descent-Optimization – Training algorithm jo MLP weights update karta hai
• 3.3.02-Dropout-Regularization – Deep MLPs mein overfitting rokta hai
• 4.1.01-Convolutional-Neural-Networks – Images ke liye specialized architecture (fully-connected MLPs se zyada efficient)
• 2.1.05-Feature-Engineering – MLPs automatically woh features seekhte hain jinke liye traditional ML mein manual engineering chahiye

"MLP ko ek detective team ki tarah socho jo ek mystery solve kar rahi hai. Pehla detective (input layer) saare clues (features) collect karta hai. Phir detectives ke groups (hidden layers) clues discuss karke combine karte hain—pehla group simple cheezein notice karta hai ('yeh suspect scene ke paas tha'), agla group unhe combine karta hai ('yeh suspect scene ke paas tha AUR uske paas koi alibi nahi'), aur aise aage badhte hain. Aakhir mein, head detective (output layer) final decision leta hai is par ki saari teams ne kya find kiya.

Har detective ki apni specialty hoti hai (alag alag weights), aur woh vote karte hain ki kya important hai. Jitni zyada detective teams ho (zyada layers), utne complex patterns woh recognize kar sakte hain. Lekin bahut zyada teams ho toh chhoti chhoti details par argue karne lag jaate hain jo matter nahi karti (overfitting)!

'Activation function' har detective ka decision rule hai: sigmoid hai 'Main partially sure hoon', ReLU hai 'Agar mujhe kuch suspicious dikha toh bataunga, warna chup rahunga', aur softmax hai 'Inme se saare suspects mein, yeh hai jo mere hisaab se sabse zyada guilty hai'."

Har layer representation ko refine karti hai. Data sirf forward flow karta hai (feedforward). Hidden layers hi magic hain—yeh automatically intermediate features seekhti hain.

#flashcards/ai-ml

What is a multi-layer perceptron (MLP)? :: Ek feedforward neural network jisme input aur output ke beech mein kam se kam ek hidden layer hoti hai, jahan har layer agली se fully connected hoti hai. Yeh non-linear decision boundaries seekh sakta hai.

What activation function should you use for multi-class classification output? :: Softmax, kyunki yeh logits ko classes par ek valid probability distribution mein convert karta hai (saare positive, sum 1 hota hai).

Concept Map