Multi-layer perceptron architecture

What is an MLP?

Each layer contains multiple neurons (nodes), and every neuron in layer $l$ is connected to every neuron in layer $l+1$ (fully connected). Information flows forward only—no cycles.

Why Multiple Layers?

Single-layer perceptrons can only learn linearly separable functions. The XOR problem famously cannot be solved by a single layer. Adding hidden layers allows the network to learn non-linear decision boundaries by composing simple transformations.

$z_j^{(l)} = \sum_{i} w_{ji}^{(l)} a_i^{(l-1)} + b_j^{(l)}$

$a_j^{(l)} = \sigma(z_j^{(l)})$

Where:

• $a_i^{(l-1)}$ = activations from previous layer (inputs to this neuron)
• $w_{ji}^{(l)}$ = weight from neuron $i$ in layer $l-1$ to neuron $j$ in layer $l$
• $b_j^{(l)}$ = bias term for neuron $j$
• $z_j^{(l)}$ = pre-activation (weighted sum)
• $a_j^{(l)}$ = activation (after applying non-linearity)
• $\sigma$ = activation function (sigmoid, ReLU, tanh, etc.)

Why this formula? Each neuron performs a weighted vote of its inputs, adds a bias (shifts the decision boundary), then applies a non-linear function. Without $\sigma$, stacking layers would collapse to a single linear transformation: $W_3(W_2(W_1 x)) = (W_3 W_2 W_1)x = W_{combined} x$. The non-linearity breaks this and enables learning complex patterns.

Architecture Components

Input Layer

• Purpose: Holds the raw feature vector $\mathbf{x} = [x_1, x_2, \ldots, x_n]^T$
• No computation: These "neurons" just pass values forward
• Size: Determined by your data (e.g., 784 for 28×28 pixel images, 100 for 100 features)

Hidden Layers

How many neurons per layer? Common heuristics:

• Start with 2-3x the input size
• Use powers of 2 (32, 64, 128, 256) for computational efficiency
• More neurons = more capacity but risk of overfitting
• Deeper (more layers) often beats wider (more neurons per layer) for the same parameter count

Derivation of layer output (vectorized form):

For the entire layer $l$ with $m$ neurons:

$\mathbf{z}^{(l)} = W{(l)} \mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}$

$\mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)})$

Where $W^{(l)}$ is an $m \times n$ matrix ($n$ = number of neurons in layer $l-1$), and $\sigma$ is applied element-wise.

Why matrix form? Efficiently computes all neurons in parallel. Modern libraries (NumPy, PyTorch) are optimized for matrix operations, making this100-1000x faster than loping over individual neurons.

Output Layer

• Size: Depends on the task
  • Binary classification: 1 neuron with sigmoid ($p \in [0,1]$)
  • Multi-class: $K$ neurons with softmax (probabilities sum to 1)
  • Regression: 1 neuron, no activation or linear activation

$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$

Why exponentiation?

1. Makes all outputs positive
2. Larger logits become much larger probabilities (amplifies differences)
3. Outputs sum to 1 (valid probability distribution)

Activation Functions

Common Choices

ReLU (Rectified Linear Unit): $\sigma(z) = \max(0, z)$

• Why? Fast to compute, doesn't saturate for positive values, enables training deep networks
• Drawback: "Dying ReLU" problem—neurons can get stuck at 0 if weights push $z$ always negative

Sigmoid: $\sigma(z) = \frac{1}{1 + e^{-z}}$

• Why? Outputs in $(0, 1)$, interpretable as probability
• Drawback: Saturates (gradient near 0 for large $|z|$), causes vanishing gradients

Tanh: $\sigma(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$

• Why? Zero-centered (mean output = 0), symmetric
• Drawback: Also saturates, but less problematic than sigmoid

Leaky ReLU: $\sigma(z) = \max(0.01z, z)$

• Why? Fixes dying ReLU by allowing small negative gradients

Task: Classify if a tumor is malignant (1) or benign (0) from10 features.

Architecture:

• Input layer: 10 neurons (features: radius, texture, perimeter, etc.)
• Hidden layer 1: 16 neurons, ReLU activation
• Hidden layer 2: 8 neurons, ReLU activation
• Output layer: 1 neuron, sigmoid activation

Forward pass for one sample $\mathbf{x} = [x_1, \ldots, x_{10}]^T$:

1. Hidden layer 1: $\mathbf{z}^{(1)} = W^{(1)} \mathbf{x} + \mathbf{b}^{(1)} \quad \text{(16×10 matrix times 10×1 vector = 16×1)}$ $\mathbf{a}^{(1)} = \text{ReLU}(\mathbf{z}^{(1)}) \quad \text{(element-wise)}$

   Why this step? We're projecting the 10D input into a 16D space where patterns might be more separable.

2. Hidden layer 2: $\mathbf{z}^{(2)} = W^{(2)} \mathbf{a}^{(1)} + \mathbf{b}^{(2)} \quad \text{(8×16 times 16×1 = 8×1)}$ $\mathbf{a}^{(2)} = \text{ReLU}(\mathbf{z}^{(2)})$

   Why this step? Further refining the representation into8D space that captures higher-level tumor characteristics.

3. Output layer: $z^{(3)} = W^{(3)} \mathbf{a}^{(2)} + b^{(3)} \quad \text{(1×8 times 8×1 = scalar)}$ $\hat{y} = \sigma(z^{(3)}) = \frac{1}{1 + e^{-z^{(3)}}}$ Why sigmoid? Squashes output to $[0,1]$ so we can interpret as probability: $P(\text{malignant} \mid \mathbf{x})$.

If $\hat{y} = 0.89$, we predict malignant (typically threshold at 0.5).

Parameter count: $(10 \times 16 + 16) + (16 \times 8 + 8) + (8 \times 1 + 1) = 160 + 16 + 128 + 8 + 8 1 = 321$ parameters.

Task: Classify 28×28 grayscale images into digits 0-9.

Architecture:

• Input: 784 neurons (28×28 flattened pixels)
• Hidden 1: 128 neurons, ReLU
• Hidden 2: 64 neurons, ReLU
• Output: 10 neurons (one per class), softmax

Forward pass:

1. Flatten image: $\mathbf{x} \in \mathbb{R}^{784}$

2. Hidden 1: $\mathbf{z}^{(1)} = W^{(1)} \mathbf{x} + \mathbf{b}^{(1)}, \quad \mathbf{a}^{(1)} = \text{ReLU}(\mathbf{z}^{(1)}) \in \mathbb{R}^{128}$

   Why 128 neurons? Captures stroke patterns, curves, edges.

3. Hidden 2: $\mathbf{z}^{(2)} = W^{(2)} \mathbf{a}^{(1)} + \mathbf{b}^{(2)}, \quad \mathbf{a}^{(2)} = \text{ReLU}(\mathbf{z}^{(2)}) \in \mathbb{R}^{64}$

   Why 64 neurons? Combines stroke patterns into digit-like features (loops, lines, intersections).

4. Output: $\mathbf{z}^{(3)} = W^{(3)} \mathbf{a}^{(2)} + \mathbf{b}^{(3)} \in \mathbb{R}^{10}$ $\mathbf{\hat{y}} = \text{softmax}(\mathbf{z}^{(3)})$

   Why softmax? Converts 10 scores (logits) into probabilities: $[\hat{y}_0, \hat{y}_1, \ldots, \hat{y}_9]$ where $\sum_i \hat{y}_i = 1$.

If output is $[0.01, 0.02, 0.05, 0.78, 0.03, 0.02, 0.04, 0.01, 0.02, 0.02]$, we predict class 3 (highest probability).

Parameter count: $(784 \times 128 + 128) + (128 \times 64 + 64) + (64 \times 10 + 10) = 100,352 + 8,256 + 650 = 109,258$ parameters.

Why this step? The network learns which pixel combinations correspond to which digits through training (backpropagation + gradient descent).

Universal Approximation Theorem

What does this mean? Given enough neurons, an MLP can theoretically learn any pattern. But: this says nothing about how many neurons "enough" is, or whether we can actually find the right weights (training might fail).

In practice, deeper networks (more layers, fewer neurons per layer) train better and generalize better than very wide shallow networks.

Design Considerations

Width vs. Depth

Width = neurons per layer
Depth = number of layers

Modern trend: Deeper is better (ResNets have 50-200+ layers).

Why deeper?

• Each layer learns a level of abstraction (layer 1: edges, layer 2: textures, layer 3: parts, layer 4: objects)
• Fewer parameters for the same capacity (more efficient)
• Better generalization (deeper networks have implicit regularization)

Trade-off: Very deep networks are harder to train (vanishing/exploding gradients) → need techniques like batch normalization, skip connections (ResNets).

Hyperparameters to Choose

1. Number of hidden layers: Start with 2-3, increase if underfitting
2. Neurons per layer: Common: 32, 64, 128, 256
3. Activation functions: ReLU for hidden layers (default), sigmoid/softmax for output
4. Initialization: Xavier/He initialization (prevents vanishing/exploding activations at start)
5. Regularization: Dropout, L2 penalty to prevent overfitting

Why it feels right: More capacity should learn more complex patterns.

Steel-man: If you have infinite data and perfect optimization, yes. In reality, more capacity leads to overfitting—the network memorizes training data instead of learning general patterns. The test error increases even as training error decreases.

Fix: Use regularization (dropout, L2), early stopping, or reduce capacity. The goal is the sweet spot: enough capacity to capture true patterns, not so much that you fit noise.

Why it feels right: Activations are just functions, should work anywhere.

Steel-man: The output activation must match your loss function and task. For binary classification with BCE loss, you need sigmoid (outputs in $[0,1]$). For multi-class with cross-entropy, you need softmax (valid probability distribution). Using ReLU in output for classification would give negative values or $> 1$, breaking probability interpretation and loss computation.

Fix: Match output activation to task:

• Binary classification → sigmoid
• Multi-class → softmax
• Regression → linear (no activation) or ReLU if outputs must be positive

Why it feels right: Weights already control the transformation, bias seems redundant.

Steel-man: Without bias, every decision boundary/hyperplane must pass through the origin. For a single neuron $z = wx$, the activation switches at $x = 0$ always. With bias $z = wx + b$, it switches at $x = -b/w$ (shiftable threshold). Bias gives the network freedom to shift functions vertically, crucial for fitting data that doesn't center at zero.

Fix: Always include bias unless you've standardized inputs to zero mean AND have a specific reason (very rare).

Connections

• 3.1.01-SinglePerceptron-and-Linear-Separability – MLPs solve the XOR problem single perceptrons cannot
• 3.1.03-Backpropagation-Algorithm – How MLPs actually learn (gradient-based weight updates)
• 3.1.04-Activation-Functions – Deep dive into sigmoid, ReLU, tanh, and others
• 3.2.01-Gradient-Descent-Optimization – Training algorithm that updates MLP weights
• 3.3.02-Dropout-Regularization – Prevents overfitting in deep MLPs
• 4.1.01-Convolutional-Neural-Networks – Specialized architecture for images (more efficient than fully-connected MLPs)
• 2.1.05-Feature-Engineering – MLPs automatically learn features that traditional ML requires manual engineering for

"Think of an MLP like a team of detectives solving a mystery. The first detective (input layer) collects all the clues (features). Then groups of detectives (hidden layers) discuss and combine clues—the first group notices simple things ('this suspect was near the scene'), the next group combines those ('this suspect was near the scene AND has no alibi'), and so on. Finally, the head detective (output layer) makes the final decision based on what all the teams found.

Each detective has their specialty (different weights), and they vote on what's important. The more detective teams you have (more layers), the more complex patterns they can recognize. But too many teams might argue over tiny details that don't matter (overfitting)!

The 'activation function' is like each detective's decision rule: sigmoid is'I'm partially sure', ReLU is 'If I see something suspicious, I'll tell you, otherwise I stay quiet', and softmax is 'Out of all these suspects, here's who I think is most guilty'."

Each layer refines the representation. Data flows forward only (feedforward). Hidden layers are the magic—they learn intermediate features automatically.

#flashcards/ai-ml

What is a multi-layer perceptron (MLP)? :: A feedforward neural network with at least one hidden layer between input and output, where each layer is fully connected to the next. It can learn non-linear decision boundaries.

What activation function should you use for multi-class classification output? :: Softmax, because it converts logits into a valid probability distribution (all positive, sum to 1) over classes.

Concept Map

Hinglish (regional understanding)