Image classification pipeline

Overview

The image classification pipeline is the end-to-end workflow that transforms raw pixel data into predicted class labels using convolutional neural networks. Understanding this pipeline is critical because it reveals how CNs systematically extract hierarchical features and make decisions—the foundation of computer vision systems.

[!intuition] Why Do We Need a Pipeline?

Raw images are just arrays of numbers (pixel values). A human sees "cat" or "dog" instantly, but a computer sees [[123, 45, 67], [89, 201, 34], ...]. The pipeline bridges this gap by:

1. Standardizing input → All images same size, normalized values
2. Extracting features hierarchically → Edges → Textures → Parts → Objects
3. Making probabilistic decisions → "83% dog, 15% cat, 2% wolf"

The key insight: We don't manually engineer features (like "whiskers" or "flopy ears"). The CNN learns what features matter through training data.

[!definition] Pipeline Components

The image classification pipeline consists of:

1. Preprocessing Stage: Resize, normalize, augment input images
2. Feature Extraction Stage: Convolutional + pooling layers learn hierarchical representations
3. Classification Stage: Fully connected layers map features to class probabilities
4. Postprocessing Stage: Interpret outputs, apply thresholds, visualize

[!formula] Pipeline Mathematics

1. Preprocessing: Normalization

Why normalize? Neural networks learn faster when inputs are on similar scales (typically [0,1] or [-1,1]).

Derivation from first principles:

Start with raw pixel values $x_{\text{raw}} \in [0, 255]$.

Step 1: Subtract mean $\mu$ to center data around zero:

Why this step? Centers the data so positive/negative values balance. Helps gradient descent converge faster (symmetric around origin).

Step 2: Divide by standard deviation $\sigma$ to scale variance:

Why this step? Makes variance = 1 across all features. Without this, features with large values dominate the loss function.

Common practice: Use ImageNet statistics ($\mu = [0.485, 0.456, 0.406]$, $\sigma = [0.229, 0.224, 0.225]$ for RGB) for transfer learning.

2. Forward Pass Through CNN

The forward pass transforms input $X \in \mathbb{R}^{H \times W \times 3}$ through $L$ layers:

Derivation of the flow:

Layer 1-3 (Convolutional blocks): Extract local features

Each convolution: $h^{(l)}_{i,j,k} = \sum_{m,n,c} w^{(l)}_{m,n,c,k} \cdot h^{(l-1)}_{i+m, j+n, c}$

Why convolution? Shares weights across spatial locations → translation invariance. Learns edges (layer 1), textures (layer 2), parts (layer 3).

Pooling: $h^{(l)}_{\text{pooled}} = \text{MaxPool}(h^{(l)})$ reduces spatial dimensions by 2×.

Why pooling? (1) Reduces parameters → less overfitting. (2) Creates spatial invariance → "cat in top-left" same as "cat in center".

Layer 4-5 (Fully connected): Global reasoning

Why flatten? Transition from spatial feature maps to feature vector for classification.

Output layer: Map to class logits

where $C$ = number of classes.

3. Softmax: Logits to Probabilities

The softmax function converts raw scores (logits) to valid probabilities:

Derivation from first principles:

Goal: Transform arbitrary real numbers $z_i$ into probabilities $\hat{y}_i$ such that:

1. $\hat{y}_i \in [0, 1]$
2. $\sum_{i=1}^{C} \hat{y}_i = 1$

Step 1: Exponentiate to make all positive:

Why this step? $e^x > 0$ for all $x$. Also, $e^x$ is monotonic: higher logit → higher probability.

Step 2: Normalize to sum to 1:

Why this step? Division by the sum forces $\sum \hat{y}_i = 1$ (probability axiom).

Why not just normalize $z_i$ directly? Because softmax amplifies differences: if $z_1 = 10, z_2 = 8$, then $\hat{y}_1 / \hat{y}_2 \approx e^2 \approx 7.4$ (strong preference). Linear normalization would give $10/18 \div 8/18 = 1.25$ (weak preference).

4. Loss Function: Cross-Entropy

The cross-entropy loss measures prediction error:

Derivation from information theory:

Step 1: For a single example, true label is $y$ (one-hot: $y_{\text{true}} = 1$, others = 0). We want to maximize $\log(\hat{y}_{\text{true}})$.

Why? Maximizing log probability = maximizing probability (log is monotonic). Log makes math nicer (product → sum in gradients).

Step 2: For one-hot $y$, only the true class contributes:

Why this step? If $\hat{y}_{\text{true}} = 0.9$, loss = $-\log(0.9) = 0.105$ (small). If $\hat{y}_{\text{true}} = 0.1$, loss = $-\log(0.1) = 2.303$ (large). Penalizes wrong predictions heavily.

Step 3: Average over all $N$ examples:

Why average? Makes loss scale-invariant to batch size.

[!example] Worked Example 1: Single Image Through Pipeline

Problem: Classify a224×224 RGB image of a dog using a pre-trained ResNet-50.

Step 1 - Preprocessing:

Original image: $X_{\text{raw}} \in [0, 255]^{224 \times 224 \times 3}$

Normalize using ImageNet stats:

For red channel at pixel (100, 100): $X_{\text{raw}} = 180$

Why this step? Matches the distribution ResNet was trained on → better transfer learning.

Step 2 - Feature Extraction:

After conv1 (7×7, stride 2): $112 \times 112 \times 64$ After layer1 (residual blocks): $56 \times 56 \times 256$ After layer2: $28 \times 28 \times 512$ After layer3: $14 \times 1024$ After layer4: $7 \times 7 \times 2048$

Why dimensions shrink? Each pooling/stride divides spatial size by 2. Depth increases to capture more complex features.

After global average pooling: $1 \times 1 \times 2048$ → flattened to $\mathbb{R}^{2048}$

Why global average pooling? Reduces 7×7 feature map to single vector (one number per channel). Spatial location no longer matters—we care about "is there a dog?" not "where is the dog?".

Step 3 - Classification:

Fully connected layer: $z = W \cdot h + b$, where $W \in \mathbb{R}^{1000 \times 2048}$ (ImageNet has 1000 classes).

Suppose $z = [3.2, 0.8, -1.5, 5.7, \ldots]$ (logits).

Softmax for class 0 (dog): $\hat{y}_0 = \frac{e^{3.2}}{e^{3.2} + e^{0.8} + e^{-1.5} + e^{5.7} + \ldots}$

If denominator $\approx e^{5.7}$ (largest logit dominates), then $\hat{y}_0 = e^{3.2}/e^{5.7} = e^{-2.5} \approx 0.082$ (8%).

Suppose $z_{\text{dog-class}} = 5.7$ is highest → $\hat{y}_{\text{dog}} \approx 0.85$ (85% confidence).

Why this step? Softmax spreads probability mass according to relative logit magnitudes. High logit → high probability.

Step 4 - Postprocessing:

Decision: $\arg\max_c \hat{y}_c = \text{dog}$

Confidence threshold: If max probability< 0.7, output "uncertain".

Why threshold? Prevents overconfident wrong predictions on out-of-distribution images.

[!example] Worked Example 2: Batch Processing

Problem: Process a batch of 32 images through the pipeline. How does batching affect computation?

Step 1 - Batch Construction:

Individual images: $X_i \in \mathbb{R}^{224 \times 224 \times 3}$ for $i = 1, \ldots, 32$

Stacked batch: $\mathcal{B} \in \mathbb{R}^{32 \times 224 \times 3}$

Why batching? (1) GPU paralelism: All32 convolutions happen simultaneously. (2) Stable gradients: Averaging over 32 examples reduces noise.

Step 2 - Forward Pass:

Convolution on batch:

Why this step? Same weights $W$ applied to all 32 images (shared parameters). Output: $\mathbb{R}^{32 \times 112 \times 112 \times 64}$.

After all layers: $\mathcal{B}^{(L)} \in \mathbb{R}^{32 \times 1000}$ (32 logit vectors).

Step 3 - Loss Computation:

For image $i$ with true label $y_i$:

Batch loss:

Why this step? Average loss over batch provides stable gradient estimate for backpropagation.

Example numbers: If28/32 images classified correctly (probability > 0.9), 4 wrong:

Why this matters? Low loss (< 0.5) indicates good performance. High loss (> 2) indicates poor predictions.

[!mistake] Common Mistakes

Mistake 1: Forgetting to Normalize Test Images

Wrong approach: Train with normalized images, test with raw [0, 255] pixels.

Why it feels right: "The model should handle any input."

Steel-man: The instinct is correct that models should be robust, but distribution shift breaks learned features. If training images have mean 0 and variance 1, but test images have mean 127 and variance 1000, the neurons will receive inputs far outside their training range. ReLU activations that learned to fire at 0.5 will now see values like 500—completely saturating or dying.

The fix: Always apply the same preprocessing pipeline (resize, normalization, augmentation disabled) at test time. Save the training statistics ($\mu$, $\sigma$) and reuse them.

Code example:

# Training
train_mean = train_data.mean()
train_std = train_data.std()
 
# Testing (CORRECT)
test_normalized = (test_data - train_mean) / train_std

Mistake 2: Applying Softmax Before Cross-Entropy Loss

Wrong approach:

logits = model(x)
probs = softmax(logits)
loss = cross_entropy(probs, y)  # WRONG

Why it feels right: "Cross-entropy needs probabilities, so I should softmax first."

Steel-man: The confusion arises because mathematically, cross-entropy is defined on probabilities. However, numerically, computing $e^{z_i}$ for large $z_i$ causes overflow. If $z_i = 1000$, then $e^{1000}$ is too large to represent in float32. The "log-sum-exp" trick used internally by frameworks requires raw logits.

The fix: Use cross_entropy_with_logits (PyTorch: F.cross_entropy, TensorFlow: tf.nn.softmax_cross_entropy_with_logits). These functions combine softmax + log + cross-entropy with numerical stability.

Why it works: Internally computes:

LogSumExp trick: $\text{LSE}(z) = \max(z) + \log\left(\sum_j e^{z_j - \max(z)}\right)$ prevents overflow.

Mistake 3: Using Softmax for Multi-Label Classification

Wrong approach: Image has both "cat" and "dog" (multi-label). Use softmax → forces $\hat{y}_{\text{cat}} + \hat{y}_{\text{dog}} = 1$.

Why it feels right: "Softmax worked for single-label, so it should work for multi-label."

Steel-man: The issue is softmax's normalization constraint. If an image truly contains both cat and dog, we want $\hat{y}_{\text{cat}} = 0.9$ AND $\hat{y}_{\text{dog}} = 0.85$ (both high). But softmax forces them to compete: if $\hat{y}_{\text{cat}} = 0.9$, then $\hat{y}_{\text{dog}} \leq 0.1$ (since they must sum to 1).

The fix: Use sigmoid activation for each class independently:

Now $\sum_c \hat{y}_c$ can be anything (0 to $C$). Use binary cross-entropy loss:

Example: Image contains cat (label 1) and dog (label 1). With sigmoid, can predict $\hat{y}_{\text{cat}} = 0.92$, $\hat{y}_{\text{dog}} = 0.87$ simultaneously.

[!recall]- Explain Like I'm Twelve

Imagine you're a detective trying to identify animals in photos. Here's your process:

Step 1 - Get the photo ready: All photos need to be the same size (like making all your Pokémon cards the same dimensions). You also adjust the brightness so they all look similar—not too dark, not too bright.

Step 2 - Look for clues (features): First, you notice simple things like edges (is there a round shape? pointy ears?). Then you combine those to see textures (fur? scales?). Then body parts (four legs? a tail?). Finally, you put it all together: "This is a dog!"

The cool part? The computer figures out what clues matter BY ITSELF. You don't tell it "look for whiskers." It discovers through practice that whiskers mean cat.

Step 3 - Make your guess: Instead of saying "It's a dog!" with 100% certainty, you say "I'm 85% sure it's a dog, 10% sure it's a wolf, 5% sure it's a cat." That's what softmax does—turns your confidence into percentages that add up to 100%.

Step 4 - Learn from mistakes: If you said "90% cat" but it was actually a dog, the system feels really bad (high loss). Next time, it adjusts its detective skills to avoid that mistake.

The whole pipeline is like an assembly line at a factory: raw photo → cleaned photo → extracted clues → final answer. Each step does one job well, and together they solve the puzzle!

[!mnemonic] PREP-SCORE

Preprocess: Resize and normalize Run forward pass: Conv layers extract features Exponentiate: Softmax logits to probabilities Predict: Choose argmax class

Softmax for probabilities Cross-entropy for loss Optimize with backprop Repeat for each batch Evaluate on test set

Connections

• Convolutional layers: The core feature extraction mechanism
• Batch normalization: Stabilizes training in deep pipelines
• Data augmentation: Expands preprocessing to improve generalization
• Transfer learning: Reuses trained pipelines for new tasks
• Softmax activation: Converts logits to probabilities
• Cross-entropy loss: Measures classification error
• Backpropagation: Computes gradients to update pipeline weights
• ResNet architecture: A specific pipeline with skip connections
• Multi-label classification: Modified pipeline for multiple simultaneous classes
• Object detection: Extends pipeline to localize objects (not just classify)

#flashcards/ai-ml

What are the four main stages of the image classification pipeline? :: 1) Preprocessing (resize, normalize), 2) Feature extraction (conv+pooling layers), 3) Classification (FC layers), 4) Postprocessing (interpret outputs)

Derive the softmax function from the requirement that outputs must be valid probabilities :: Need P(y_i) ∈ [0,1] and Σ P(y_i) = 1. Step 1: Exponentiate logits to make positive: ỹ_i = e^(z_i). Step 2: Normalize by sum: ŷ_i = e^(z_i) / Σ_j e^(z_j). This satisfies both constraints.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)