Image classification pipeline: Question bank — AI-ML deep dive

This is a question bank of conceptual traps for the Image classification pipeline. Each item is a one-line reveal: read the prompt, answer in your head, then check. No heavy arithmetic here — these test whether you truly understand why each stage exists and when the usual reasoning breaks. First, three tiny reminders so no symbol below is a surprise.

Recall What is a

logit? A raw, unbounded real-number score $z_{i}$ that the network outputs for class $i$ before it is turned into a probability. It can be negative, zero, or large — it is not yet a probability.

Recall What does

softmax do, in one sentence? It maps a whole vector of logits $z_{1}, \dots, z_{C}$ to probabilities $\overset{y}{^}_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$ that are all in $[0, 1]$ and sum to $1$ .

Recall What is a

one-hot label? A label vector with a single $1$ at the true class and $0$ everywhere else, e.g. "dog" $= [0, 1, 0, \dots]$ .

True or false — justify

TF · Normalizing with ImageNet mean/std is always correct, even for X-ray or satellite images.

False — those statistics describe natural RGB photos; medical or satellite pixels have a different distribution, so you should compute (or re-estimate)

μ, σ

on your own data (see Transfer learning).

TF · Softmax outputs are calibrated probabilities you can trust as "85% sure."

False — softmax guarantees the numbers sum to 1, but a network can be badly over-confident; a 0.85 output does not mean it is right 85% of the time without calibration.

TF · Adding a constant

c

to every logit changes the softmax output.

False —

e^{z_{i} + c} = e^{c} e^{z_{i}}

and the

e^{c}

cancels top and bottom, so softmax is shift-invariant; this is exactly why we subtract

max z

for numerical stability.

TF · Cross-entropy loss looks at the probabilities of all classes for a one-hot label.

False — with a one-hot label only the true class term survives, so

L = - lo g (\overset{y}{^}_{true})

; the other classes affect the loss only indirectly through the softmax denominator.

TF · Max pooling is what gives a CNN translation invariance; convolution alone does not.

Half-true — convolution gives equivariance (a shifted input gives a shifted feature map), and pooling/global-average-pooling then converts that equivariance into invariance to small shifts.

TF · A larger training loss on a batch always means worse predictions.

False — loss also depends on confidence; a correct-but-unsure prediction (

\overset{y}{^}_{true} = 0.4

) can have higher loss than a barely-wrong one, so accuracy and loss can disagree.

TF · Global average pooling throws away where the object is, which is a bug.

False — it is deliberate: for classification we only ask "is a dog present?", not "where?"; keeping location would matter for Object detection, not for a single class label.

TF · Because softmax always outputs a distribution, a CNN can honestly say "I don't know."

False — softmax forces the mass to sum to 1 over the known classes, so an out-of-distribution image still gets a confident-looking label; you need a threshold or open-set method to express "unknown."

TF · If two classes have equal logits, softmax gives each exactly

1/ C

times the leftover mass.

True for those two — equal logits give equal probabilities to each other, though their shared value still depends on all the other logits in the denominator.

Spot the error

ERR · "We normalize pixels to speed training, so we should also normalize the one-hot labels the same way."

Wrong — labels are categorical targets, not features on a scale; normalizing them destroys their meaning. Only inputs are normalized.

ERR · "To turn logits into probabilities I just divide each

z_{i}

\sum_{j} z_{j}

Wrong — logits can be negative or sum to zero, breaking that division, and it ignores the exponential amplification softmax needs to reflect relative confidence.

ERR · "Cross-entropy uses

lo g

mainly because it is a smooth function."

Incomplete — the deeper reason is that maximizing

lo g \overset{y}{^}_{true}

equals maximizing likelihood, and

lo g

turns products of probabilities into sums, making gradients tractable (link to Backpropagation).

ERR · "Batching 32 images changes the learned weights

W

per image."

Wrong — the same shared weights are applied to all 32 images; batching parallelizes computation and averages the gradient, it does not create per-image weights.

ERR · "Pooling reduces overfitting because it adds trainable parameters."

Wrong — max/average pooling has no trainable parameters; it reduces overfitting by shrinking the spatial map and thus cutting downstream parameters.

ERR · "For a multi-label image (cat AND dog) I still use softmax + cross-entropy as usual."

Wrong — softmax forces one probability distribution that competes across classes; multi-label needs per-class sigmoids (see Multi-label classification).

ERR · "I normalized the training set with training-set stats, so I'll recompute stats on the test set for fairness."

Wrong — you must apply the training

μ, σ

to test data; recomputing leaks information and creates a train/test mismatch.

Why questions

WHY · Why exponentiate logits before normalizing instead of just clamping them positive?

Exponentiation makes the mapping monotonic and multiplicative, so a logit gap of 2 becomes a probability ratio

e^{2} \approx 7.4

— softmax rewards relative confidence, which linear clamping cannot express.

WHY · Why does spatial size shrink while channel depth grows as we go deeper?

Strides/pooling compress where information to save computation, while more channels store what information — increasingly abstract features (edges → parts → objects) need more feature dimensions.

WHY · Why apply a confidence threshold after argmax?

Argmax always picks something; the threshold lets the system abstain on ambiguous or out-of-distribution inputs instead of emitting a confident wrong label.

WHY · Why is Batch normalization often placed inside the feature-extraction stage rather than as preprocessing?

Preprocessing normalizes only the raw input once; batch norm renormalizes activations at each layer during training, keeping deep signals well-scaled as the network learns.

WHY · Why does Data augmentation belong in the preprocessing stage and not the classification stage?

Augmentation perturbs inputs (flips, crops, colour jitter) to teach invariances before features are extracted; doing it after feature extraction would not expose the conv layers to the variety they must learn to ignore.

WHY · Why can ResNet architecture stack very deep layers where plain CNNs fail?

Its residual skip connections let gradients flow through identity shortcuts, avoiding the vanishing-gradient collapse that plagues very deep plain stacks during Backpropagation.

WHY · Why average the cross-entropy over the batch instead of summing it?

Averaging makes the loss magnitude (and thus the gradient step) independent of batch size, so learning-rate tuning stays stable when you change how many images you group.

Edge cases

EDGE · What does softmax output if every logit is identical?

A perfectly uniform distribution,

\overset{y}{^}_{i} = 1/ C

for all

i

— the network expresses maximum uncertainty, which is the correct behaviour when no class is favoured.

EDGE · What happens to cross-entropy if the model assigns

\overset{y}{^}_{true} = 0

- lo g (0) = + \infty

— the loss blows up, which is why implementations add a tiny

ϵ

or fuse softmax with the log for numerical safety.

EDGE · With one huge logit and the rest very negative, what is the softmax?

It saturates toward a near one-hot vector (the winner

\approx 1

, others

\approx 0

); the gradient there is tiny, which can slow further learning — a mild vanishing signal.

EDGE · A single-class problem (

C = 1

): what does softmax give?

Always exactly

1

— with one class the denominator equals the numerator, so softmax is degenerate and a sigmoid/binary formulation should be used instead.

EDGE · What if a test image is a class the network was never trained on?

Softmax still distributes all mass over the known classes and may look confident; only a threshold or open-set/out-of-distribution method can flag it as "none of these."

EDGE · What if two classes are perfectly tied for the max probability?

argmax breaks the tie arbitrarily (usually the lowest index), so a genuinely ambiguous case is silently resolved — a reason to inspect the full distribution, not just the top-1 label.

EDGE · A zero-variance channel (

σ = 0

) during normalization: what breaks?

Division by zero — this occurs for a constant/blank channel; you must guard with

σ + ϵ

or drop the dead channel before normalizing.