Inception - GoogLeNet: Question bank — AI-ML deep dive

This is a rapid-fire concept bank for the parent Inception note. No heavy arithmetic here (that lives in the cost-analysis pages) — every item targets a misconception or a boundary case that the Inception idea invites. Cover the reveal, say your answer out loud, then check.

Symbols and pictures you need first

Before any trap makes sense, we must agree on what every letter means. A feature map is a little box of numbers with three sizes: a height, a width, and a depth (how many channels stack behind each pixel).

Definition The symbols used everywhere below

$H$ = height in pixels of the feature map (number of rows). Domain: positive integer.
$W$ = width in pixels (number of columns). Domain: positive integer.
$C_{in}$ = input channel count — how many numbers sit behind each pixel coming in. Positive integer.
$C_{o u t}$ = output channel count — how many numbers behind each pixel going out. Positive integer.
$C_{mi d}$ = bottleneck width — the (usually smaller) channel count we squeeze down to before an expensive conv. Positive integer, typically $C_{mi d} < C_{in}$ .
$k$ = kernel size — the side length of the square filter window ( $k = 1, 3, 5$ here). A $k \times k$ filter looks at a $k$ -by- $k$ patch of pixels. Positive integer.
$h$ = a row index naming which pixel row we mean, $h = 1, 2, \dots, H$ .
$w$ = a column index naming which pixel column we mean, $w = 1, 2, \dots, W$ .
$c$ = an input-channel index, $c = 1, 2, \dots, C_{in}$ — it just names which input channel we are talking about in a sum.
$o$ = an output-channel index, $o = 1, 2, \dots, C_{o u t}$ — it names which output channel we are producing.

The figure above shows one feature map as a box: the front face is $H \times W$ pixels, the depth into the page is the channels. Everything on this page is about reshaping that box cheaply.

Deriving when a bottleneck pays off

True or false — justify

A "1×1 convolution has no receptive field, so it is just an identity operation."

False. It has no spatial extent but it still spans all input channels, so it computes a learned linear combination

y_{h, w, o} = \sum_{c} w_{o, c} x_{h, w, c}

across channels — a per-pixel projection, not an identity.

A "The four branches of an Inception module must all output the same number of channels so they can be concatenated."

False. They must share the same spatial size

H \times W

; their channel counts differ freely (e.g. 64, 128, 32, 32) and simply add up along the depth axis after concatenation.

A "Depth-concatenation loses information compared to summing the branches."

False. Concatenation stacks all branch outputs side by side, so every feature is preserved; the next layer's weights then decide what to emphasise. Summing (as in ResNet) would blend and could cancel signals.

A "Because Inception tries every filter size, it has more parameters than a plain deep stack of 3×3 convs."

False (usually). The 1×1 bottlenecks shrink channel depth before the 5×5 and 3×3 convs, so the parallel design is actually cheaper; GoogLeNet (~6.8M params) has roughly 20× fewer parameters than VGG16 (~138M) (see 3.4.05-VGG).

A "Auxiliary classifiers make GoogLeNet more accurate at test time."

False. They are discarded at test time and exist to push gradient signal into middle layers during training; later Inception variants removed them with no accuracy loss (3.4.11-Inception-v2v3).

A "The 1×1 conv before a 5×5 conv always saves computation."

False. It only helps when

C_{mi d} (C_{in} + k^{2} C_{o u t}) < k^{2} C_{in} C_{o u t}

. If

C_{mi d}

is chosen close to

C_{in}

, the bottleneck adds cost instead of removing it.

A "Adding a ReLU after a 1×1 conv changes nothing since it's already just a linear mix."

False. Without ReLU the 1×1 is purely linear; the ReLU makes the channel mixing nonlinear, which is why 1×1 convs can act as genuine feature transformers, not just dimensionality reducers.

A "The pooling branch reduces the spatial size of the module output."

False. The 3×3 max-pool inside the module uses stride 1 with same padding, so it preserves

H \times W

; only the between-module max-pools downsample.

Spot the error

"To make the module cheaper, put the 1×1 bottleneck after the 5×5 conv."

The expensive 5×5 conv would still run over the full

C_{in}

channels — the whole point is to shrink depth before the costly conv, so the bottleneck must come first.

"An Inception module fuses scales, so we can stop stacking modules — one is enough."

One module fuses scales at one depth. Stacking modules lets multi-scale features feed into more multi-scale features, building deeper abstraction; GoogLeNet uses nine.

"We can drop the pooling branch since 1×1/3×3/5×5 already cover all filter sizes."

The pooling branch contributes translation invariance and a different summary statistic (max), which no conv branch provides; removing it loses that complementary feature.

"Since auxiliary classifiers fight vanishing gradients, give them weight 1.0 like the main loss."

A weight of 1.0 lets the auxiliary objectives dominate and pull the network toward intermediate-layer predictions; the 0.3 weight keeps them as regularizers, not primary goals.

"1×1 convs are just PCA on the channel dimension."

PCA is unsupervised, linear, and orthogonal; a 1×1 conv learns task-driven weights via backprop and becomes nonlinear with ReLU — far more expressive than PCA.

"Concatenating branches means the network must use all of them, so wasted branches hurt accuracy."

The downstream weights can drive an unhelpful branch's contribution toward zero, so an unused branch is inert, not harmful — it's soft selection via learning.

"A 3×3 conv branch with valid padding can still be concatenated with a 1×1 branch."

With valid padding the 3×3 branch shrinks to

(H - 2) \times (W - 2)

while the 1×1 branch stays

H \times W

— the front faces no longer match, so depth-concatenation fails; both must use same padding.

Why questions

Why use parallel filter sizes instead of picking the best one per layer?

We don't know a priori which receptive field matters at layer N, so Inception computes all scales at once and lets backprop weight them — turning an architecture guess into a learned choice.

Why does a 1×1 conv qualify as "dimensionality reduction with learned weights"?

It maps

C_{in}

channels to a smaller

C_{o u t}

via a learned linear projection per pixel, compressing channel depth like PCA but with supervised, nonlinear weights.

Why did GoogLeNet need auxiliary classifiers in 2014 but modern nets don't?

In 2014 there was no BatchNorm and no skip connections, so gradients vanished in deep stacks (3.2.06-Vanishing-Gradients); later, BatchNorm and residual links in 3.4.09-ResNet fixed gradient flow directly.

Why concatenate along the channel axis rather than adding branch outputs?

Concatenation keeps every scale's features distinct and available; addition (as in ResNet) requires matching channel counts and blends features, which serves a different purpose.

Why were the exact bottleneck sizes (96, 16, ...) chosen by hand in the original paper?

They were manually tuned to balance cost and expressivity; this hand-tuning is exactly what later Neural Architecture Search automated.

Why does the 1×1 conv act "like a fully-connected layer applied per-pixel"?

At each spatial location it takes the

C_{in}

-length channel vector and outputs a

C_{o u t}

-length vector via a weight matrix — the same operation as a dense layer, shared across all pixels.

Edge cases

What happens to the bottleneck savings when

k = 1

(a 1×1 "conv" you try to bottleneck)?

With

k = 1

the inequality becomes

C_{mi d} (C_{in} + C_{o u t}) < C_{in} C_{o u t}

; this can only be satisfied by a very small

C_{mi d}

, and there's no

k^{2}

spatial cost to amortize, so bottlenecking a 1×1 conv rarely helps and often adds work.

What if the intermediate bottleneck width

C_{mi d}

equals

C_{in}

Then no channels are reduced, the bottleneck is a pure (possibly wasteful) extra layer, and total cost exceeds the naive conv — the savings vanish entirely.

What if a branch's downstream weights all learn to be zero?

That branch's features are effectively ignored, so the module gracefully degrades to using the remaining branches — capacity is "wasted" but accuracy is not harmed.

At test time, what portion of GoogLeNet is actually run?

The main path only; both auxiliary classifiers are removed, so the deployed network is lighter than the training-time graph.

What happens to a pooling-branch output if the pool had stride 2 instead of 1?

Its front face would halve to roughly

\frac{H}{2} \times \frac{W}{2}

, breaking depth-concatenation with the other branches (which stay

H \times W

) — stride-1 with same padding is required to keep front faces aligned.

If a conv branch uses valid instead of same padding, can it still be concatenated?

No — valid padding shrinks the front face while other branches keep

H \times W

, so the boxes no longer align; same padding across all branches is what guarantees compatible concatenation.

C_{o u t}

of a 5×5 branch is very large, does a 16-channel bottleneck still help?

Yes, and even more so: the

k^{2} C_{in} C_{o u t}

term grows with

C_{o u t}

, so a small

C_{mi d}

cutting

C_{in}

down first saves increasingly large amounts as

C_{o u t}

rises.

Locking it in

Recall Spatial vs channel — the axis confusion

1×1 convs act on the channel axis; 3×3/5×5 convs act on the spatial axis. Most "false" traps come from swapping these.

Recall Concatenation vs summation

Depth-concatenation stacks boxes and keeps every feature; ResNet-style summation adds them and can cancel. Different tools, different jobs.

Recall Training-only vs test-time

Auxiliary classifiers exist only to fight vanishing gradients during training and are removed at test time — never a test-accuracy booster.

Recall Same padding is the glue

Every branch uses same padding, stride 1, so all four output boxes share one $H \times W$ and can be concatenated.