Inception - GoogLeNet: Exercises — AI-ML deep dive

This page is a self-test ladder for Inception - GoogLeNet. Work each problem before opening its solution. The levels climb from recognition (spotting what a piece does) to mastery (designing and defending a module). Every arithmetic answer here is machine-checked.

Before we start, one symbol contract so nothing appears un-earned:

See the parallel-branch picture you will keep referring to:

Level 1 — Recognition

Exercise 1.1 (L1)

An Inception module has four parallel branches. Name each branch's job in one phrase, and state what operation joins their outputs at the end.

Recall Solution

Branch 1 — $1 \times 1$ conv: pure channel mixing, no spatial reach.
Branch 2 — $1 \times 1 \to 3 \times 3$ : bottleneck, then medium receptive field.
Branch 3 — $1 \times 1 \to 5 \times 5$ : bottleneck, then large receptive field.
Branch 4 — $3 \times 3$ max-pool $\to 1 \times 1$ : keep spatial info, then shrink channels.
Join: depth-concatenation — stack all branch outputs along the channel axis. Because every branch keeps the same $H \times W$ , the grids line up and simply pile up in depth.

Exercise 1.2 (L1)

In one sentence, what question is an Inception module answering that a single fixed $3 \times 3$ stack does not?

Recall Solution

"Which filter size matters at this layer?" — instead of committing to one scale, it runs all scales in parallel and lets backprop weight them.

Exercise 1.3 (L1)

True/False: auxiliary classifiers stay in the network at test time. Justify.

Recall Solution

False. They exist only during training to feed gradient into middle layers (fighting vanishing gradients) and are discarded at test time.

Level 2 — Application

Exercise 2.1 (L2)

A $1 \times 1$ conv turns a $28 \times 28 \times 192$ volume into $28 \times 28 \times 64$ . How many ops?

Recall Solution

$k = 1$ , so $k^{2} = 1$ . $Ops = 28 \cdot 28 \cdot 1 \cdot 192 \cdot 64 = 784 \cdot 12288 = 9, 633, 792.$

Exercise 2.2 (L2)

Direct (no bottleneck) $5 \times 5$ conv, $C_{in} = 192 \to C_{o u t} = 32$ , on a $28 \times 28$ map. Ops?

Recall Solution

$28 \cdot 28 \cdot 25 \cdot 192 \cdot 32 = 784 \cdot (25 \cdot 192 \cdot 32) = 784 \cdot 153600 = 120, 422, 400.$

Exercise 2.3 (L2)

Same target as 2.2, but first bottleneck $192 \to 16$ with a $1 \times 1$ , then $5 \times 5$ $16 \to 32$ . Total ops? Reduction factor vs. 2.2 (2 d.p.)?

Recall Solution

Step 1 ( $1 \times 1$ , $192 \to 16$ ): $28 \cdot 28 \cdot 1 \cdot 192 \cdot 16 = 784 \cdot 3072 = 2, 408, 448$ . Step 2 ( $5 \times 5$ , $16 \to 32$ ): $28 \cdot 28 \cdot 25 \cdot 16 \cdot 32 = 784 \cdot 12800 = 10, 035, 200$ . Total $= 12, 443, 648$ . Reduction $= 120, 422, 400/12, 443, 648 \approx 9.68 \times$ .

Level 3 — Analysis

Exercise 3.1 (L3)

For the single module in the parent note (input $28 \times 28 \times 192$ ; branches output $64$ , $128$ , $32$ , $32$ ), verify the concatenated depth and give the final volume shape.

Recall Solution

Depth $= 64 + 128 + 32 + 32 = 256$ . Spatial size is unchanged (every branch keeps $28 \times 28$ ). Output: $28 \times 28 \times 256$ .

Exercise 3.2 (L3)

Derive the exact break-even bottleneck size $C_{mi d}$ where bottlenecking a $k \times k$ conv costs the same as the naive conv. Then evaluate for $k = 5, C_{in} = 256, C_{o u t} = 32$ .

Recall Solution

Set bottleneck cost = naive cost (drop the shared $H \cdot W$ ): $C_{in} C_{mi d} + k^{2} C_{mi d} C_{o u t} = k^{2} C_{in} C_{o u t} .$ Factor $C_{mi d}$ on the left:

\;\Rightarrow\; C_{mid} = \frac{k^2 C_{in} C_{out}}{C_{in} + k^2 C_{out}}.$$ Plug in: numerator $=25\cdot256\cdot32 = 204800$; denominator $=256 + 25\cdot32 = 1056$. $$C_{mid} = 204800/1056 \approx 193.94.$$ **Reading:** any $C_{mid} < 194$ saves ops. The paper's choice $C_{mid}=16$ sits far below this, hence the huge saving.

Exercise 3.3 (L3)

GoogLeNet's total loss is $L_{total} = L_{main} + 0.3 L_{aux1} + 0.3 L_{aux2}$ . If a batch gives $L_{main} = 2.0$ , $L_{aux1} = 2.5$ , $L_{aux2} = 3.0$ , compute $L_{total}$ and the fraction contributed by the auxiliaries.

Recall Solution

$L_{total} = 2.0 + 0.3 (2.5) + 0.3 (3.0) = 2.0 + 0.75 + 0.90 = 3.65.$ Auxiliary fraction $= 1.65/3.65 \approx 0.4521$ , i.e. about $45.21%$ .

Level 4 — Synthesis

Exercise 4.1 (L4)

Design a bottleneck for a $14 \times 14 \times 480$ input feeding a $3 \times 3$ conv to $208$ output channels. Pick $C_{mi d} = 96$ . Compute naive vs. bottleneck ops and the reduction factor (2 d.p.).

Recall Solution

Naive ( $3 \times 3$ , $480 \to 208$ ): $14 \cdot 14 \cdot 9 \cdot 480 \cdot 208 = 196 \cdot (9 \cdot 480 \cdot 208) = 196 \cdot 898560 = 176, 117, 760$ . Bottleneck:

$1 \times 1$ $480 \to 96$ : $196 \cdot 1 \cdot 480 \cdot 96 = 196 \cdot 46080 = 9, 031, 680$ .
$3 \times 3$ $96 \to 208$ : $196 \cdot 9 \cdot 96 \cdot 208 = 196 \cdot 179712 = 35, 223, 552$ .
Total $= 44, 255, 232$ . Reduction $= 176, 117, 760/44, 255, 232 \approx 3.98 \times$ .

Exercise 4.2 (L4)

You must build an Inception module whose output is $14 \times 14 \times 512$ using four branches with depths $128, 256, 64, 64$ . Confirm the depths sum correctly, and explain why the pooling branch needs its own $1 \times 1$ conv rather than outputting the raw pooled volume.

Recall Solution

Depths: $128 + 256 + 64 + 64 = 512$ ✓. Why the $1 \times 1$ after pooling: max-pool preserves the input's channel count (here whatever $C_{in}$ is, often large). Concatenating that raw depth would make the module's output balloon layer after layer. The $1 \times 1$ projects it down to the assigned $64$ channels, keeping depths controlled and the concatenation on-budget. This is the same dimensionality reduction idea, applied to the pool branch.

Level 5 — Mastery

Exercise 5.1 (L5)

A colleague proposes replacing all Inception bottlenecks with $C_{mi d} = C_{in}$ ("don't reduce, just pass through, keep more info"). Using the break-even formula, prove this can only ever increase or equal cost, never reduce it, and explain the practical consequence.

Recall Solution

With $C_{mi d} = C_{in}$ the bottleneck total (per position) is $C_{in} \cdot C_{in} + k^{2} C_{in} C_{o u t} = C_{in}^{2} + k^{2} C_{in} C_{o u t},$ versus naive $k^{2} C_{in} C_{o u t}$ . The difference is $+ C_{in}^{2} > 0$ . So you pay the full $k \times k$ conv plus an extra $1 \times 1$ — strictly worse. Consequence: a bottleneck only helps when $C_{mi d} < C_{mi d}^{break-even} = \frac{k ^{2} C _{in} C _{o u t}}{C _{in} + k ^{2} C _{o u t}}$ ; "not reducing" defeats the entire purpose.

Exercise 5.2 (L5)

Argue, quantitatively where possible, why modern Inception-v2/v3 could remove auxiliary classifiers with no accuracy loss, whereas 2014 GoogLeNet needed them.

Recall Solution

Auxiliaries existed to inject gradient into middle layers when deep-net gradients shrank toward zero. Two later tools solved that at the source:

BatchNorm normalizes activations so gradient magnitudes stay stable across depth (see Inception-v2/v3).
Skip connections (ResNet) give gradient a direct additive highway. With gradient flow fixed, the aux classifiers' reason to exist vanishes — hence they were dropped with no accuracy cost. This is the parent note's Mistake 3: aux classifiers were a training crutch, not an accuracy booster. Design choices should be justified by the problem they solve, not copied.

Exercise 5.3 (L5)

Contrast the design philosophy of hand-tuning Inception's branch depths (as in 2014) with automating them. Name the vault topic that automates this and state the trade-off in one line.

Recall Solution

The original branch depths ( $64, 96, 128, 16, 32, \dots$ ) were hand-tuned by researchers. Neural Architecture Search automates the choice by searching the space of depths/branches. Trade-off: NAS removes human guesswork and can find better configs, but costs enormous compute to search — you trade engineer hours for GPU hours.

Recall Quick self-test (reveal)

Break-even bottleneck formula ::: $C_{mi d} = \frac{k ^{2} C _{in} C _{o u t}}{C _{in} + k ^{2} C _{o u t}}$ Total loss with two auxiliaries ::: $L_{main} + 0.3 L_{aux1} + 0.3 L_{aux2}$ What joins the four branches ::: depth-concatenation along the channel axis Why pool branch needs a $1 \times 1$ ::: pooling keeps channel depth; the $1 \times 1$ reduces it to budget

Parent: Inception - GoogLeNet · Hinglish
Prereqs: 3.4.01-CNN-Basics · 3.4.05-VGG · 3.2.06-Vanishing-Gradients · 2.3.04-Dimensionality-Reduction
Forward: 3.4.09-ResNet · 3.4.11-Inception-v2v3 · 4.1.03-Neural-Architecture-Search