Worked examples — Inception - GoogLeNet

This page is the "get your hands dirty" companion to the parent Inception note. There we learned what an inception module is and why the 1×1 bottleneck saves work. Here we grind through every kind of number the topic can throw at you, so no exam or interview question can surprise you.

Before we start, two words we will use constantly, defined from zero:

---

The scenario matrix

Every question about inception modules lands in one of these cells. The worked examples below are labelled with the cell(s) they cover. Ten cells (A–J), covered by ten worked examples.

#	Case class	What makes it tricky	Covered by
A	Plain conv cost (no bottleneck)	baseline — the number to beat	Ex 1
B	Bottleneck cost ($C_{mid}$ small)	two stages added together	Ex 2
C	Break-even / degenerate $C_{mid}$	when does the bottleneck stop helping?	Ex 3
D	$C_{mid}=C_{in}$ (identity-ish, no reduction)	bottleneck can hurt	Ex 3
E	Full module: 4 branches, depth-concat	keep spatial size, add channels	Ex 4
F	1×1 conv as pure channel mixer ($k=1$)	limiting case, no spatial extent	Ex 5
G	Pooling branch (pool then 1×1)	pooling doesn't change $C$; 1×1 does	Ex 6
H	Auxiliary-loss arithmetic (weighted sum)	the $0.3$ weights	Ex 7
I	Real-world word problem (memory/latency budget)	translate English → formula	Ex 8, Ex 9
J	Exam twist (solve for unknown $C_{mid}$)	algebra, inequality direction	Ex 10

---

[!example] Example 1 — Cell A: the naive $5\times 5$ cost

Statement. Input $28\times 28\times 256$. Apply a $5\times 5$ conv producing $32$ output channels, no bottleneck. How many operations?

Forecast: guess the order of magnitude first — thousands? millions? hundreds of millions? Write it down before reading on.

1. Count output pixels. $H\cdot W=28\times 28=784$. Why this step? Every output pixel is an independent little computation; we need to know how many there are.
2. Count weights per output pixel. The filter is $5\times 5$ over all $256$ input channels: $k^2\cdot C_{in}=25\times 256=6400$. Why this step? Each output value sums a product over the whole $5\times 5\times 256$ patch — that's $6400$ multiply-accumulates.
3. Multiply by number of output channels. We make $32$ separate output grids: $\times 32$. Why this step? Each of the $32$ filters repeats the whole thing independently.
4. Combine. $784\times 6400\times 32=160,563,200$ operations.

Verify: Units sanity — $784$ (pixels) $\times$ $6400$ (ops/pixel/channel) $\times 32$ (channels) has no leftover unit, pure op count. $\approx 1.6\times 10^8$, i.e. hundreds of millions. This is our baseline to beat.

---

[!example] Example 2 — Cell B: same job with a 1×1 bottleneck

Statement. Same input $28\times 28\times 256$ and same goal ($5\times 5$ → $32$ channels), but first squeeze the $256$ channels down to $C_{mid}=16$ with a $1\times 1$ conv.

Forecast: we added an extra layer. Will total ops go up or down? Guess a factor.

1. Stage 1: the $1\times 1$ squeeze. $k=1$ so $k^2=1$. Ops $=784\times (1\times 256\times 16)=784\times 4096=3,211,264$. Why this step? Cheaply mash $256$ channels into $16$ — a per-pixel linear combination, no spatial cost.
2. Stage 2: the $5\times 5$ on the skinny map. Now $C_{in}=16$. Ops $=784\times (25\times 16\times 32)=784\times 12800=10,035,200$. Why this step? The expensive $5\times 5$ now runs over only $16$ channels, not $256$.
3. Add the stages. $3,211,264+10,035,200=13,246,464$. Why this step? The two convs run in sequence, so total work is their sum.
4. Compare to Ex 1. $160,563,200/13,246,464\approx 12.1$.

Verify: Even though we added a layer, we cut ops by $\approx 12\times$. The whole trick: pay a tiny toll ($3.2$M) to make the giant toll ($160$M) small ($10$M). ✔ matches the parent note's $12.1\times$.

---

[!example] Example 3 — Cells C & D: when the bottleneck stops helping

The parent gave the beneficial condition: $C_{mid}(C_{in}+k^2{C}_{out})<k^2{C}_{in}{C}_{out}.$ Let's find the exact $C_{mid}$ where the trick breaks even, and show it can hurt.

Statement. $C_{in}=256,C_{out}=32,k=5$. (a) What $C_{mid}$ is the break-even point? (b) What happens at $C_{mid}=256$ (no reduction at all)?

Forecast: guess whether break-even $C_{mid}$ is closer to $16$ or to $256$.

1. Write the break-even equality. Set the two sides equal: $C_{mid}(256+25\cdot 32)=25\cdot 256\cdot 32.$ Why this step? Break-even = the $C_{mid}$ where bottleneck cost exactly equals naive cost.
2. Compute constants. $25\cdot 32=800$, so left factor $256+800=1056$. Right side $=204,800$. Why this step? Reduce to a one-variable linear equation.
3. Solve. $C_{mid}=204,800/1056\approx 193.9$. Why this step? Any $C_{mid}$ below $\approx 194$ saves ops; above it, the bottleneck costs more than naive.
4. Case D, $C_{mid}=256$. Ops $=784[(256\cdot 256)+(25\cdot 256\cdot 32)]=784[65,536+204,800]=784\times 270,336=211,943,424$. Why this step? At $C_{mid}=C_{in}$ the $1\times 1$ does no reduction, so you pay it on top of the full naive cost.

Verify: Naive cost was $160,563,200$. Case D gives $211,943,424>160,563,200$ — the "bottleneck" made things worse. ✔ This is why the paper picks tiny $C_{mid}$ like $16$: only a big squeeze pays off. (Break-even $\approx 194<256$ confirms the direction.)

---

[!example] Example 4 — Cell E: a full inception module, all four branches

Statement. Input $28\times 28\times 192$. Configure:

• Branch 1: $1\times 1$, $64$ filters.
• Branch 2: $1\times 1$ (96) → $3\times 3$ (128).
• Branch 3: $1\times 1$ (16) → $5\times 5$ (32).
• Branch 4: $3\times 3$ max-pool → $1\times 1$ (32). Find the output shape.

Forecast: guess the final channel count and whether $H,W$ change.

1. Spatial size stays $28\times 28$. Every branch uses "same" padding and pooling with stride 1, so all outputs are $28\times 28$ grids. Why this step? Concatenation (next step) requires identical $H,W$ — you can only stack grids of the same shape.
2. Read each branch's output depth. The last number in each branch is its channel count: $64,128,32,32$. Why this step? Depth = number of filters in the final conv of that branch.
3. Depth-concatenate. Stack the four boxes along the channel axis: $64+128+32+32=256$. Why this step? Concatenation glues grids together; no averaging, no loss — every branch's features survive.
4. Output shape. $28\times 28\times 256$. Why this step? Combining the unchanged spatial size (step 1) with the summed depth (step 3) gives the complete shape the next layer will receive.

Verify: $H,W$ unchanged (good — an inception module is a "same-size" block), depth grew $192\to 256$. Look at the figure: four coloured stacks of equal height/width, glued side-by-side into one taller stack. ✔

---

[!example] Example 5 — Cell F: the $1\times 1$ conv as a pure channel mixer

Statement. Input $28\times 28\times 192$, apply a $1\times 1$ conv with $64$ filters. (a) Ops? (b) What does one output pixel actually compute?

Forecast: with $k=1$, does spatial neighbourhood matter at all?

1. Ops with $k=1$. $k^2=1$, so ops $=784\times (1\times 192\times 64)=784\times 12288=9,633,792$. Why this step? No spatial extent means each output pixel only looks at the one pixel directly beneath it — but across all $192$ channels.
2. What one pixel computes. For output channel $j$ at location $(h,w)$: $y_{hwj}={\sum }_{c=1}^{192}{w}_{cj}{x}_{hwc}.$ Why this step? This is a learned weighted sum of the $192$ channel values at that spot — a mini fully-connected layer applied at every pixel.
3. Interpretation. It's dimensionality reduction with learned weights ($192\to 64$), unlike PCA which uses fixed statistical directions. Why this step? This is the whole reason 1×1 convs exist — cheap, learnable channel compression.

Verify: With ReLU after step 2 it becomes nonlinear, so it is strictly more expressive than PCA (a fixed linear map). Op count $9.6$M is tiny versus a $5\times 5$ over the same input ($25\times$ larger). ✔

---

[!example] Example 6 — Cell G: the pooling branch

Statement. Branch 4 of a module: input $28\times 28\times 192$ → $3\times 3$ max-pool (stride 1, "same") → $1\times 1$ conv with $32$ filters. What is the shape after pooling, and after the $1\times 1$?

Forecast: does max-pooling change the channel count? Guess before step 2.

1. After max-pool. "Same" stride-1 pooling keeps spatial size and does not touch depth: $28\times 28\times 192$. Why this step? Pooling picks the max within each $3\times 3$ window per channel independently — grids stay $192$, size stays $28\times 28$.
2. Why a $1\times 1$ afterwards? Pooling left us with $192$ channels, but the module only budgets $32$ for this branch. The $1\times 1$ squeezes $192\to 32$. Why this step? Without it, concatenating a $192$-channel pool branch would blow up the output depth every layer.
3. After the $1\times 1$. $28\times 28\times 32$. Ops for the $1\times 1$: $784\times (1\times 192\times 32)=784\times 6144=4,816,896$. Why this step? Confirms the branch stays cheap while still injecting pooled, translation-robust features.

Verify: Depth path $192\stackrel{\text{pool}}{\to }192\stackrel{1\times 1}{\to }32$. Matches branch 3+4 giving $32+32=64$, which together with $64+128$ gives the $256$ of Ex 4. ✔ Cross-consistent.

---

[!example] Example 7 — Cell H: auxiliary-loss arithmetic

Statement. During training GoogLeNet minimises ${\mathcal{L}}_{\text{total}}={\mathcal{L}}_{\text{main}}+0.3{\mathcal{L}}_{\text{aux1}}+0.3{\mathcal{L}}_{\text{aux2}}.$ Suppose one batch gives ${\mathcal{L}}_{\text{main}}=2.0,{\mathcal{L}}_{\text{aux1}}=2.5,{\mathcal{L}}_{\text{aux2}}=3.0$. (a) Total loss? (b) What fraction comes from the auxiliaries?

Forecast: guess whether the auxiliaries contribute more or less than half.

1. Weighted sum. ${\mathcal{L}}_{\text{total}}=2.0+0.3(2.5)+0.3(3.0)=2.0+0.75+0.90=3.65$. Why this step? The $0.3$ down-weights the auxiliaries so they nudge, not dominate.
2. Auxiliary fraction. $(0.75+0.90)/3.65=1.65/3.65\approx 0.452$. Why this step? Shows they are meaningful but sub-half — deliberately kept below the main signal.
3. Interpretation. These extra losses shove gradient into the middle of the net, fighting vanishing gradients; they are removed at test time. Why this step? Ties the number back to the purpose — they are a training crutch, later replaced by ResNet skip-connections.

Verify: Total $3.65>{\mathcal{L}}_{\text{main}}=2.0$ (auxiliaries add signal). Auxiliary fraction $0.452<0.5$ (they never outvote the main task). ✔ Consistent with "regularizer, not primary objective".

---

[!example] Example 8 — Cell I: real-world budget word problem

Statement. You must run a $5\times 5$ conv, $256$-channel input, $64$ outputs, on a $14\times 14$ map, on a tiny always-on sensor chip that can do only $3\times 10^7$ ops per frame. Show the naive conv won't fit, then pick the largest $C_{mid}$ (a $1\times 1$ bottleneck) that does.

Forecast: will $C_{mid}$ need to be tiny (like 8) or can you afford ~100?

1. Naive cost, confirm it fails. $196\times (25\times 256\times 64)=196\times 409600=80,281,600\approx 8.0\times 10^7$. Why this step? $8.0\times 10^7>3\times 10^7$ — the naive conv is nearly $3\times$ over budget, so we must use a bottleneck.
2. Bottleneck cost as function of $C_{mid}$. $196[(256C_{mid})+(25C_{mid}64)]=196C_{mid}(256+1600)=196\times 1856C_{mid}=363,776C_{mid}$. Why this step? Express cost as a straight line in $C_{mid}$ so we can solve an inequality.
3. Solve $363,776C_{mid}\le 3\times 10^7$. $C_{mid}\le 3\times 10^7/363,776\approx 82.5$. Why this step? Round down: largest integer that fits is $C_{mid}=82$.
4. Check the winner. Cost $=363,776\times 82=29,829,632\approx 2.98\times 10^7$, just under the $3\times 10^7$ ceiling. Why this step? Confirms the chosen $C_{mid}$ actually fits — always plug the integer back before committing.

Verify: $29,829,632<3\times 10^7<80,281,600$: bottleneck fits, naive does not. ✔ $C_{mid}=83$ would give $363,776\times 83=30,193,408>3\times 10^7$ — over budget, confirming $82$ is the boundary. Method (solve linear inequality in $C_{mid}$) generalises to any chip limit; in practice these picks are automated by Neural Architecture Search.

---

[!example] Example 9 — Cell I: a comfier budget (design choice, not the maximum)

Statement. Same $5\times 5$, $256\to 64$, on a $14\times 14$ map — but now on a phone GPU with a generous $3\times 10^8$ ops/frame budget. Both naive and bottleneck fit. Pick a sensible $C_{mid}$ and quantify the saving.

Forecast: when everything fits, do you still bottleneck? Guess a factor.

1. Confirm both fit. Naive $=80,281,600\approx 8.0\times 10^7<3\times 10^8$. Plenty of headroom. Why this step? When the budget is loose, cost is no longer the hard constraint — accuracy and speed trade off.
2. Pick $C_{mid}=64$ (a $4\times$ squeeze). Bottleneck cost $=363,776\times 64=23,281,664\approx 2.3\times 10^7$. Why this step? $64$ is a common design choice — a moderate squeeze that keeps enough capacity while cutting work.
3. Saving factor. $80,281,600/23,281,664\approx 3.45$. Why this step? Even with a loose budget, the bottleneck still runs $\approx 3.4\times$ cheaper — free speed at little accuracy cost.

Verify: $23,281,664<80,281,600$: bottleneck cheaper even when both fit. Saving $\approx 3.45\times$. ✔ Lesson: the bottleneck is worth it for speed, not only to squeeze under a hard ceiling.

---

[!example] Example 10 — Cell J: exam twist, solve for the unknown

Statement. A $3\times 3$ conv, $C_{in}=192$, $C_{out}=128$. You want the bottleneck version to cost at most half the naive version. What is the largest $C_{mid}$?

Forecast: half the cost — will $C_{mid}$ be roughly half of $C_{in}$, or much smaller?

1. Naive cost per pixel. $k^2{C}_{in}{C}_{out}=9\times 192\times 128=221,184$ (drop $HW$, it cancels). Why this step? Both sides share $HW$, so ratios ignore spatial size entirely.
2. Bottleneck cost per pixel. $C_{in}{C}_{mid}+k^2{C}_{mid}{C}_{out}=C_{mid}(192+9\times 128)=C_{mid}(192+1152)=1344C_{mid}$. Why this step? Factor $C_{mid}$ to isolate the unknown.
3. Impose "at most half". $1344C_{mid}\le \frac{1}{2}\times 221,184=110,592$. Why this step? Translate "half the cost" into an inequality — mind the direction ($\le$, not $=$).
4. Solve. $C_{mid}\le 110,592/1344=82.28\dots$ → largest integer $C_{mid}=82$. Why this step? Must round down; $C_{mid}=83$ would exceed half.

Verify: At $C_{mid}=82$: cost $=1344\times 82=110,208\le 110,592$ ✔. At $C_{mid}=83$: $1344\times 83=111,552>110,592$ ✘. So $82$ is exactly the boundary. ✔

---

Recall Self-test

Naive $5\times 5$ over $28\times 28\times 256$ to $32$ ch: op count? ::: $160,563,200$ With $C_{mid}=16$ bottleneck, total ops? ::: $13,246,464$ (≈12.1× fewer) Does max-pooling change the channel count? ::: No — it acts per channel; only the following $1\times 1$ changes depth A $1\times 1$ conv computes what, at each pixel? ::: A learned weighted sum over all input channels ($==$ per-pixel dimensionality reduction) Why the $0.3$ weight on auxiliary losses? ::: So they regularize / feed gradient without dominating the main objective Break-even $C_{mid}$ for $256\to 32$, $k=5$? ::: ≈$194$; below it saves, above it costs more

See also: 3.4.01-CNN-Basics · 3.4.05-VGG (uniform filters, the thing Inception rebels against) · 3.4.11-Inception-v2v3 (drops the auxiliaries) · 3.4.08 Inception - GoogLeNet (Hinglish).