Visual walkthrough — Inception - GoogLeNet

This page rebuilds the single most important idea in Inception/GoogLeNet from the ground up: why placing a tiny 1×1 convolution before a big 5×5 convolution slashes the cost by roughly 12× — without throwing away the network's power.

We assume you know nothing beyond "an image is a grid of numbers." Every symbol is earned before it is used. If you want the CNN groundwork first, see 3.4.01-CNN-Basics.

---

Step 1 — What a feature map actually is

Three numbers describe this stack:

$$\underset{\text{height}}{\underbrace{H}}\times \underset{\text{width}}{\underbrace{W}}\times \underset{\text{channels = number of sheets}}{\underbrace{C}}$$

• $H$ — how many rows of pixels (the picture is this tall).
• $W$ — how many columns of pixels (this wide).
• $C$ — how many sheets are stacked (this deep). Each sheet is called a channel.

In the parent's example the input is $28\times 28\times 192$: a $28\times 28$ picture, $192$ pattern-sheets deep.

---

Step 2 — What one convolution filter costs to run

WHAT. A filter of size $k\times k$ is a little cube of learned numbers. We slide it over the input; at every position it multiplies-and-adds to produce one output number.

WHY count multiplications? Because a computer's time is dominated by how many multiply-adds it does. If we can count those, we can compare designs before building them.

PICTURE. Look at the sliding cube below. At one landing spot it covers a $k\times k$ patch across all $C_{in}$ input channels, so it does $k^2\times C_{in}$ multiplications to make one output value.

Reading it term by term:

• $H\cdot W$ — the filter lands once per output pixel, so this many spots.
• $k^2$ — at each spot it reads a $k\times k$ square (e.g. $5\times 5=25$).
• $C_{in}$ — it reads that square on every input sheet.
• $C_{out}$ — we run that many different filters to build $C_{out}$ output sheets.

Everything from here is just plugging numbers into this one formula.

---

Step 3 — The naive 5×5: measuring the explosion

WHAT. Take the parent's scenario. Input $28\times 28\times 256$. We want a $5\times 5$ conv making $32$ output sheets. Plug straight into Step 2.

WHY. To feel how bad the expensive path is, we need its exact operation count as a baseline.

$${\text{Ops}}_{\text{naive}}=\underset{784}{\underbrace{28\cdot 28}}\times \underset{25}{\underbrace{5^2}}\times \underset{C_{in}}{\underbrace{256}}\times \underset{C_{out}}{\underbrace{32}}$$ $$=784\times 25\times 256\times 32=160,563,200\text{ ops}$$

PICTURE. The pink block below is that number — over 160 million multiply-adds for a single layer. The villain is that $C_{in}=256$ sits inside the big $\times k^2$ multiplication: every one of the 25 patch-cells is read across all 256 sheets.

The insight that fixes this: what if we shrink $256$ down to a small number first, cheaply, and only then pay for the $5\times 5$?

---

Step 4 — The 1×1 conv: a thinning valve that reads no neighbours

WHAT. A 1×1 convolution is Step 2 with $k=1$. Its patch is a single pixel — it looks at no neighbours, only at the tower of channel-values standing over one pixel.

WHY this tool and not another? We need something that changes the channel count ($C$) while leaving the picture size ($H,W$) untouched, and does it cheaply. A 1×1 conv is exactly that: because $k^2=1$, its cost has no $k^2$ blow-up. It is the only conv that reshapes depth without touching space.

PICTURE. At one pixel it takes the $C_{in}$ numbers stacked above it and forms a weighted sum — a learned recipe — to produce each new channel.

Its cost (from Step 2 with $k=1$):

$${\text{Ops}}_{1\times 1}=H\cdot W\cdot 1^2\cdot C_{in}\cdot C_{mid}=H\cdot W\cdot C_{in}\cdot C_{mid}$$

---

Step 5 — Two cheap layers instead of one expensive one

WHAT. Replace the naive path with a valve then the big conv:

1. $1\times 1$ conv: shrink $256\to 16$ channels.
2. $5\times 5$ conv: work on just $16$ channels, produce $32$.

WHY. Because the $5\times 5$'s cost (Step 2) is proportional to $C_{in}$. If we hand it $C_{in}=16$ instead of $256$, we cut its bill by $16\times$ — and the valve itself is nearly free.

PICTURE. Watch the fat 256-deep block get squeezed through the narrow 16-deep neck before the expensive $5\times 5$ ever touches it.

Step 1 — the valve:

$$\underset{784}{\underbrace{28\cdot 28}}\cdot \underset{C_{in}}{\underbrace{256}}\cdot \underset{C_{mid}}{\underbrace{16}}=3,211,264$$

Step 2 — the big conv on the thinned stack (note $C_{in}$ is now $16$):

$$\underset{784}{\underbrace{28\cdot 28}}\cdot \underset{5^2}{\underbrace{25}}\cdot \underset{C_{mid}}{\underbrace{16}}\cdot \underset{C_{out}}{\underbrace{32}}=10,035,200$$

Add them:

$${\text{Ops}}_{\text{bottleneck}}=3,211,264+10,035,200=13,246,464\text{ ops}$$

---

Step 6 — The payoff, and the rule for when it pays

WHAT. Divide the naive cost by the bottleneck cost.

$$\frac{160,563,200}{13,246,464}\approx 12.1\times \text{ fewer ops}$$

WHY it works algebraically. Write both costs and cancel the shared $H\cdot W$. The bottleneck wins when:

$$\underset{\text{valve}}{\underbrace{C_{in}{C}_{mid}}}+\underset{\text{big conv}}{\underbrace{k^2{C}_{mid}{C}_{out}}}<\underset{\text{naive}}{\underbrace{k^2{C}_{in}{C}_{out}}}$$

Factor $C_{mid}$ out of the left:

$$C_{mid}(C_{in}+k^2{C}_{out})<k^2{C}_{in}{C}_{out}$$

The left grows linearly in the small number $C_{mid}$; the right is a fixed big product. So as long as $C_{mid}$ is chosen small, the left stays under the right. Plugging $k=5,C_{in}=256,C_{out}=32,C_{mid}=16$:

$$16(256+25\cdot 32)=16(256+800)=16,896\ll 25\cdot 256\cdot 32=204,800$$

PICTURE. Two bars: the towering naive cost vs. the short bottleneck cost — same job done, one twelfth of the work.

---

Step 7 — The degenerate cases (never let the reader get surprised)

$k=1$ (the valve applied to itself). Then $k^2=1$ and the inequality becomes $C_{mid}(C_{in}+C_{out})<C_{in}{C}_{out}$ — usually false for tiny channel counts, so you'd never bottleneck a 1×1 conv. There is nothing to save; a 1×1 is already the cheapest conv. That's exactly why Branch 1 of the module is a bare 1×1.

$C_{mid}=C_{in}$ (no shrinking). The valve degenerates to a full-width layer. Cost becomes $C_{in}^2+k^2{C}_{in}{C}_{out}$, which is larger than naive $k^2{C}_{in}{C}_{out}$. Confirms Step 6's warning: zero savings, pure waste.

$C_{mid}=0$ (channel wiped out). The neck passes nothing; every downstream feature is zero. This is the limiting worst case of over-squeezing — the network loses all information. Real designs keep $C_{mid}$ comfortably above zero.

The pooling branch. Branch 4 pools first (no learned cost), then a 1×1 trims channels so the concatenation doesn't balloon. Same valve idea, different order.

Recall Check yourself on the edge cases

Why don't we bottleneck a 1×1 conv? ::: A 1×1 has $k^2=1$, so there is no $k^2$ blow-up to tame — it's already minimal. Adding a valve would only add cost. What happens if $C_{mid}=C_{in}$? ::: No shrinking occurs; total cost exceeds naive, so you've wasted a layer. What breaks if $C_{mid}=0$? ::: The neck passes zero channels; all information is destroyed downstream.

---

The one-picture summary

The entire derivation in one image: the fat input meets two roads. The naive road (pink) drags all 256 channels through a $5\times 5$ conv → 160M ops. The Inception road (blue) squeezes to 16 channels with a nearly-free 1×1 valve, then runs the $5\times 5$ → 13M ops. Same output shape, 12× less work. Stack four such branches side by side and depth-concatenate their outputs — that is one Inception module.

Recall Feynman retelling — say it back in plain words

A feature map is a stack of pattern-sheets: tall, wide, and deep. A convolution slides a little cube over it, and the number of multiply-adds is how many landing spots times patch area times sheets read in times sheets made out. A big $5\times 5$ conv over a very deep stack is brutal, because the deep count sits inside the big patch multiplication — 160 million operations for one layer. The trick: a $1\times 1$ conv looks at no neighbours, only at the tower of numbers over a single pixel, and mixes them into fewer channels. So we thin the stack from 256 down to 16 with this cheap valve, then run the $5\times 5$ on the skinny stack. The valve is almost free, the big conv now reads $16$ instead of $256$, and the total drops to 13 million — twelve times cheaper. The catch: the neck must be narrow; if you don't shrink much, you pay more, not less. Run four such thinned branches in parallel (1×1, 3×3, 5×5, pooling) and glue their outputs together by depth — you've built one Inception module, and you can afford to stack nine of them. Compare with VGG's uniform stacks and ResNet's skip connections to see the design space; the vanishing-gradient worry that motivated the auxiliary heads is covered in 3.2.06-Vanishing-Gradients, and the refinements live in Inception v2/v3.