Foundations — Inception - GoogLeNet

Before you can read the parent note you need to own every piece of notation it throws at you. We build them in order — each one uses only the ones before it.

1. What is a feature map? The $H\times W\times C$ box

Everything in a CNN is a box of numbers. When we write $28\times 28\times 192$, we mean:

• $H$ = height = how many rows of numbers (here 28)
• $W$ = width = how many columns (here 28)
• $C$ = channels = how many such 28×28 grids are stacked behind each other (here 192)

Look at the figure: the front face is the spatial part ($H\times W$), the depth into the page is the channel count $C$. When the parent says "output $28\times 28\times 64$", it just means this box got 64 grids deep.

Why the topic needs it: every branch of an Inception module produces one of these boxes, and the final step stacks them. You cannot understand "concatenate along depth" without first seeing the box.

Prerequisite: this is the language of CNN basics.

2. What a convolution actually does

A convolution slides a small window (a filter or kernel) across the feature map and, at each position, multiplies-and-adds to produce one output number.

• $k$ = the kernel size (the side length of the window). $k=3$ means a 3×3 window.
• $C_{in}$ = channels the filter reads from (input depth).
• $C_{out}$ = number of filters = channels it writes out (output depth).

In the figure the yellow 3×3 window sits on the input; every position it visits produces one number in the output grid (blue). Because the window has size $k$, it "sees" a $k\times k$ neighbourhood — this is the receptive field.

3. The special case $k=1$: the 1×1 convolution

Set $k=1$. Now the window is a single pixel deep in space but still reaches through all $C_{in}$ channels.

The symbol $\sum$ (capital Greek "sigma") just means "add up all the terms" — here, add up over every channel $c$ from $1$ to $C_{in}$.

Look at the red arrow drilling straight through the channel stack at one pixel: it collapses $C_{in}$ numbers into $C_{out}$ numbers. This is why we call it a bottleneck — if $C_{out}<C_{in}$, the box gets thinner, and that is dimensionality reduction with weights the network learns (more flexible than fixed methods like PCA).

Why the topic needs it: the 1×1 conv is the single trick that makes Inception affordable. Everything in the cost analysis rests on it.

4. Counting the work: what "operations" means

The parent multiplies big numbers to compare costs. Here is where those numbers come from.

Each output value of a $k\times k$ conv requires $k^2\cdot C_{in}$ multiply-adds (one per number the window covers). We produce $H\cdot W$ such outputs per filter, and $C_{out}$ filters:

Plug $k=1$ and you recover the 1×1 cost $H\cdot W\cdot C_{in}\cdot C_{out}$ — the $k^2=1$ term simply vanishes. That is why 1×1 convs are cheap: no $k^2$ factor.

Why the topic needs it: the entire "12× fewer operations" argument is this one formula applied twice.

5. Depth-concatenation: stacking the branches

Four branches each hand back a box of the same $H\times W$ but possibly different depth. To combine them:

The spatial sizes must match for this to be legal, which is why every branch is designed to keep $28\times 28$ (or whatever $H\times W$ the module lives at).

6. Loss and gradient — the training symbols

The parent writes ${\mathcal{L}}_{\text{total}}={\mathcal{L}}_{\text{main}}+0.3{\mathcal{L}}_{\text{aux1}}+0.3{\mathcal{L}}_{\text{aux2}}$.

• $\mathcal{L}$ (script "L") = the loss = a single number measuring how wrong the network's prediction is. Smaller is better.
• Training nudges every weight to lower $\mathcal{L}$ using the gradient (the direction of steepest decrease).
• In very deep nets the gradient signal shrinks as it travels backward — the vanishing gradient problem. The extra $0.3{\mathcal{L}}_{\text{aux}}$ terms inject fresh gradient into the middle, like a booster halfway down a long pipe.

The $0.3$ is a weight: it makes the helper losses count 30% as much as the main one, so they assist without dominating.

Why the topic needs it: the auxiliary classifiers exist only to fix vanishing gradients — later solved more cleanly in ResNet.

7. How it all feeds together

Related roads from here: earlier fixed-size stacks in VGG, the successor Inception v2/v3, and automating the hand-tuned channel counts with Neural Architecture Search.

Equipment checklist

Cover the right side and answer aloud.