Before you can read the parent note comfortably, you need a small toolbox of ideas: what a pixel actually is, what a "feature map" is, what those H×W shapes with depth mean, why convolutions shrink images, and what "upsampling" even does. This page builds every one of them from the ground up. Nothing here assumes you have seen the notation before line one.
Figure 1 below is our reference grid — every later idea is really a statement about it.
In Figure 1, each cell is one pixel, and the yellow-outlined cell holds a stack of 3 numbers (R, G, B) drawn to its right. "Classifying a pixel" (the whole job of segmentation) means: given that little stack of numbers and its neighbours, decide which class this square belongs to — road, car, sky.
Images and the network's internal data are all rectangular stacks of numbers. We describe their size with three letters.
So (H×W×3) means "a grid H tall, W wide, with a stack of 3 numbers behind each square."
Figure 2 shows this box picture.
In Figure 2 the front face is H×W (the spatial layout you'd see), and the depth going backwards is Cch (the numbers you don't see spatially — they live "behind" each pixel). This box picture is the single most useful mental model for the entire parent note.
Between input and output the network holds many boxes, called feature maps.
When the parent says "7×7×4096 feature map," read it as: a box only 7 tall and 7 wide (very coarse — few positions left) but 4096 numbers deep (very rich — lots of learned patterns, so Cch=4096 here). That trade — small in space, huge in depth — is the heart of why segmentation is hard.
In Figure 3: the yellow 3×3 window sits on the blue input. Because it must fit entirely inside, its centre can never touch the outer border — so with no padding a 5×5 input becomes a 3×3 output. Convolutions with no padding shrink the picture by k−1 each time. This is exactly why the parent's U-Net example goes 572→568 (two 3×3 convs remove 2+2=4).
Before we write the output-size formula, one small piece of notation:
This is the aggressive downsampling the parent complains about. See Pooling Layers for the full treatment. Picture it as summarising each little 2×2 neighbourhood by its loudest response — you keep the presence of a feature but lose exactly where inside the window it was.
In Figure 4 a 2×2 input (blue) is spread out with zeros (grey) into a larger grid, then a kernel (yellow) sweeps it to fill in real numbers — the output (green) is bigger than the input. Because the kernel's weights are learned, the network discovers its own way to fill the gaps, rather than using the fixed rules of 6a.
Notice this is the ordinary-convolution formula rearranged to grow instead of shrink — that's literally why it's called transposed.
The parent's loss formula looks scary; every piece is simple.
The double sum ∑i∑j is nothing exotic — it just says "do this for every row i and every column j," i.e. every pixel in the grid from Section 1. See Loss Functions for cross-entropy in full generality.
Recall Why divide by
HW?
Averaging (not just summing) keeps the loss the same scale regardless of image size, so learning rates behave consistently across resolutions.
Every prerequisite box flows into the parent topic on the right. Notice both shrinking (conv, pool) and growing (upsampling) feed in — that push-and-pull is the U-shape.
Cover the right side and answer aloud; reveal to check.
What does a single pixel of a colour image store?
Three numbers — Red, Green, Blue intensity, each 0–255.
In (H×W×Cch), what does each letter mean?
Height (rows), Width (columns), Channels (numbers stacked behind each pixel).
Why do we distinguish Cch from Ccls?
Cch counts channels (box depth); Ccls counts classes. They're unrelated except at the final layer where box depth equals number of classes.
Why is the output label map shape (H×W) and not (H×W×C)?
Each pixel needs only ONE class-ID number, so no channel depth is required.
Why does a 3×3 valid convolution shrink a 5×5 input to 3×3?
The window's centre can't reach the border, losing k−1=2 pixels per side.
What is the difference between valid and same padding?
Valid (p=0) shrinks the output; same padding adds p=(k−1)/2 zeros so stride-1 output keeps the input size.
What does the floor function ⌊x⌋ do?
Rounds x down to the nearest whole number, since pixel counts must be integers.
What do stride s and padding p each control?
Stride = how far the window jumps (bigger stride, smaller output); padding = zero-rings added so the window reaches the edges.
What does max pooling keep and what does it throw away?
Keeps the largest value in each window (the feature's presence); throws away its exact within-window location.
Name two fixed upsampling methods and their trade-off.
Nearest-neighbour (blocky) and bilinear (smooth); both are parameter-free but can't invent lost sharp boundaries.
How does a transposed convolution enlarge a feature map, and how does it differ from fixed upsampling?
It inserts s−1 zeros then runs a LEARNED convolution; unlike fixed methods its filling is trained, not a fixed rule.
Why is "deconvolution" a misleading name for it?
It restores the SIZE only, not the lost values — it does not invert the original convolution.
What are Ei and Di in the skip formula?
Ei = encoder feature map at level i (fine detail); Di = decoder feature map at level i (deep semantics).
Difference between add and concatenate for skip connections?
Add needs equal shapes and blends into one box; concat stacks channels (Ce+Cd) and lets later layers choose.
With valid padding, why must you crop before concatenating?
Encoder boxes are slightly larger (border nibbled by convs), so they're centre-cropped (e.g. 64→56) to match the decoder size; same padding avoids this.
What does softmax do and why use the exponential?
Turns raw class scores into probabilities summing to 1; the exponential keeps everything positive, is smooth, and is differentiable for training.
In the loss, what does the double sum ∑i∑j range over?
Every pixel — all rows i and all columns j of the grid.
Why divide the loss by HW?
To average per pixel so the loss scale is independent of image size.
