Semantic segmentation (U-Net, FCN): Exercises

This page is a self-test ladder for Semantic segmentation (U-Net, FCN). Cover the solution, try the problem, then reveal. Each problem is labelled L1 → L5 by cognitive demand. Every symbol used below is defined the moment it appears — nothing is assumed.

Before we begin, one shared picture of the shapes we will juggle all page long:

Figure — Semantic segmentation (U-Net, FCN)

Level 1 — Recognition

L1.1 — Detection vs. segmentation output shape

An image is $256 \times 256 \times 3$ . You run (a) an object detector and (b) a semantic segmentation network with $C = 5$ classes. What is the shape of the output each one produces?

Recall Solution

Detector: a short list of boxes, each box being $(x, y, w, h, class)$ — a handful of numbers, not tied to image size. It says "there is a thing around here."
Segmentation: a dense label map the same height and width as the input, $256 \times 256$ , where every entry is a class ID in ${0, 1, 2, 3, 4}$ . Before the argmax it is $256 \times 256 \times 5$ logits.

The key recognition: segmentation output resolution matches the input; detection output does not. See Object Detection for the box-based alternative.

L1.2 — Add or concatenate?

Which architecture adds skip features and which concatenates them: FCN or U-Net?

Recall Solution

FCN uses element-wise addition: $D + E$ , both must have the same channel count $C$ .
U-Net uses concatenation along the channel axis: stacking $E$ (encoder, $C_{e}$ channels) on top of $D$ (decoder, $C_{d}$ channels) to make $C_{e} + C_{d}$ channels.

Mnemonic below.

Level 2 — Application

L2.1 — Transposed convolution output size

Why this formula and not the normal-conv one? A normal convolution shrinks a map: it slides a window and outputs fewer positions. Transposed convolution answers the opposite question — "what input size, run forward, would have produced this output?" — so we solve the forward-conv shape equation backwards. Geometrically it inserts $s - 1$ zeros between every pair of input pixels (dilating the grid), then does an ordinary convolution over that inflated grid. The picture:

Problem: input $H_{in} = 7$ , kernel $k = 4$ , stride $s = 2$ , padding $p = 1$ . Compute $H_{o u t}$ .

Recall Solution

Substitute directly: $H_{o u t} = 2 \cdot (7 - 1) + 4 - 2 \cdot 1 = 2 \cdot 6 + 4 - 2 = 12 + 2 = 14.$ So $7 \times 7 \to 14 \times 14$ : the map exactly doubled, which is the standard "upsample ×2" step of a decoder. That is why a $k = 4, s = 2, p = 1$ transposed conv is the everyday choice.

L2.2 — FCN-32s one-shot upsample

FCN-32s takes a $7 \times 7$ score map and upsamples it back to a $224 \times 224$ input in one transposed convolution with stride $s = 32$ . If padding $p = 0$ , what kernel $k$ makes $H_{o u t} = 224$ exactly?

Recall Solution

Set $H_{o u t} = 224$ , $H_{in} = 7$ , $s = 32$ , $p = 0$ in $H_{o u t} = s (H_{in} - 1) + k - 2 p$ : $224 = 32 \cdot (7 - 1) + k - 0 = 192 + k \Rightarrow k = 32.$ A kernel of $32$ makes sense: to "smear" one coarse pixel over a $32 \times 32$ block you need a filter that wide. This is why FCN-32s outputs are blurry — one $7 \times 7$ decision is spread over the whole image.

L2.3 — Downsampling count

A U-Net encoder halves spatial size at each $2 \times 2$ max-pool (see Pooling Layers). Starting from $512 \times 512$ , after how many pool steps do you reach $32 \times 32$ ?

Recall Solution

Each pool divides by $2$ . We need $512/ 2^{n} = 32$ : $2^{n} = 512/32 = 16 \Rightarrow n = 4.$ Four pooling stages. (Check: $512 \to 256 \to 128 \to 64 \to 32$ , four arrows. ✓)

Level 3 — Analysis

L3.1 — Why crop in the original U-Net?

The original U-Net uses valid (no-padding) $3 \times 3$ convolutions. At one skip level the encoder map is $64 \times 64$ but the decoder's upsampled map is $56 \times 56$ . Explain why they differ and compute the crop.

Recall Solution

A valid $3 \times 3$ conv removes a $1$ -pixel border on each side (a $3 \times 3$ window can't be centred on the outer ring), so each conv shrinks the side by $2$ . Two convs per block ⇒ shrink by $4$ . Because encoder and decoder go through different numbers of these shrinking convs before meeting, their sizes drift apart.

To concatenate they must match. Crop the encoder $64 \times 64$ down to $56 \times 56$ by removing $(64 - 56) /2 = 4$ pixels from every side (a centre crop). Now both are $56 \times 56$ and can be stacked.

Modern U-Nets add padding $p = 1$ to $3 \times 3$ convs so nothing shrinks and no crop is needed — this keeps input and output the same size.

L3.2 — Channels after concatenation

At a decoder level, the encoder skip feature is $56 \times 56 \times 512$ and the decoder's upsampled feature is $56 \times 56 \times 512$ . What is the shape after the U-Net concatenation? Then a $3 \times 3$ conv with $512$ output filters follows — what shape is its output (with padding)?

Recall Solution

Concatenation stacks channels: $512 + 512 = 1024$ , giving $56 \times 56 \times 1024$ . Using the concatenation rule $Skip = Concat (E, D) \in R^{H \times W \times (C_{e} + C_{d})},$ with $C_{e} = C_{d} = 512$ . The following $3 \times 3$ conv (padded) keeps $H, W$ and sets channels to its filter count: output $56 \times 56 \times 512$ . So concatenation doubles channels and the very next conv compresses them back — that conv is where the two paths are actually fused.

L3.3 — Receptive field intuition

Why does the U-Net bottleneck (smallest, deepest map) carry the most semantic information despite having the fewest pixels?

Recall Solution

Each pooling stage doubles the region of the original image that one deep pixel "sees" — its receptive field. After 4 pools a single bottleneck pixel summarises a large patch, so it can answer "what object is here" (needs global context) but not "where exactly is the boundary" (needs local detail, long since pooled away). That's precisely why skip connections re-inject the shallow, high-resolution encoder features into the decoder — deep = what, shallow = where. Related idea in Residual Networks (ResNet), where skips instead help gradient flow.

Level 4 — Synthesis

L4.1 — Pixel-wise cross-entropy by hand

Problem: a tiny $1 \times 1$ image ( $H = W = 1$ ), $C = 3$ classes. Logits at the single pixel are $\hat{Y} = [2.0, 1.0, 0.1]$ and the true class is $Y = 0$ . Compute the loss $L$ (natural log). Round to 3 decimals.

Recall Solution

Step 1 — softmax denominator (WHY: normalises scores into probabilities). $Z = e^{2.0} + e^{1.0} + e^{0.1} = 7.389 + 2.718 + 1.105 = 11.212.$ Step 2 — probability of the true class $0$ . $p_{0} = \frac{e ^{2.0}}{Z} = \frac{7.389}{11.212} = 0.659.$ Step 3 — loss for the one pixel (WHY: $- lo g$ of the correct-class prob; the $\frac{1}{H W} = 1$ here). $L = - lo g (0.659) = 0.417.$ The model is only $66%$ sure of the right class, so it pays a moderate $0.417$ nats — not zero, not huge.

L4.2 — Weighted loss for class imbalance

Background is $95%$ of pixels, cells are $5%$ . To stop the network from lazily predicting "all background," we use weighted cross-entropy $L_{w} = - \sum_{i, j} w (Y_{i, j}) lo g P (\hat{Y}_{i, j} = Y_{i, j})$ (see Loss Functions). A common choice sets each class weight inversely proportional to its frequency. Compute the weights for background and cell, then normalise so the smaller weight equals $1$ .

Recall Solution

Inverse-frequency weights (WHY: rare classes get amplified so each contributes equally overall). $w_{bg} \propto \frac{1}{0.95} = 1.053, w_{cell} \propto \frac{1}{0.05} = 20.0.$ Normalise so the smallest = 1: divide both by $1.053$ : $w_{bg} = 1.0, w_{cell} = \frac{20.0}{1.053} = 19.0.$ A misclassified cell pixel now costs $19 \times$ a misclassified background pixel, forcing the network to care about the tiny structures. Pair this with heavy Data Augmentation since medical datasets are small.

L4.3 — Design an FCN-8s fusion path

You have VGG feature maps: pool3 = $28 \times 28$ , pool4 = $14 \times 14$ , conv7 = $7 \times 7$ , all for a $224 \times 224$ input with $C = 21$ classes. Write the ordered sequence of upsample-and-add steps that yields a $224 \times 224 \times 21$ output.

Recall Solution

$1 \times 1$ conv on conv7 → $7 \times 7 \times 21$ score map.
Transposed conv ×2 → $14 \times 14 \times 21$ ; add to ( $1 \times 1$ -conv'd) pool4 ( $14 \times 14 \times 21$ ).
Transposed conv ×2 → $28 \times 28 \times 21$ ; add to ( $1 \times 1$ -conv'd) pool3 ( $28 \times 28 \times 21$ ).
Transposed conv ×8 → $224 \times 224 \times 21$ final prediction.

Check the total factor: $2 \times 2 \times 8 = 32 = 224/7$ . ✓ The three fusions inject progressively finer detail before each blow-up, so edges stay crisp. The $1 \times 1$ convs exist only to force every skip source to exactly $C = 21$ channels so addition is legal.

Level 5 — Mastery

L5.1 — Trace a full U-Net level end to end

A padded (modern) U-Net decoder receives a $28 \times 28 \times 1024$ bottleneck. Trace one decoder level: (a) transposed conv ×2 halving channels, (b) concat with the matching $56 \times 56 \times 512$ encoder skip, (c) two padded $3 \times 3$ convs each with $512$ filters. Give the shape after each sub-step, and the channel count just before the fusion conv.

Recall Solution

(a) Transposed conv $k = 4, s = 2, p = 1$ : $28 \times 28 \times 1024 \to 56 \times 56 \times 512$ (spatial doubled, channels halved — the decoder's job).
(b) Concat with encoder $56 \times 56 \times 512$ : channels $512 + 512 = 1024$ ⇒ $56 \times 56 \times 1024$ . This is the count just before the fusion conv: 1024.
(c) First padded $3 \times 3$ conv, $512$ filters: $56 \times 56 \times 512$ . Second padded $3 \times 3$ conv, $512$ filters: $56 \times 56 \times 512$ .

End of level: $56 \times 56 \times 512$ , ready to feed the next (higher-resolution) decoder level. Notice the symmetry — every encoder resolution has an exactly mirrored decoder resolution; that mirror is the "U."

L5.2 — Why concatenation beats addition for tiny structures (argue it)

In one paragraph, using the information-preservation idea, justify U-Net's concatenation over FCN's addition specifically for microscopy cell membranes.

Recall Solution

Addition computes $E + D$ : if the encoder channel encodes "thin bright membrane" and the decoder channel encodes "dark cytoplasm," their sum is a single blurred number and the membrane signal can be cancelled or drowned. Concatenation keeps $E$ and $D$ as separate channels, so the fusion conv can learn a weight that says "trust the encoder's membrane channel here, ignore the decoder's." No information is destroyed before the network gets to decide. For hair-thin membranes that occupy a couple of pixels, that non-destructive choice is the difference between detecting them and losing them. (This "let the network choose" theme also underlies Attention Mechanisms.)

L5.3 — Consistency check across the whole page

Verify that FCN-8s's fusion factors ( $2, 2, 8$ ) and FCN-32s's single factor ( $32$ ) both recover $224$ from $7$ , i.e. that both nets are internally consistent.

Recall Solution

FCN-32s: $7 \times 32 = 224$ . ✓ FCN-8s: $7 \times 2 \times 2 \times 8 = 224$ . ✓ Both reach the input resolution; FCN-8s just distributes the $\times 32$ across three learnable steps with skip fusions in between, so its output is sharper for the same final size. This is the whole point of the FCN family: same total upsampling, more intermediate detail = better boundaries.

Recall Self-test summary (cover and recite)

Transposed-conv size formula ::: $H_{o u t} = s (H_{in} - 1) + k - 2 p$ FCN combines skips by ::: element-wise addition (channels must match, so $1 \times 1$ conv first) U-Net combines skips by ::: concatenation along channels (no channel match needed) Why the U-Net bottleneck is "semantic not spatial" ::: pooling grew the receptive field but destroyed fine location Weighted CE weight rule for imbalance ::: weight $\propto 1/ class frequency$ Cropping is needed only when ::: valid (no-padding) convolutions cause size drift