Semantic segmentation (U-Net, FCN): Visual walkthrough

The parent note told you that to go from a tiny $7 \times 7$ feature map back up to a full $224 \times 224$ label map we use a transposed convolution, and it handed you the size formula

$H_{o u t} = s \cdot (H_{in} - 1) + k - 2 p$

as if it fell from the sky. This page derives that formula from absolute zero — no matrices assumed, no "deconvolution" hand-waving. By the end you will be able to draw the whole upsampling operation on graph paper and predict the output size in your head. Everything is grounded in one running numerical example so nothing is abstract.

Prerequisites we lean on but re-explain as needed: a normal convolution's sliding window (from Pooling Layers you already know a window sliding across a grid) and the idea of a learned kernel.

Step 1 — Remember what an ordinary convolution does to size

WHAT. We slide a small square of numbers (the kernel, size $k \times k$ ) across the input grid. Each placement multiplies overlapping numbers and adds them into one output number. The stride $s$ is how many cells we jump between placements. The padding $p$ is how many rings of zeros we glue around the input first.

WHY. We start here because the transposed convolution is defined as the exact reverse of this motion. To reverse something, you must first see it clearly forwards.

PICTURE. In the figure below the input is a $5 \times 5$ grid (blue). A $3 \times 3$ kernel (yellow) slides with stride $s = 1$ , no padding. Count the landing spots along one row: the kernel's left edge can sit at columns $1, 2, 3$ — that is 3 landing spots, so the output is $3 \times 3$ (pink).

Figure — Semantic segmentation (U-Net, FCN)

Notice the direction: input $5 \to$ output $3$ , it got smaller. Transposed convolution is going to run this arrow backwards.

Step 2 — Invert the arrow: from output back to input

WHAT. Solve the forward formula for $H_{in}$ instead of $H_{o u t}$ . If a forward conv turns a size- $n$ grid into a size- $m$ grid, the transposed conv is the operation that turns size $m$ back into size $n$ .

WHY. This is the whole trick in one line. "Transposed convolution" just means: take the shrinking motion and read the size relationship in reverse. We are not inventing a new formula — we are algebraically flipping the old one.

PICTURE. Same two grids as Step 1, but now the arrow points up: the small pink $3 \times 3$ is the input, the big blue $5 \times 5$ is what we want to produce.

Take the forward formula and drop the floor for the moment (we return to it in Step 7). With $p = 0$ :

$m = \frac{n - k}{s} + 1 ⟹ n = s (m - 1) + k$

Rename to match the transposed view — the small grid is now the input $H_{in}$ , the big grid is the output $H_{o u t}$ :

$H_{o u t} = s (H_{in} - 1) + k - 2 p$

That is exactly the parent's formula. The rest of this page shows what physical motion produces it, term by term.

Step 3 — The $s (H_{in} - 1)$ term: spread the input apart

WHAT. Take the input grid and insert $s - 1$ rows and columns of zeros between every pair of neighbouring cells. Do not pad the outside yet — only the internal gaps.

WHY. The forward stride $s$ skipped cells; to undo skipping we must re-create the skipped space as empty (zero) cells. This is why the term counts gaps, $H_{in} - 1$ of them, each widened by $s$ .

PICTURE. Below: a $3 \times 3$ pink input becomes a $5 \times 5$ dilated grid when $s = 2$ (one zero inserted in each internal gap). The chalk-blue cells are the original values; the dim cells are inserted zeros.

Count the new side length: $3$ original cells with $2$ internal gaps, each gap holding $s - 1 = 1$ zero:

$3 + (3 - 1) \cdot (s - 1) = 3 + 2 \cdot 1 = 5 = s (H_{in} - 1) + 1.$

Step 4 — The $+ k$ term: now slide a stride-1 kernel over the spread grid

WHAT. Over the zero-spread grid from Step 3, run an ordinary stride-1 convolution with a learnable $k \times k$ kernel — but glue $k - 1$ zeros of outer padding so the kernel can hang off both edges.

WHY. Spreading (Step 3) only makes holes; something must fill them. The learnable kernel smears each input value into a $k \times k$ neighbourhood, painting real numbers into the zero holes. This is the "learned interpolation" — the network learns the best brush, unlike fixed bilinear upsampling.

PICTURE. The yellow kernel slides one cell at a time over the spread-out blue/dim grid; each landing sums a $3 \times 3$ patch into one output pixel of the growing green result.

Putting Step 3 and Step 4 together with no outer padding ( $p = 0$ ): the dilated grid has side $s (H_{in} - 1) + 1$ , and a stride-1 kernel of size $k$ adds $k - 1$ to the side (the kernel sticking out on the far edge):

$H_{o u t} = [s (H_{in} - 1) + 1] + (k - 1) = s (H_{in} - 1) + k .$

There is our formula with $p = 0$ . Every symbol now has a picture behind it.

Step 5 — The $- 2 p$ term: padding runs backwards too

WHAT. In the forward conv, padding $p$ added a ring of zeros, which increased the output. Running backwards, the same $p$ must shrink the transposed output — so it enters with a minus sign and doubled (one ring per side): $- 2 p$ .

WHY. Consistency. If you agree the transposed conv is the size-inverse of a forward conv that used padding $p$ , then whatever padding gave you in the forward direction it must take away in reverse. Symmetry forces the sign flip.

PICTURE. Left board: forward conv with a padding ring makes the output one cell bigger. Right board: the transposed conv with the same $p$ crops that same ring, making the output one cell smaller.

$H_{o u t} = s (H_{in} - 1) + k - 2 p$

$s (H_{in} - 1)$ :::: spreading the input (Step 3).
$+ k$ :::: the kernel painting into the holes (Step 4).
$- 2 p$ :::: outer padding removed, one ring each side (this step).

Step 6 — Full worked example: FCN-32s, $7 \to 224$

WHAT. The parent claimed FCN-32s upsamples $7 \times 7$ straight to $224 \times 224$ in "one shot ×32". Let us pick the numbers that make the formula land exactly on $224$ .

WHY. A formula you cannot plug numbers into is not yet yours. We prove the ×32 claim is real, not marketing.

PICTURE. A number line from $7$ to $224$ , annotated with the stride $s = 32$ , kernel $k = 64$ , padding $p = 16$ — every quantity tagged onto the arrow.

Plug $H_{in} = 7, s = 32, k = 64, p = 16$ :

$H_{o u t} = 32 (7 - 1) + 64 - 2 \cdot 16 = 192 + 64 - 32 = 224. ✓$

$32 (7 - 1) = 192$ :::: six gaps between seven cells, each stretched by 32.
$+ 64$ :::: a fat $64 \times 64$ kernel wide enough to smear smoothly across the huge holes.
$- 32$ :::: trims the outer padding back so we land exactly on 224, not 256.

That is why FCN-32s is "coarse": one input pixel now controls a $32 \times 32$ block of output — blocky boundaries. This is precisely the motivation the parent gave for skip connections (FCN-16s, FCN-8s) and for ResNet-style fusions of shallow, high-resolution features.

Step 7 — The degenerate & edge cases you must never trip on

WHAT. We check the corners of the formula so no input surprises you.

WHY. The contract: every quadrant, every sign, every zero. Upsampling has three sharp edges.

PICTURE. Three mini-boards side by side: (a) $H_{in} = 1$ , (b) $s = 1$ , (c) the floor-ambiguity of the forward map.

Case (a): single-pixel input, $H_{in} = 1$ . Then $s (H_{in} - 1) = 0$ — no gaps to spread because there is only one cell. The output is $k - 2 p$ : the kernel simply stamps itself once. Segmentation networks never start from $1 \times 1$ , but it confirms the term $s (H_{in} - 1)$ vanishes correctly.

Case (b): stride one, $s = 1$ . Then $s (H_{in} - 1) = H_{in} - 1$ and no zeros are inserted (Step 3 gap = $s - 1 = 0$ ). $H_{o u t} = H_{in} + k - 1 - 2 p$ : this is just a padded ordinary conv that grows the map by a little. Transposed conv with $s = 1$ does not upsample — a common beginner trap.

Case (c): the checkerboard artifact. When $k$ is not divisible by $s$ , some output pixels are hit by more kernel overlaps than others, producing a faint checkerboard. The formula still gives the right size, but the quality suffers — which is why practitioners pick $k$ a multiple of $s$ (e.g. $k = 64, s = 32$ in Step 6: $64/32 = 2$ , clean).

The one-picture summary

Everything on this page is one sentence: spread the input with $s - 1$ zeros between cells (giving the $s (H_{in} - 1)$ term), slide a learnable $k \times k$ kernel over it (giving $+ k$ ), then crop the outer padding (giving $- 2 p$ ).

Recall Feynman retelling — say it back in plain words

A normal convolution walks a little window across a picture and shrinks it. To grow a picture back, I run that motion in reverse. First I take my small grid and pull the pixels apart, dropping $s - 1$ empty cells into every gap — that stretches the side to $s (H_{in} - 1) + 1$ . Those gaps are empty, so I slide a learnable brush of width $k$ across the stretched grid; it smears real numbers into the holes and adds $k - 1$ more to the side. Finally, if the forward version had used padding, I crop that same padding off — twice, once per side — so I subtract $2 p$ . Add it up and I get $H_{o u t} = s (H_{in} - 1) + k - 2 p$ . In FCN-32s I choose $s = 32, k = 64, p = 16$ and turn a $7 \times 7$ into a crisp $224 \times 224$ . The only traps: $s = 1$ doesn't actually upsample, a single-pixel input just stamps the kernel once, and if $k$ isn't a multiple of $s$ I get an ugly checkerboard.

Recall Quick self-test

With $H_{in} = 28, s = 2, k = 2, p = 0$ , what is $H_{o u t}$ ? ::: $2 (28 - 1) + 2 - 0 = 54 + 2 = 56$ — matching U-Net's $28 \to 56$ decoder step. Why concatenate rather than add these upsampled features to encoder features in U-Net? ::: Concatenation keeps all channels so later convs can choose what to fuse; addition forces a fixed linear blend and can lose detail.

Related: Loss Functions (pixel-wise cross-entropy consumes this full-resolution output), Data Augmentation and Batch Normalization (stabilise training of these deep encoder–decoders), Transfer Learning (the FCN encoder is a pretrained backbone), Object Detection and Attention Mechanisms (alternative dense-prediction ideas).