This page hunts the misconceptions that segmentation invites. Each item is a one-line reveal: read the prompt, commit to an answer in your head, then check the reasoning. These are the traps that catch people who "kind of" understand semantic segmentation but haven't nailed the why.
Before the traps, we pin down three things this page needs so it stands on its own: what the image dimensions mean, what "pool3 / pool4 / conv7 / valid convolution" mean, and exactly how transposed convolution changes size (a formula the parent note stated but never unpacked).
Figure 1 (alt text): Two side-by-side grids. On the left, a plain 6×6 pixel grid labelled with a green double-arrow marking its height H=6 and a red double-arrow marking its width W=6. An arrow points right to the segmentation map on the right: the same 6×6 shape, but every cell is now colored — blue cells at the top are "sky", an orange blob is a "car", gray cells are "road". Takeaway: segmentation keeps the exact same grid shape and just paints each cell with its class.
Look at the figure: the left grid is a 6×6 image (so H=6, W=6); the right grid is its segmentation map — same shape, but each cell is now colored by its class. That "same shape, colored by class" picture is the whole goal of segmentation.
The parent note gave the upsampling size rule but never defined its symbols or showed where the shape comes from. Let's earn it.
Figure 2 (alt text): A three-row diagram showing transposed convolution on a tiny row of 3 blue input cells labelled x0, x1, x2. Top row — the 3 original inputs touching each other. Middle row — the same inputs spread apart with one gray "0" zero-cell inserted in each gap (an orange double-arrow marks the gap width = stride s=2). Bottom row — a red box marks a k=2 sliding window producing green output cells y0…y4. A side box shows the arithmetic Hout=s(Hin−1)+k−2p=2(3−1)+2−0=6. Takeaway: spreading by the stride, then convolving, makes the grid grow.
Now read the size off that picture, one move at a time:
Semantic segmentation and object detection produce the same kind of output.
False.Object Detection outputs a few bounding boxes with class labels; segmentation outputs a full H×W label map — one class ID per pixel, so a "dog" is thousands of pixels, not one box.
Semantic segmentation distinguishes two different dogs standing side by side.
False. Semantic segmentation only labels categories, so both dogs become the same "dog" region. Telling instances apart is instance segmentation — a different task.
A plain classifier CNN (VGG/ResNet) already outputs a per-pixel label map.
False. Its final layer is a single class vector for the whole image; spatial resolution has been thrown away by pooling and strided convs, so there is no pixel-level map to read off.
Transposed convolution is the mathematical inverse of convolution (it recovers the original input).
False. The name "deconvolution" is a misnomer — it only restores the shape, not the values. It is a learnable upsampler, not an inverse operator.
FCN can accept images of any size, but a VGG classifier with FC layers cannot.
True. Fully-connected layers fix the input size; replacing them with 1×1 convs makes the network "fully convolutional," so any spatial input flows through and produces a proportionally sized map.
U-Net adds encoder and decoder features at each skip connection, just like FCN.
False. U-Net concatenates along the channel axis (keeping both feature sets intact); FCN uses element-wise addition. That is the headline architectural difference between them.
Weighted cross-entropy is only useful when all classes appear equally often.
False. It exists because classes are imbalanced (e.g. 5% cells vs 95% background). Upweighting the rare class stops the model from trivially predicting "background everywhere." See Loss Functions.
The original U-Net's output has the same spatial size as its input.
False. With valid (no-padding) convolutions, a 572×572 input yields a 388×388 output — smaller. Modern padded U-Nets keep sizes equal; that's a design choice, not a law.
Skip connections in FCN/U-Net exist mainly to fix vanishing gradients (like Residual Networks (ResNet)).
False here. In segmentation their primary job is to reinject spatial detail ("where") from shallow layers, since deep layers only know "what." The gradient benefit is secondary.
Increasing 1×1 conv channels to C means "C classes get a score at every spatial location."
True. A 1×1 conv with C filters runs a tiny classifier at each pixel of the feature map, producing a length-C score vector per location.
"FCN-32s upsamples in one ×32 step, so it gives sharper boundaries than FCN-8s."
Backwards. One big ×32 jump is the coarsest; FCN-8s fuses pool3 (28×28, fine detail) and pool4 (14×14) features before smaller upsampling steps, giving sharper edges.
"We concatenate pool4 features directly onto conv7 features without any change."
You must first apply a 1×1 conv to pool4 (the map after the 4th pooling stage) to remap it to C channels so the shapes match before fusing — and FCN adds rather than concatenates.
"U-Net doubles the number of spatial pixels every time we go deeper in the encoder."
The encoder halves spatial size (via pooling) and doubles the channel count, not the pixels. It compresses space, expands features.
"Cross-entropy for segmentation sums over classes only, treating the image as one prediction."
It sums over every pixel(i,j)and normalizes over classes at each pixel — each pixel is its own independent classification.
"Because U-Net uses valid convolutions, no cropping of encoder features is ever needed."
Valid (no-padding) convolutions are exactly why cropping is needed — they shrink each side, so encoder maps come out slightly larger than the decoder and are center-cropped to match before concatenation.
"Transposed conv with stride s shrinks the input by a factor of s."
It enlarges it. Using Hout=s(Hin−1)+k−2p, a larger stride gives a larger output; stride here inserts zeros between inputs (see Figure 2).
"Pooling is bad for segmentation, so U-Net avoids it entirely."
U-Net usesPooling Layers in its encoder to build context; it just counteracts the resolution loss with the decoder and skip connections.
Why can't we just run a classifier on tiny image patches instead of building a segmentation network?
You could, but it's absurdly slow (one forward pass per pixel), wastes overlapping computation, and lacks large-scale context. A fully convolutional net does it all in one pass with shared features.
Why does U-Net concatenate features while FCN adds them?
Addition forces a fixed linear blend and can cancel misaligned features; concatenation preserves all information and lets the following convs learn which encoder details to use — valuable for subtle medical structures.
Why do deep layers lose "where" but keep "what"?
Repeated pooling/striding shrinks spatial resolution, blurring exact positions, while stacking convs builds abstract semantic features — so you gain category knowledge and lose precise location.
Why is Data Augmentation especially critical for U-Net's original use case?
Biomedical datasets are tiny (few annotated microscopy images), so rotations, flips, and elastic deformations manufacture variety and prevent overfitting where real data is scarce.
Why use a pretrained backbone (via Transfer Learning) for FCN but often train U-Net from scratch?
FCN targets natural images resembling ImageNet, so ImageNet features transfer well; U-Net's medical images look nothing like ImageNet, so learned-from-scratch features can fit better — plus its dataset is small enough to still train.
Why does compressing spatial size justify doubling the channel count in the encoder?
As you throw away spatial locations you need more feature channels to encode the increasingly abstract patterns — capacity moves from "where" to "what."
Why put Batch Normalization between conv and ReLU in modern U-Nets when the original had none?
BatchNorm stabilizes and speeds training by normalizing activations; the 2015 paper predated its widespread use, so modern reimplementations add it for better convergence.
What happens to pixel-wise cross-entropy if one class never appears in the ground truth mask?
That class contributes no target terms, so it gets no direct gradient from those pixels — the model may never learn it well unless it appears in other images or is upweighted.
If every pixel in an image belongs to the same class (all sky), does the loss still make sense?
Yes — it's just the average cross-entropy over identical targets. A confident, correct model gives near-zero loss; nothing degenerate happens.
What does the segmentation output look like at the exact boundary between two objects?
Boundary pixels get soft, near-tied class scores; the argmax picks one crisp label per pixel, which is why fine skip connections matter to place that boundary accurately.
With valid convolutions, why can't the network label pixels at the very edge of the input?
Valid convs have no padding, so border pixels lack a full receptive field — the output shrinks and the outermost input pixels simply aren't predicted (hence the overlap-tile strategy).
What breaks if a decoder's upsampled map and the encoder skip map differ by one pixel in size?
Concatenation requires identical H×W; a mismatch throws a shape error, which is exactly why cropping (valid U-Net) or padding alignment (modern U-Net) is mandatory.
For a 2-class (foreground/background) task, could you replace softmax cross-entropy with a single sigmoid per pixel?
Yes — binary segmentation reduces to one probability per pixel via sigmoid + binary cross-entropy, mathematically equivalent to the 2-class softmax.
Recall Quick self-test
Name the single biggest architectural difference between FCN and U-Net. ::: FCN adds upsampled + encoder features; U-Net concatenates them at every level.
What geometric quantity do skip connections restore that deep layers destroyed? ::: Spatial "where" information (precise location / boundaries).
Why does transposed convolution enlarge rather than shrink? ::: Stride inserts zeros between input values, spreading the grid so Hout=s(Hin−1)+k−2p grows with s.
