Semantic segmentation (U-Net, FCN)

Overview

Semantic segmentation is the task of classifying every pixel in an image into a category (e.g., road, car, person, sky). Unlike object detection which draws bounding boxes, segmentation produces a dense prediction map where each pixel gets a label.

Key difference: Object detection = "there's a dog at coordinates (x, y, w, h)". Semantic segmentation = "these2,347 pixels ARE the dog."

The Core Problem: From Classification to Dense Prediction

Why is this hard? Standard CNNs like VGG/ResNet downsample agressively (stride-2 convs, pooling) to build abstract features. By the final layer, a224×224 image becomes 7×7 feature maps. We've lost spatial resolution—but segmentation needs the original resolution back to classify each pixel!

Fully Convolutional Networks (FCN)

FCN (Long et al., 2015) was the first architecture to tackle segmentation end-to-end with CNs.

Architecture Breakdown

Step 1: Encoder (Downsampling)

• Start with a pretrained classifier backbone (e.g., VGG-16)
• Remove the FC layers at the end
• The network now outputs a feature map, say $7 \times 7 \times 4096$ for a $224 \times 224$ input

Why this step? We need rich semantic features (what objects are present). Deep layers capture high-level abstractions, but they're spatially coarse.

Step 2: 1×1 Convolution to Class Scores

• Apply a $1 \times 1$ conv with $C$ filters (number of classes)
• Output: $7 \times 7 \times C$ score map

Why? Each spatial location gets a class prediction vector. This is like running a classifier at every location.

Step 3: Upsampling (Transposed Convolution)

• Use transposed convolution (also called deconvolution, though that's a misnomer) to increase spatial size
• Example: FCN-32s upsamples by×32 in one shot: $7 \times 7 \rightarrow 224 \times 224$

Derivation: Regular convolution reduces size by striding. Transposed convolution does the opposite: it inserts zeros between input values (dilates the input by the stride), then applies regular convolution. With stride $s$, the input is effectively expanded by $s$ times, then convolved.

Why transposed conv? It's a learnable upsampling—the network learns the best way to interpolate, rather than fixed bilinear upsampling.

The Skip Connection Innovation

FCN-32s (one-shot ×32 upsampling) is too coarse. FCN-16s and FCN-8s add skip connections:

• FCN-16s: Fuse the×32 upsampled map with features from pool4 (higher resolution), then upsample ×16
• FCN-8s: Fuse with pool3 as well, upsample ×8

How? Element-wise addition of upsampled deep features with earlier-layer features (after a 1×1 conv to match channel counts).

Why this sequence? Gradual refinement: coarse semantics + medium details + fine details.

U-Net: The Medical Imaging Champion

U-Net (Ronneberger et al., 2015) was designed for biomedical segmentation (e.g., cell nuclei in microscopy) where training data is scarce.

Architecture Details

Encoder (Contracting Path):

• Repeated: 3×3 conv → ReLU → 3×3 conv → ReLU → 2×2 max pool (stride 2)
• Each downsampling doubles feature channels (64 → 128 → 256 → 512 → 1024)
• Result: progressively smaller spatial size, richer feature representation

Why double channels? As spatial info compresses, we need more feature capacity to encode the abstract patterns.

Bottleneck:

• Lowest resolution (e.g., 28×28 with 1024 channels for 572×572 input)
• Highest semantic abstraction

Decoder (Expansive Path):

• Repeated: 2×2 transposed conv (stride 2, upsamples) → concatenate with encoder features (skip connection) → 3×3 conv → ReLU → 3×3 conv → ReLU
• Each upsampling halves feature channels (1024 → 512 → 256 → 128 → 64)
• Final 1×1 conv maps to C classes

Derivation: Concatenation along the channel axis preserves all information from both paths. The subsequent convolutions learn to fuse them. This differs from FCN's addition, which can lose information if features are not perfectly aligned.

Why concatenate not add? Medical images have subtle differences (cell membranes, small structures). Concatenation lets the network choose which encoder features to use, rather than forcing a linear combination.

Decoder:

• Upsample: 56×56×512, crop encoder64×64 to 56×56, concat → 56×56×1024
• After convs: 52×52×512
• (Repeat: upsample, crop-concat, conv...)
• Final: 388×388×C

Why crop? The original U-Net uses no padding (valid convolutions), so encoder feature maps are slightly larger. They're center-cropped to match decoder size.

Why this step matters: In medical imaging, data augmentation (rotation, flip) is used heavily. The no-padding design was a quirk of the original paper; modern U-Nets use padding to keep sizes aligned.

U-Net vs. FCN

Training Semantic Segmentation Models

Derivation: Each pixel is an independent classification problem. We sum cross-entropy over all pixels.

For U-Net with class imbalance (e.g., cells are5% of pixels, background95%), use weighted cross-entropy: $\mathcal{L}_{\text{weighted}} = -\sum_{i,j} w(Y_{i,j}) \cdot \log P(\hat{Y}_{i,j} = Y_{i,j})$ where $w(c) = \frac{N}{\text{count}(c)}$ (inverse frequency weighting).

Why? Without weighting, the model can get95% accuracy by predicting all background. Weighting forces it to care about rare classes.

Derivation: Dice coefficient measures overlap: $\frac{2 |A \cap B|}{|A| + |B|}$. Minimizing $1 - \text{Dice}$ maximizes overlap.

Why use it? Dice is a soft version of IoU (Intersection over Union), differentiable and robust to class imbalance.

Evaluation Metrics

Why mIoU? It penalizes both false positives (predicting class where it's not) and false negatives (missing the class). Pixel accuracy can be misleading if classes are imbalanced.

Why this matters: Even though we got 80% of the car pixels, the extra false alarms reduce IoU to 72.7%.

Common Mistakes

Worked Example: U-Net Segmentation on128×128 Image

Setup: 3 classes (background, cell, membrane). Input: 128×128×3 RGB image.

Encoder:

1. Conv1: 3×3 conv (64 filters, padding=same) → 128×128×64 → ReLU → 3×3 conv → 128×128×64 → ReLU
2. MaxPool: 2×2 stride 2 → 64×64
3. Conv2: 3×3 conv (128 filters) → 64×64×128 → ReLU → 3×3 conv → 128×128 (typo: should be 64×64×128) → ReLU
4. MaxPool: 2×2 stride 2 → 32×32×128
5. Conv3: 3×3 conv (256 filters) → 32×32×256 → ... → 32×32×256
6. MaxPool → 16×16×256
7. Conv4: 3×3 conv (512 filters) → 16×16×512
8. MaxPool → 8×512

Bottleneck:

• Conv5: 3×3 conv (1024 filters) → 8×8×1024

Decoder:

1. Upsample: 2×2 transposed conv (stride 2) → 16×16×512
   • Concatenate with Conv4 output (16×16×512) → 16×16×1024
   • 3×3 conv (512 filters) → 16×16×512
2. Upsample: 2×2 transposed conv → 32×32×256
   • Concat with Conv3 → 32×32×512
   • 3×3 conv (256 filters) → 32×256
3. Upsample: 2×2 transposed conv → 64×64×128
   • Concat with Conv2 → 64×64×256
   • 3×3 conv (128 filters) → 64×64×128
4. Upsample: 2×2 transposed conv → 128×128×64
   • Concat with Conv1 → 128×128×128
   • 3×3 conv (64 filters) → 128×128×64

Output:

• 1×1 conv (3 filters for 3 classes) → 128×128×3 logits
• Softmax → 128×128 class predictions

Why each step?

• Downsampling: Compress to capture context ("there's a cell here")
• Upsampling: Expand back to pixel resolution
• Concatenation: "Here's what the cell looks like at high-res (from encoder), fuse it with the upsampled semantic info (from decoder)"

Numerical check: At the first decoder layer, we upsample 8×8→16×16. With stride-2 transposed conv, the formula gives: $2 \cdot (8-1) + 2 - 0 = 16$. ✓

Modern Extensions

1. Atrous/Dilated Convolutions: Used in DepLab. Instead of pooling, use dilated convs to keep resolution while expanding receptive field.
2. Attention U-Net: Add attention gates in skip connections to focus on relevant regions.
3. SegFormer: Transformer-based encoder with lightweight MLP decoder. State-of-the-art on many benchmarks.

Semantic segmentation is teaching a computer to do that. The computer looks at a photo and colors every pixel based on what it recognizes. A self-driving car uses this to see "these pixels are the road, these are a pedestrian, these are a parked car."

U-Net is like a magnifying glass that works both ways: First, it zoms out (encoder) to understand "oh, there's a person in this area." Then it zooms back in (decoder) to carefully trace around every part of the person. The clever part? While zoming back in, it remembers what the picture looked like before, so it doesn't lose the tiny details like fingers or hair. That memory is the "skip connection"—like leaving sticky notes at each zoom level.

Connections

• Convolutional Layers: U-Net and FCN are built entirely of conv layers (and transposed conv)
• Pooling Layers: Max pooling downsamples in encoder; transposed conv upsamples in decoder
• Residual Networks (ResNet): Modern segmentation uses ResNet backbones instead of VGG
• Transfer Learning: FCN typically uses ImageNet-pretrained encoder; U-Net often trains from scratch
• Data Augmentation: Critical for U-Net in medical imaging (rotation, elastic deformation)
• Batch Normalization: Modern U-Nets add BN after each conv for stable training
• Loss Functions: Cross-entropy, Dice loss, Focal loss (for hard examples)
• Object Detection: Segmentation provides pixel-level masks vs. bounding boxes
• Attention Mechanisms: Attention U-Net uses gates to focus on relevant features

#flashcards/ai-ml

What is semantic segmentation? :: The task of classifying every pixel in an image into a semantic category (e.g., road, car, person), producing a dense prediction map where each pixel gets a class label.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)