Semantic segmentation (U-Net, FCN) — Hinglish notes

3.4.14 · AI-ML › Convolutional Neural Networks

Overview

Semantic segmentation ek aisa task hai jisme image ke har pixel ko ek category mein classify kiya jaata hai (jaise road, car, person, sky). Object detection se alag — jo sirf bounding boxes draw karta hai — segmentation ek dense prediction map produce karta hai jisme har pixel ko ek label milta hai.

Key difference: Object detection = "coordinates (x, y, w, h) par ek dog hai." Semantic segmentation = "ye 2,347 pixels dog HAIN."

Figure — Semantic segmentation (U-Net, FCN)

Core Problem: Classification se Dense Prediction tak

Ye mushkil kyun hai? Standard CNNs jaise VGG/ResNet aggressively downsample karte hain (stride-2 convs, pooling) taaki abstract features ban sakein. Final layer tak, ek 224×224 image 7×7 feature maps ban jaati hai. Humne spatial resolution kho di hai — lekin segmentation ko har pixel classify karne ke liye original resolution wapas chahiye!

Fully Convolutional Networks (FCN)

FCN (Long et al., 2015) pehli aisi architecture thi jisne segmentation ko CNNs ke saath end-to-end tackle kiya.

Architecture Breakdown

Step 1: Encoder (Downsampling)

Ek pretrained classifier backbone se shuru karo (jaise VGG-16)
End ke FC layers remove karo
Ab network ek feature map output karta hai, jaise $224 \times 224$ input ke liye $7 \times 7 \times 4096$

Ye step kyun? Humein rich semantic features chahiye (kaun se objects present hain). Deep layers high-level abstractions capture karte hain, lekin wo spatially coarse hote hain.

Step 2: Class Scores ke liye 1×1 Convolution

$C$ filters (number of classes) ke saath ek $1 \times 1$ conv apply karo
Output: $7 \times 7 \times C$ score map

Kyun? Har spatial location ko ek class prediction vector milta hai. Ye aise hai jaise har location par ek classifier run ho.

Step 3: Upsampling (Transposed Convolution)

Spatial size badhane ke liye transposed convolution use karo (ise deconvolution bhi kehte hain, lekin wo ek misnomer hai)
Example: FCN-32s ek baar mein ×32 upsample karta hai: $7 \times 7 \to 224 \times 224$

Derivation: Regular convolution striding se size ghataata hai. Transposed convolution iska ulta karta hai: ye input values ke beech zeros insert karta hai (stride se input ko dilate karta hai), phir regular convolution apply karta hai. Stride $s$ ke saath, input effectively $s$ times expand ho jaata hai, phir convolve hota hai.

Transposed conv kyun? Ye ek learnable upsampling hai — network seekhta hai ki best tarike se interpolate kaise karna hai, fixed bilinear upsampling ki jagah.

Skip Connection Innovation

FCN-32s (ek-shot ×32 upsampling) bahut coarse hai. FCN-16s aur FCN-8s skip connections add karte hain:

FCN-16s: ×32 upsampled map ko pool4 ke features (higher resolution) ke saath fuse karo, phir ×16 upsample karo
FCN-8s: pool3 ke saath bhi fuse karo, ×8 upsample karo

Kaise? Upsampled deep features ka earlier-layer features ke saath element-wise addition (channel counts match karne ke liye 1×1 conv ke baad).

Ye sequence kyun? Gradual refinement: coarse semantics + medium details + fine details.

U-Net: Medical Imaging Champion

U-Net (Ronneberger et al., 2015) biomedical segmentation ke liye design kiya gaya tha (jaise microscopy mein cell nuclei) jahan training data scarce hoti hai.

Architecture Details

Encoder (Contracting Path):

Baar baar: 3×3 conv → ReLU → 3×3 conv → ReLU → 2×2 max pool (stride 2)
Har downsampling feature channels double karta hai (64 → 128 → 256 → 512 → 1024)
Result: progressively smaller spatial size, richer feature representation

Channels double kyun? Jaise spatial info compress hoti hai, abstract patterns encode karne ke liye zyada feature capacity chahiye.

Bottleneck:

Sabse low resolution (jaise 572×572 input ke liye 28×28 with 1024 channels)
Highest semantic abstraction

Decoder (Expansive Path):

Baar baar: 2×2 transposed conv (stride 2, upsamples) → encoder features ke saath concatenate (skip connection) → 3×3 conv → ReLU → 3×3 conv → ReLU
Har upsampling feature channels halve karta hai (1024 → 512 → 256 → 128 → 64)
Final 1×1 conv C classes mein map karta hai

Derivation: Channel axis ke along concatenation dono paths se saari information preserve karta hai. Baad ke convolutions inhe fuse karna seekhte hain. Ye FCN ke addition se alag hai, jo information lose kar sakta hai agar features perfectly aligned na hon.

Add ki jagah concatenate kyun? Medical images mein subtle differences hote hain (cell membranes, small structures). Concatenation network ko choose karne deta hai ki kaun se encoder features use karne hain, bajaaye ek linear combination force karne ke.

Decoder:

Upsample: 56×56×512, encoder 64×64 ko 56×56 par crop karo, concat → 56×56×1024
Convs ke baad: 52×52×512
(Repeat: upsample, crop-concat, conv...)
Final: 388×388×C

Crop kyun? Original U-Net no padding use karta hai (valid convolutions), isliye encoder feature maps thode bade hote hain. Inhe decoder size match karne ke liye center-crop kiya jaata hai.

Ye step kyun matter karta hai: Medical imaging mein, data augmentation (rotation, flip) heavily use hoti hai. No-padding design original paper ki ek khasiyat thi; modern U-Nets padding use karte hain taaki sizes aligned rahein.

U-Net vs. FCN

Aspect	FCN	U-Net
Skip connections	Upsampled + encoder ka addition	Har level par concatenation
Symmetry	Asymmetric (classifier backbone)	Symmetric encoder-decoder
Use case	Natural images (ImageNet-pretrained)	Medical/small datasets (scratch se train)
Output size	Input jaisi	Alag ho sakti hai (original U-Net) ya same (modern)

Semantic Segmentation Models ko Train Karna

Derivation: Har pixel ek independent classification problem hai. Hum saare pixels pe cross-entropy sum karte hain.

U-Net ke liye class imbalance ke saath (jaise cells 5% pixels hain, background 95%), weighted cross-entropy use karo: $L_{weighted} = - \sum_{i, j} w (Y_{i, j}) \cdot lo g P (\hat{Y}_{i, j} = Y_{i, j})$ jahan $w (c) = \frac{N}{count ( c )}$ (inverse frequency weighting).

Kyun? Weighting ke bina, model saara background predict karke 95% accuracy le sakta hai. Weighting use karne se wo rare classes ki bhi parwah karta hai.

Derivation: Dice coefficient overlap measure karta hai: $\frac{2∣ A \cap B ∣}{∣ A ∣ + ∣ B ∣}$ . $1 - Dice$ minimize karna overlap maximize karta hai.

Ise kyun use karein? Dice, IoU (Intersection over Union) ka ek soft version hai — differentiable aur class imbalance ke liye robust.

Evaluation Metrics

mIoU kyun? Ye false positives (jahan nahi hai wahan class predict karna) aur false negatives (class miss karna) dono ko penalize karta hai. Pixel accuracy misleading ho sakti hai agar classes imbalanced hon.

Ye kyun matter karta hai: Bhale hi humne car ke 80% pixels sahi predict kiye, extra false alarms IoU ko 72.7% tak reduce kar dete hain.

Common Mistakes

Worked Example: 128×128 Image par U-Net Segmentation

Setup: 3 classes (background, cell, membrane). Input: 128×128×3 RGB image.

Encoder:

Conv1: 3×3 conv (64 filters, padding=same) → 128×128×64 → ReLU → 3×3 conv → 128×128×64 → ReLU
MaxPool: 2×2 stride 2 → 64×64
Conv2: 3×3 conv (128 filters) → 64×64×128 → ReLU → 3×3 conv → 128×128 (typo: should be 64×64×128) → ReLU
MaxPool: 2×2 stride 2 → 32×32×128
Conv3: 3×3 conv (256 filters) → 32×32×256 → ... → 32×32×256
MaxPool → 16×16×256
Conv4: 3×3 conv (512 filters) → 16×16×512
MaxPool → 8×8×512

Bottleneck:

Conv5: 3×3 conv (1024 filters) → 8×8×1024

Decoder:

Upsample: 2×2 transposed conv (stride 2) → 16×16×512
- Conv4 output (16×16×512) ke saath Concatenate karo → 16×16×1024
- 3×3 conv (512 filters) → 16×16×512
Upsample: 2×2 transposed conv → 32×32×256
- Conv3 ke saath Concat karo → 32×32×512
- 3×3 conv (256 filters) → 32×32×256
Upsample: 2×2 transposed conv → 64×64×128
- Conv2 ke saath Concat karo → 64×64×256
- 3×3 conv (128 filters) → 64×64×128
Upsample: 2×2 transposed conv → 128×128×64
- Conv1 ke saath Concat karo → 128×128×128
- 3×3 conv (64 filters) → 128×128×64

Output:

1×1 conv (3 classes ke liye 3 filters) → 128×128×3 logits
Softmax → 128×128 class predictions

Har step kyun?

Downsampling: Context capture karne ke liye compress karo ("yahan ek cell hai")
Upsampling: Pixel resolution tak wapas expand karo
Concatenation: "Ye raha cell ka high-res representation (encoder se), ise upsampled semantic info ke saath fuse karo (decoder se)"

Numerical check: Pehle decoder layer mein, hum 8×8→16×16 upsample karte hain. Stride-2 transposed conv ke saath, formula deta hai: $2 \cdot (8 - 1) + 2 - 0 = 16$ . ✓

Modern Extensions

Atrous/Dilated Convolutions: DeepLab mein use hote hain. Pooling ki jagah, resolution rakhte hue receptive field expand karne ke liye dilated convs use karo.
Attention U-Net: Relevant regions par focus karne ke liye skip connections mein attention gates add karo.
SegFormer: Transformer-based encoder with lightweight MLP decoder. Kai benchmarks par state-of-the-art.

Recall Ek 12-Saal Ke Bacche Ko Samjhao

Socho tum ek picture color kar rahe ho, lekin kisi ke banaye lines ke andar rehne ki bajaye, tumhe khud lines draw karni hain. Picture mein har ek dot (pixel) ko ek color chahiye, aur color depend karta hai ki wo dot kis cheez ka hissa hai: Kya wo sky hai? Ek tree? Ek insaan?

Semantic segmentation computer ko yehi karna sikhana hai. Computer ek photo dekhta hai aur har pixel ko color karta hai — jo kuch bhi wo pehchaanta hai. Ek self-driving car iske liye use karti hai taaki dekh sake "ye pixels road ke hain, ye ek pedestrian ke hain, ye ek parked car ke hain."

U-Net ek aisi magnifying glass ki tarah hai jo dono taraf kaam karti hai: Pehle, ye zoom out karti hai (encoder) taaki samjhe "oh, is area mein ek insaan hai." Phir ye wapas zoom in karti hai (decoder) taaki insaan ke har hisse ko carefully trace kare. Clever part? Wapas zoom in karte waqt, ye yaad rakhti hai ki picture pehle kaisi dikhti thi, taaki chhoti details jaise ungliyan ya baal na kho jaayein. Wo memory hi "skip connection" hai — jaise har zoom level par sticky notes chhod dena.

Connections

Convolutional Layers: U-Net aur FCN poori tarah conv layers (aur transposed conv) se bane hain
Pooling Layers: Max pooling encoder mein downsample karta hai; transposed conv decoder mein upsample karta hai
Residual Networks (ResNet): Modern segmentation VGG ki jagah ResNet backbones use karta hai
Transfer Learning: FCN typically ImageNet-pretrained encoder use karta hai; U-Net aksar scratch se train karta hai
Data Augmentation: Medical imaging mein U-Net ke liye critical hai (rotation, elastic deformation)
Batch Normalization: Modern U-Nets stable training ke liye har conv ke baad BN add karte hain
Loss Functions: Cross-entropy, Dice loss, Focal loss (hard examples ke liye)
Object Detection: Segmentation pixel-level masks deta hai vs. bounding boxes
Attention Mechanisms: Attention U-Net relevant features par focus karne ke liye gates use karta hai

#flashcards/ai-ml

Semantic segmentation kya hai? :: Har pixel ko ek semantic category mein classify karne ka task (jaise road, car, person), ek dense prediction map produce karta hai jahan har pixel ko ek class label milta hai.

FCN ek classifier ko segmentation network mein kaise convert karta hai?

Fully-connected layers ko 1×1 convolutions se replace karke (fully convolutional banata hai) aur coarse feature maps ko original image resolution tak wapas upsample karne ke liye transposed convolutions add karke.

FCN mein skip connections ka kya purpose hai?

Early layers (pool3, pool4) se high-resolution spatial details ko deep layers ki low-resolution semantic information ke saath fuse karna, segmentation output mein boundary accuracy improve karne ke liye.

Transposed convolution output size ka formula kya hai?

H_{o u t} = s \cdot (H_{in} - 1) + k - 2 p

, jahan

s

stride hai,

k

kernel size hai, aur

p

padding hai. Ye effectively values ke beech zeros insert karke input ko dilate karta hai, phir convolve karta hai.

U-Net aur FCN ke beech key architectural difference kya hai?

U-Net har resolution level par concatenation ke saath symmetric encoder-decoder use karta hai, jabki FCN addition of upsampled features with encoder layers use karta hai aur asymmetric hai (classifier backbone).

U-Net skip connections add karne ki jagah concatenate kyun karta hai?

Concatenation encoder aur decoder dono paths ki saari information preserve karta hai, agle convolutions ko ye choose karne deta hai ki kaun se features use karne hain — medical imaging mein critical hai jahan subtle differences matter karte hain.

Segmentation ke liye pixel-wise cross-entropy loss kya hai?

L = - \frac{1}{H W} \sum_{i, j} lo g P (\hat{Y}_{i, j} = Y_{i, j})

, jahan hum saare pixels pe cross-entropy sum karte hain, har pixel ko ek independent classification problem maante hue.

Dice loss kya hai aur segmentation ke liye ise kyun use karein?

L_{Dice} = 1 - \frac{2 \sum p _{i, j} g _{i, j}}{\sum p _{i, j} + \sum g _{i, j}}

, IoU ka ek differentiable version hai jo class imbalance ke liye robust hai — medical image segmentation mein heavily use hota hai.

Mean Intersection over Union (mIoU) kya hai?

Per-class IoU scores ka average:

mIoU = \frac{1}{C} \sum_{c} \frac{TP _{c}}{TP _{c} + FN _{c} + FP _{c}}

, har class ke liye false positives aur false negatives dono ko penalize karta hai.

Segmentation mein upsampling se pehle global average pooling kyun use nahi kar sakte?

Global pooling ek 1×1 feature vector produce karta hai jisme koi spatial information nahi hoti — ek single vector se 224×224 pixel-wise map reconstruct nahi ho sakta, kyunki saari "where" information kho jaati hai.

U-Net mein bina padding ke convolutions use karne se kya hota hai?

Har 3×3 conv feature map ko per side 2 pixels se shrink karta hai; kai layers ke baad, output input se bahut chhota ho jaata hai, pixel-level alignment tod deta hai (original U-Net mein ye issue tha, center-cropping ki zaroorat padti thi).

U-Net downsampling karte waqt feature channels double kyun karta hai?

Jaise spatial dimensions compress hoti hain, deeper layers par progressively abstract semantic patterns encode karne ke liye zyada feature capacity ki zaroorat hoti hai.

Decoder mein transposed convolution ka kya role hai?

Ye learnable upsampling perform karta hai, spatial dimensions badhata hai jabki network ko optimal tarike se features interpolate karna sikhne deta hai, bilinear interpolation jaise fixed methods ki jagah.

Weighted cross-entropy class imbalance ko kaise address karta hai?

Har class ko loss weight

w (c) = \frac{N}{count ( c )}

assign karke, taaki rare classes (jaise cells = 5% pixels) loss mein zyada contribute karein, model ko inhe ignore karne se rokne ke liye.

U-Net bottleneck kya hai aur ye kyun important hai?

Sabse low spatial resolution layer (jaise 1024 channels ke saath 8×8) jo highest-level semantic abstraction capture karta hai — ye represent karta hai "kaun se objects present hain" decoder ke "exactly kahan" reconstruct karne se pehle.