Object detection (R-CNN, YOLO, SSD)

Overview

Object detection is the task of simultaneously locating (where?) and classifying (what?) multiple objects within an image. Unlike image classification (which only answers "what?") or semantic segmentation (which labels every pixel but doesn't distinguish instances), object detection outputs bounding boxes with class labels for each detected object.

The fundamental challenge: turning a classification network into a localization + classification system that works for variable numbers of objects.

The Evolution: Three Paradigms

1. R-CNN Family (Region-based Methods)

WHY this architecture?

• Separates "where to look" (region proposals) from "what is it" (classification)
• Leverages CNNs' strength at fixed-size image classification
• Reduces search space from all possible boxes (millions) to promising candidates

HOW it works:

$\text{Input Image} \xrightarrow{\text{Selective Search}} \{R_1, R_2, \ldots, R_n\} \xrightarrow{\text{Warp to fixed size}} \text{CNN} \xrightarrow{\text{SVM}} \text{Class + Box}$

Each region $R_i$ is:

1. Warped to a fixed size (227×227 for AlexNet)
2. Passed through a CNN to extract a feature vector
3. Classified with an SVM
4. Refined with a bounding box regressor

WHY these formulas?

• Translation ($d_x, d_y$): Offset as fraction of box width/height (scale-invariant)
• Scale ($d_w, d_h$): Exponential ensures positive width/height, makes learning easier (log-space is more linear)

The network learns $d_x, d_y, d_w, d_h$ by minimizing:

where $t_i$ are the true transformation targets:

Fast R-CNN improvement: Instead of running CNN2000 times, extract features once for the whole image, then use RoI (Region of Interest) pooling to extract fixed-size features for each proposal.

Faster R-CNN improvement: Replace Selective Search with a learned Region Proposal Network (RPN).

For each anchor:

• Objectness score: $p_{\text{obj}} = \sigma(z_{\text{obj}})$ (is there an object?)
• Box refinement: $(d_x, d_y, d_w, d_h)$ as above

Loss function combines both:

$\mathcal{L} = \frac{1}{N_{\text{cls}}} \sum_i \mathcal{L}_{\text{cls}}(p_i, p_i^*) + \lambda \frac{1}{N_{\text{reg}}} \sum_i p_i^* \mathcal{L}_{\text{reg}}(d_i, t_i)$

WHY anchor boxes?

• Handle multiple scales/aspect ratios without resizing
• Provide reference coordinates for box regression
• Typical: 3scales × 3 ratios = 9 anchors per position

After NMS (Non-Maximum Suppression), keep top 300. For each:

1. Extract 7×7 RoI pooled features
2. Two fully connected layers
3. Output: 21 classes (20 objects + background) + 4×21 box offsets

Why this step? The 7×7 pooling standardizes features regardless of proposal size. The 4×21 offsets allow class-specific box refinement (a "car" box might need different adjustments than a "person" box).

2. YOLO (You Only Look Once)

WHY single-stage?

• Speed: One forward pass instead of thousands
• Global context: Sees entire image, reducing background false positives
• Trade-off: Slightly lower accuracy on small objects

HOW YOLO works:

Divide image into $S \times S$ grid (e.g., 7×7). Each grid cell predicts:

• $B$ bounding boxes (each with5 values: $x, y, w, h, \text{confidence}$)
• $C$ class probabilities

Output tensor: $S \times S \times (B \cdot 5 + C)$

For YOLO v1: $7 \times 7 \times (2 \cdot 5 + 20) = 7 \times 7 \times 30$

where:

• $(x, y)$: Box center relative to grid cell (values0–1)
• $(w, h)$: Box size relative to image (values 0–1)
• $\text{confidence} = P(\text{object}) \cdot \text{IoU}_{\text{pred}}^{\text{truth}}$

WHY this confidence formula?

• If no object: $P(\text{object}) = 0$, confidence should be 0
• If object exists: Confidence reflects how well the box fits (IoU)
• During training, this is supervised. At test time, it's predicted.

Each grid cell also predicts:

Final class-specific confidence:

WHY each term?

1. Localization loss (x, y): MSE on box centers. Only penalize the box "responsible" for the object ($\mathbb{1}_{ij}^{\text{obj}} = 1$ if box $j$ in cell $i$ has highest IoU).

2. Size loss (w, h): Use $\sqrt{w}, \sqrt{h}$ because small deviations in large boxes matter less than in small boxes. Square root makes errors more comparable across scales.

3. Confidence loss (has object): Penalize when exists but confidence is low.

4. Confidence loss (no object): Weighted lower ($\lambda_{\text{noobj}} = 0.5$) because most boxes contain no object — prevents overwhelming the gradient.

5. Classification loss: Standard MSE on class probabilities for cells containing objects.

$\lambda_{\text{coord}} = 5$ increases importance of localization.

Cell (3, 4) contains a dog's center:

• Box 1predicts: $(0.6, 0.4, 0.3, 0.5, 0.85)$ (relative to cell and image)
• Cell predicts: $P(\text{dog}|\text{object}) = 0.92$
• Final: $0.92 \times 0.85 = 0.782$ confidence for "dog"

Why this step? Cell (3, 4) is responsible because the object's center is in that cell. The box with highest IoU (box 1) is the one penalized during training. The 0.782 final score combines "is there an object?" with "what is it?".

YOLO v2/v3 improvements:

• Anchor boxes (like Faster R-CNN): Predict offsets from anchors instead of absolute coordinates
• Multi-scale predictions: Detect at3 different feature map resolutions (for small, medium, large objects)
• Better backbone: Darknet-53 (ResNet-like with skip connections)

3. SSD (Single Shot MultiBox Detector)

WHY multi-scale?

• Early layers (high resolution): Detect small objects
• Late layers (low resolution): Detect large objects
• Each layer specializes in a scale range

HOW it works:

Take 6 different feature maps from the CNN at different stages (e.g., 38×38, 19×19, 10×10, 5×5, 3×3, 1×1). For each location in each map, predict:

where $c$ = number of classes, 4 = box offsets.

Localization:

Same as Faster R-CNN box regression: predict offsets $l_{cx}, l_{cy}, l_w, l_h$ relative to default box $d$.

Classification:

for each class $i$ (including background).

where:

• $N$ = number of matched default boxes
• $\alpha = 1$ (weight balance)

Localization loss (Smooth L1):

Confidence loss (Softmax):

where $\hat{c}_i^p = \frac{\exp(c_i^p)}{\sum_p \exp(c_i^p)}$

WHY Smooth L1?

• When error is small (|x| < 1): Behaves like L2 (gradients don't vanish)
• When error is large (|x| ≥ 1): Behaves like L1 (less sensitive to outliers, more stable)

Feature map 5×5 (late layer):

• Receptive field: ~300 pixels
• Default boxes: 6 boxes per location (scales 0.6-0.9, more aspect ratios)
• Good for detecting: cars, buses, large objects

Why this step? A small object (e.g., 30×30 pixels) would be barely visible in the 5×5 map (just 0.5 cells), but well-represented in the 38×38 map (4-5 cells). Multi-scale matching ensures each object is predicted where it's most salient.

Key Techniques Across All Methods

Non-Maximum Suppression (NMS)

All detectors output many overlapping boxes. NMS keeps only the best box per object.

WHY this works? Boxes with high IoU are detecting the same object. Keep the most confident one, discard redundant detections.

Derivation of IoU (Intersection over Union):

Given two boxes $A$ and $B$:

WHY this metric?

• IoU = 1: Perfect overlap (same box)
• IoU = 0: No overlap
• IoU ∈ [0, 1]: Measures box similarity geometrically
• Standard: IoU > 0.5 considered a "match" for evaluation

Hard Negative Mining

Most default boxes/anchors contain background, creating class imbalance.

Solution: After matching, sort boxes by confidence loss and keep top-K negatives (e.g., 3:1 ratio negative:positive).

WHY? Without this, the network learns to predict "background" everywhere and never detects objects. Hard negatives are the most confusing backgrounds (high loss) — focusing on these improves discrimination.

Comparison: When to Use Each

| Method | Speed (FPS) | Accuracy (mAP) | Best For | Trade-off | |--------|-------------|----------|--------| | Faster R-CNN | ~76% | High-accuracy applications (medical, autonomous vehicles) | Slow, two-stage | | YOLO v3 | ~30 | ~55% | Real-time video (surveillance, sports) | Mises small objects | | SSD | ~25 | ~68% | Balanced speed/accuracy (mobile apps) | Middle ground |

WHY these trade-offs?

• Two-stage (R-CNN): Region proposals → separate classification = more computation, better localization
• Single-stage (YOLO/SSD): Direct prediction = faster, but harder to learn precise localization
• Multi-scale (SSD): Better than YOLO on small objects, but slower than YOLO

Common Mistakes

Why it feels right: Both are between 0 and 1, both relate to detection quality.

The fix:

• Objectness/Confidence: "Is there any object here?" (binary)
• Classification: "What object is it?" (multi-class)
• Final score = objectness × classification

In YOLO: Each grid cell predicts one set of class probabilities (shared by all boxes) and per-box confidences. You multiply them for class-specific scores.

Steel-man: The confusion arises because in Faster R-CNN, these are combined in the RPN (objectness) and R-CNN head (classification), making the separation less obvious.

Why it feels right: MSE is the standard regression loss.

The fix: Consider two errors:

• Box1: 10×10 predicted as 15×15 (error = 5)
• Box 2: 100×100 predicted as 105×105 (error = 5)

With MSE on w, h: Both have loss = 25.

But Box 2's error is only 5% relative error, while Box 1's is 50%!

With MSE on $\sqrt{w}, \sqrt{h}$:

• Box 1: $\sqrt{10} \approx 3.16$, $\sqrt{15} \approx 3.87$, error ≈ 0.71, loss ≈ 0.5
• Box 2: $\sqrt{100} = 10$, $\sqrt{105} \approx 10.25$, error ≈ 0.25, loss ≈ 0.06

Now Box 1's loss is 8× larger, correctly reflecting the larger relative error.

Steel-man: The intuition "absolute error should matter equally" works for quantities of similar scale, but bounding boxes span orders of magnitude. Square root compresses the range, making loss more proportional to relative error.

Why it feels right: More anchors → more coverage → better chance of matching ground truth.

The fix:

• Computational cost: Linear in number of anchors (slower training, inference)
• Label assignment ambiguity: With too many anchors, multiple anchors match the same object equally well → inconsistent gradients
• Overfitting: More parameters to learn anchor-specific patterns

Sweet spot: 3 scales × 3 aspect ratios = 9 anchors captures most object variations. Empirically validated in papers.

Steel-man: The intuition comes from "more capacity = more learning power," which works for model depth/width. But anchors are hand-designed priors, not learned features — more isn't always better, smart selection is.

Active Recall

Old way (R-CNN): The computer says, "Let me check if Waldo is in this box... nope. How about this box? Nope. This one? Maybe..." It checks2000 boxes one by one! Super slow.

New way (YOLO): The computer divides the picture into a grid, like a checkerboard. Each square shouts out, "I see a person here! Confidence: 85%!" and "I see a dog here! 92%!" All at once! Super fast.

Even better way (SSD): Like YOLO, but uses multiple grids—a coarse one for big things (cars) and a fine one for tiny things (faces). Like using a magnifying glass for small details and binoculars for distant objects.

The computer learns by showing it thousands of labeled pictures: "This is where the cat is, draw a box around it." Gradually, it learns cat-shapes, car-shapes, person-shapes. When it gets one wrong, we tell it, "The box should be a bit to the left" or "That's not a cat, it's a dog!" and it adjusts.

The magic: It not only says what (classification), but also where (localization)—drawing boxes around every object!

Or: Refined, Yappy, Scalable

Connections

• Convolutional Neural Networks: The backbone feature extractors (VGG, ResNet, Darknet)
• Image Classification: Object detection extends classification with localization
• Semantic Segmentation: Pixel-level labeling vs. box-level in detection
• Transfer Learning: Detectors often start with ImageNet-pretrained backbones
• Anchor Boxes: Core technique in Faster R-CNN, YOLO v2+, SSD
• Non-Maximum Suppression: Post-processing to remove duplicate detections
• Intersection over Union (IoU): Evaluation metric and matching criterion
• Batch Normalization: Stabilizes training in deep detector backbones
• Data Augmentation: Critical for detection (random crops, flips, color jitter)
• Instance Segmentation: Next step beyond detection (Mask R-CNN)

Flashcards

#flashcards/ai-ml

What is the fundamental difference between image classification and object detection? :: Image classification outputs a single label for the whole image. Object detection outputs multiple bounding boxes with class labels, solving both "what" (classification) and "where" (localization).

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)