Object detection (R-CNN, YOLO, SSD): Visual walkthrough

You have seen the parent note say things like "predict $(x, y, w, h)$ " and "IoU tells us how well the box fits." But where do these numbers come from? What does it mean for a network to draw a box? This page builds the whole idea of a predicted bounding box — from a blank grid of pixels to a scored, refined rectangle — one picture at a time.

We will not assume you know what a coordinate, an anchor, a regression target, or IoU is. We earn each one.

Parent: Object detection · prerequisites: Image Classification, Anchor Boxes, Intersection over Union (IoU), Non-Maximum Suppression.

Step 1 — What is an image, and what is a box on it?

Figure — Object detection (R-CNN, YOLO, SSD)

WHAT. We lay a coordinate system over the image. By convention in vision, $x$ grows to the right and $y$ grows downward (the origin is the top-left corner — unlike school graphs).

WHY downward? Because pixels are stored row-by-row from the top; the first row is $y = 0$ . Matching the code convention avoids sign confusion later.

PICTURE. A rectangle sitting on this grid is fully described by four numbers:

center, left-right x_{c}, center, up-down y_{c}, width w, height h

$x_{c}, y_{c}$ = the coordinates of the center dot of the box.
$w$ = how wide the box is (in pixels).
$h$ = how tall it is.

That is the entire target. Everything else on this page is a clever way to produce these four numbers.

Step 2 — Why not just have the network spit out the four numbers?

WHAT. We stop predicting absolute pixels. Instead we chop the image into an $S \times S$ grid of cells (the YOLO idea) and say: the cell that contains the object's center is responsible for it.

WHY a grid? Two reasons:

It converts "search the whole image" into "each cell answers a small local question."
It gives every prediction a reference point — the cell's own corner — so the numbers become local and small.

PICTURE. The object's center falls in one cell (highlighted). That cell predicts the center as a fraction of its own width:

x = how far across the cell, 0 to 1 \frac{x _{c} - cell _{x}}{cell width}, y = \frac{y _{c} - cell _{y}}{cell height}

$x_{c} - cell_{x}$ = distance from the cell's left edge to the center.
Divide by cell width → a number between 0 and 1. $0$ = center hugs the left edge, $1$ = right edge.

Width and height are normalized against the whole image instead, so they also live in $[0, 1]$ :

w = \frac{box width}{image width}, h = \frac{box height}{image height}

Now every one of the four numbers is a clean fraction. That is what makes the target learnable.

Step 3 — The anchor: giving the network a rough guess to correct instead of invent

WHAT. At each grid position we stamp down $k$ anchors $P = (P_{x}, P_{y}, P_{w}, P_{h})$ of fixed sizes and aspect ratios. The network outputs four offsets $(d_{x}, d_{y}, d_{w}, d_{h})$ that transform the anchor $P$ into the predicted box $\hat{G}$ .

WHY offsets and not the box directly? Because a small correction to a good starting shape is a gentle, mostly-linear thing to learn. Inventing absolute geometry is not.

PICTURE. The dotted grey box is the anchor $P$ . The solid orange box is the corrected prediction $\hat{G}$ . The transformation (from the parent note):

\hat{G}_{x} = scale by anchor width P_{w} \cdot how far to shift, in widths d_{x} + start at anchor center P_{x}

\hat{G}_{w} = anchor width P_{w} \cdot multiply, always positive exp (d_{w})

Read term-by-term:

$d_{x}$ shifts the center sideways, measured in anchor widths — so the same $d_{x}$ means "10% of the box" whether the box is big or small (scale-invariant).
$exp (d_{w})$ = $e^{d_{w}}$ . Why the exponential? A width cannot be negative, and $e^{(anything)}$ is always positive. If $d_{w} = 0$ , then $exp (0) = 1$ and the width is unchanged — a natural "do nothing" default.

Step 4 — How do we score a box? Meet IoU

WHAT. Take two rectangles. Compute the area they share (intersection) and the area they cover together (union). Divide.

IoU = \frac{Area of overlap}{Area of union} = \frac{∣ A \cap B ∣}{∣ A \cup B ∣}

$∣ A \cap B ∣$ = the overlapping patch (teal in the figure).
$∣ A \cup B ∣$ = total footprint of both boxes = $∣ A ∣ + ∣ B ∣ - ∣ A \cap B ∣$ (we subtract the overlap so it's not double-counted).

WHY divide? Raw overlap area is unfair — big boxes overlap more just by being big. Dividing by the union normalizes: the result always lands in $[0, 1]$ no matter the box sizes.

PICTURE / cases.

Perfect match → overlap = union → IoU $= 1$ .
No touching → overlap $= 0$ → IoU $= 0$ .
Half-covering → typically around IoU $\approx 0.3$ – $0.5$ .

This single number drives two things: which anchor is "responsible" (highest IoU with the object), and the confidence target below.

Step 5 — Confidence: does this box even contain an object?

WHAT. Confidence is defined (parent note) as:

confidence = is anything here? \in [0, 1] P (object) \times how snug is the box? IoU_{pred}^{truth}

WHY multiply? Both factors must be high to earn a high score:

If there's no object, $P (object) = 0$ → confidence $0$ , even if the box happens to be well-placed on nothing.
If there is an object but the box is loose, IoU is low → confidence drops. It honestly reports "there's a thing, but my box is sloppy."

PICTURE. The figure shows three cases as bars: empty cell (conf $\approx 0$ ), object with tight box (conf high), object with loose box (conf medium).

To finish the job — what object — the cell also predicts class probabilities $P (C_{i} ∣ object)$ , and we combine:

class-specific score P (C_{i}) = if there’s an object, is it a dog? P (C_{i} ∣ object) \times confidence P (object) \times IoU

Worked number from the parent: $P (dog ∣ obj) = 0.92$ , confidence $= 0.85$ → final $0.92 \times 0.85 = 0.782$ .

Step 6 — Why the size loss uses a square root

WHAT. In the localization loss we compare $w$ and $\overset{w}{^}$ rather than $w$ and $\overset{w}{^}$ :

(w - \overset{w}{^})^{2} + (h - \hat{h})^{2}

WHY ? The square-root curve is steep for small values and flat for large ones. So the same absolute pixel error produces a bigger loss on small boxes and a smaller loss on large boxes — exactly the fairness we wanted.

PICTURE. The figure plots $w$ against $w$ : a fixed step $Δ w$ at $w = 0.1$ climbs far more vertically than the same $Δ w$ at $w = 0.8$ . That vertical climb is the loss.

Step 7 — The degenerate cases (never skipped)

Case A — Empty cell (no object). $P (object) = 0$ , so confidence $= 0$ , and $1^{obj} = 0$ kills every localization term. Only the $λ_{noobj}$ confidence term survives, gently pushing the box's confidence toward 0.

Case B — Zero offset. If $d_{w} = d_{h} = d_{x} = d_{y} = 0$ , then $exp (0) = 1$ and the predicted box $\hat{G}$ equals the anchor $P$ exactly. This is the "I have nothing to correct" default — a sane baseline.

Case C — Non-overlapping boxes. Intersection area $= 0$ → IoU $= 0$ . The union is just $∣ A ∣ + ∣ B ∣$ , never zero (unless both boxes have zero area), so no division-by-zero.

Case D — Two boxes on one object (duplicate detections). Multiple cells/anchors may all fire on the same dog. We remove duplicates with Non-Maximum Suppression: keep the highest-confidence box, delete any other box whose IoU with it exceeds a threshold. The figure shows three overlapping dog boxes collapsing to one.

Case E — Tiny vs huge object. The grid + multi-scale feature maps (SSD) and the size loss together ensure small objects aren't ignored — small objects are detected on high-resolution early layers, large ones on coarse late layers.

The one-picture summary

The whole pipeline in one frame: pixels → grid → responsible cell → anchor → predicted offsets → scored box (confidence × IoU) → NMS cleanup → final labeled box.

Recall Feynman retelling — say it back in plain words

An image is a grid of pixels, and a box is just four numbers: where its middle is and how big it is. Predicting those raw pixel numbers is hard, so we tile the image into cells and let each cell describe a box relative to itself — nice small fractions between 0 and 1. To make it even easier we pre-place standard rectangles called anchors and only predict tiny nudges (shift the center, stretch the width). We use $exp$ for the size nudge so width is always positive, and its inverse $lo g$ to compute the training target. Each box also carries a confidence: "is there really an object here?" times "how snug is my box?" — where snugness is IoU, the shared area divided by the combined area. We train it all with a loss that only blames the responsible box, cares a lot about position ( $λ_{coord} = 5$ ), cares little about the sea of empty boxes ( $λ_{noobj} = 0.5$ ), and uses square roots on width and height so small boxes are judged as strictly as big ones. Finally, when several boxes crowd one object, non-maximum suppression keeps the most confident and deletes the rest.

Recall Quick self-test

What do the four box numbers mean? ::: Center $x$ , center $y$ , width, height — two say where the middle is, two say how big. Why predict offsets from an anchor instead of the box directly? ::: A small correction to a good starting shape is gentle and near-linear to learn; inventing absolute geometry is hard. Why $exp$ for width? ::: It's always positive (widths can't be negative), and $exp (0) = 1$ leaves the anchor unchanged. Why divide overlap by union in IoU? ::: To normalize away box size, so the score always lands in $[0, 1]$ . Why the square root on $w, h$ in the loss? ::: So the same pixel error costs more on small boxes than large ones — proportional fairness. What removes duplicate boxes on one object? ::: Non-Maximum Suppression: keep the top-confidence box, drop others with high IoU to it.