Saliency maps and Grad-CAM: Visual walkthrough

Step 1 — What a convolutional layer actually outputs

WHAT. When a photo passes through a CNN, the last convolutional layer does not output one image — it outputs a stack of small grids. In ResNet-50 this stack is $7 \times 7 \times 2048$ : that is $2048$ separate grids, each grid $7$ rows by $7$ columns.

WHY. Each grid is trained to react to one recurring visual thing — one grid might light up on fur texture, another on floppy-ear shapes, another on grass. A high number in a cell means "the thing I detect is present here, at this spot."

PICTURE. Below, three of the $2048$ grids are drawn as $7 \times 7$ boards. Brighter cell = bigger number = "my thing is here."

We name these grids with symbols so we can talk about them:

Step 2 — The class score, and why we differentiate it

WHAT. After the conv stack, the network keeps processing until it hits one number per class. Call the number for "dog" the class score $S^{dog}$ , or in general $S^{c}$ .

WHY (which tool, and why not another). We want to know: how much does each detector grid matter for saying "dog"? The mathematical tool for "if I nudge this input a little, how much does the output move?" is the partial derivative — nothing else answers a sensitivity question. So we compute

$\frac{\partial S ^{c}}{\partial A _{i, j}^{k}} .$

Read it out loud: "if the firing at spot $(i, j)$ of grid $k$ went up by a tiny bit, how much would the dog-score $S^{c}$ go up?" A big positive value = this spot pushes the score toward "dog."

WHY the raw logit, not the probability. If we differentiated the softmax probability instead, raising a different class's score would push our probability down and pollute the gradient. Differentiating the raw $S^{c}$ isolates "dog" cleanly (this is the softmax pitfall from the parent note).

PICTURE. A tiny nudge $+ ε$ at one cell travels up the network and the class-score needle twitches. The size of the twitch, divided by $ε$ , is the gradient.

Step 3 — One gradient grid per detector

WHAT. Do that derivative at every cell of every grid. The result is a stack the exact same shape as $A$ : $7 \times 7 \times 2048$ . We call one gradient grid

$G_{i, j}^{k} = \frac{\partial S ^{c}}{\partial A _{i, j}^{k}} .$

WHY. The forward pass gave us "what fired where" ( $A$ ). The backward pass gives us "what mattered where" ( $G$ ). We now have both, cell for cell, and can combine them.

WHAT IT LOOKS LIKE. Beside each activation grid sits a gradient grid of the same size. Green cells push the dog-score up ( $G > 0$ ); pink cells push it down ( $G < 0$ ).

Step 4 — Collapse each gradient grid to one number ( $α_{k}^{c}$ )

WHAT. For each detector $k$ , average its whole $7 \times 7$ gradient grid into a single number:

WHY average and not, say, the max? A feature map at the last conv layer is spatially coarse — $7 \times 7$ cells cover the whole image. We do not want the exact spot from the gradient (the spot information will come back from $A^{k}$ itself in the next step). Here we only want one honest importance weight per detector. Averaging over all spots gives a stable, un-noisy summary of "does this detector help identify a dog at all?" This averaging step is called global average pooling.

PICTURE. A $7 \times 7$ gradient grid funnels through an averaging box and drops out as a single dot $α_{k}^{c}$ . Do this $2048$ times → a row of $2048$ dots.

Step 5 — Weighted sum: rebuild a single grid

WHAT. We now build the coarse heatmap — the raw, un-cleaned Grad-CAM map, which we call $M$ . It is a single $7 \times 7$ grid made by using each weight $α_{k}^{c}$ to scale its detector's activation grid $A^{k}$ and adding them all up:

WHY this exact multiply-then-add. It is a vote. A detector that both (a) matters a lot for dogs ( $α_{k}^{c}$ big) and (b) fired strongly in a region ( $A^{k}$ big there) contributes a large bright value at that region. Detectors that don't matter ( $α_{k}^{c} \approx 0$ ) contribute nothing, wherever they fired. Because $A^{k} \geq 0$ , each vote's sign is simply the sign of $α_{k}^{c}$ : helpful detectors add brightness, harmful ones subtract it. The result $M$ is a single $7 \times 7$ grid: bright where dog-relevant detectors were active.

PICTURE. Each $7 \times 7$ activation grid is dimmed or brightened by its scalar weight, then all are stacked and summed into one $7 \times 7$ result grid.

Step 6 — ReLU: keep only the "for dog" evidence

WHAT. Apply ReLU to every cell of $M$ :

$L_{Grad-CAM}^{c} = ReLU (\sum_{k} α_{k}^{c} A^{k}) .$

WHY. A negative cell in $M$ means "this region lowers the dog-score" — evidence against dog (maybe it looks like a cat there). For localization — "where is the dog?" — negative evidence is a distraction. ReLU deletes it, keeping only spots that argue for the class. This is the "forgetting ReLU" pitfall from the parent note: skip it and background-suppression regions leak into your heatmap.

PICTURE. The signed grid (green/pink) goes in; pink (negative) cells snap to dark zero; only the positive glow survives.

Step 7 — Normalize, upsample, and overlay

WHAT. $L^{c}$ is only $7 \times 7$ and its raw numbers can be any positive scale (maybe $0$ to $12.7$ , maybe $0$ to $0.003$ ). Before we can color it, we min–max normalize it into $[0, 1]$ :

$\tilde{L}^{c} = \frac{L ^{c} - m i n L ^{c}}{m a x L ^{c} - m i n L ^{c}},$

$min L^{c}, max L^{c}$ — the smallest and largest cell values of the $7 \times 7$ map.
Subtracting the min pins the dimmest cell to $0$ ; dividing by the range pins the brightest cell to $1$ .
Now $\tilde{L}^{c} \in [0, 1]$ , so it maps cleanly onto a color scale (dark = $0$ , hot = $1$ ).

Then we stretch $\tilde{L}^{c}$ smoothly (bilinear interpolation) up to the image size $224 \times 224$ and lay it over the original photo as a translucent heatmap.

WHY normalize? A colormap needs inputs in a fixed range to be meaningful and comparable across images; the raw scale of $L^{c}$ is arbitrary. Min–max normalization makes "the brightest region in this image" always map to full glow.

WHY upsample? The $7 \times 7$ grid is correct but blocky — each cell already corresponds to a large image region. Bilinear interpolation fills the in-between pixels with a smooth ramp so the overlay reads as a soft glow instead of $49$ hard tiles. The overlay lets a human directly compare "where it glows" against "where the dog is."

PICTURE. The $7 \times 7$ glow is rescaled to $[0, 1]$ , blown up into a smooth blob, and settled over the dog's face on the photo.

Step 8 — Edge and degenerate cases

WHAT & WHY (each case, so you never get surprised):

All gradients zero for class $c$ . If the network is totally flat about "dog" here, every $α_{k}^{c} = 0 \Rightarrow M = 0 \Rightarrow$ blank heatmap. Correct: nothing pushed toward dog. (Often happens when you pick a class the model does not predict at all.)
All weights negative. Then $M \leq 0$ everywhere (since $A^{k} \geq 0$ ) and ReLU wipes it to a blank map. The honest reading: no region gives positive evidence for this class. A blank Grad-CAM is a real answer, not a bug.
One detector dominates. If a single weight, call it $α_{k^{'}}^{c}$ , dwarfs the rest, then $M \approx α_{k^{'}}^{c} A^{k^{'}}$ — the heatmap is essentially that one detector's firing pattern. Perfectly valid; the model leaned on one feature.
Very deep, tiny feature map (e.g. $1 \times 1$ ). Then $Z = 1$ , "global average" averages a single cell, and Grad-CAM degenerates to a single scalar per class — no spatial resolution left. Lesson: pick a conv layer with real spatial size ( $7 \times 7$ or larger).
Choosing the wrong layer entirely. If you attach Grad-CAM after the global-average-pooling / fully-connected head — to post-GAP feature vectors instead of a conv layer — there is no spatial grid left ( $u = v = 1$ ), so there is nothing to localize; you get one number, not a map. And in a ResNet, prefer the last conv layer that still has $7 \times 7$ spatial structure (just before the final pooling); its skip-connections have already merged, so its activations reflect the full high-level evidence. Attaching to an early conv layer instead would give sharp but low-level, non-class-discriminative maps.

PICTURE. Three miniature outcomes side by side: blank (all-negative), single-blob (one detector), and blocky (tiny feature map).

The one-picture summary

The full pipeline on one board: image → conv stack $A^{k}$ → backprop gives gradients $G^{k}$ → average each grid into weights $α_{k}^{c}$ → weight-and-sum the activations into $M$ → ReLU → normalize → upsample → overlay.

Recall Feynman retelling: the whole walkthrough in plain words

The last conv layer hands us $2048$ little $7 \times 7$ scorecards — each scorecard is one detector saying "my thing is here and here" (and these numbers are never negative, since a ReLU sits right after). We ask the network one question: which of these detectors, and how strongly, made you say "dog"? The answer is a gradient — a sensitivity. We smoosh each detector's gradient scorecard down to a single importance number ( $α$ ) by averaging it. Then we go back to the where scorecards, dim the useless detectors, brighten the important ones, and stack them into one glowing $7 \times 7$ grid. We throw away anything negative (that argued against dog), because we only want to point at the dog. We rescale the little grid to $0$ – $1$ so colors mean something, blur it up to full photo size, and lay it on top. Wherever it glows is where the model was looking. If it glows on the dog, trust it. If it glows on the watermark, you just caught the model cheating.

Recall Self-check

What does $α_{k}^{c}$ represent, in words? ::: One scalar per feature map — the overall importance of detector $k$ for class $c$ , made by averaging that map's gradient over all spatial spots. Why raw logit $S^{c}$ and not softmax probability? ::: Softmax couples all classes, so another class rising drags your gradient around; the raw logit isolates class $c$ 's sensitivity. Why average the gradients (global average pooling) instead of taking a max? ::: We only want one stable importance weight per detector; the where information comes back from $A^{k}$ itself in the weighted sum. What does ReLU remove and why? ::: It zeros negative cells — regions that argue against the class — because localization only wants positive evidence for the class. What does a completely blank Grad-CAM map mean? ::: No region gave positive evidence for that class (all weights/values were $\leq 0$ ) — a real answer, not an error. Why min–max normalize $L^{c}$ before coloring? ::: The raw scale of $L^{c}$ is arbitrary; normalizing to $[0, 1]$ makes the brightest region map to full glow so the colormap is meaningful.

🇮🇳 Prefer Hinglish? Yeh sab Hinglish mein. Related: LIME and SHAP, Attention mechanisms, Adversarial examples.