Saliency maps and Grad-CAM: Exercises — AI-ML deep dive

Definition Minimal intuitions you need before the problems (in case the parent notes aren't open)

Logit $S_{c}$ — the raw number the network outputs for class $c$ before softmax. Higher = "more confident it's class $c$ ". We differentiate this, never the softmax probability.
Feature map $A^{k}$ — inside a CNN, each convolutional layer outputs a stack of 2-D grids. The $k$ -th grid is $A^{k}$ ; a cell that lights up means "this pattern (edge, texture, part) was detected here in the image." Deep layers detect object parts; that is why Grad-CAM reads them.
Spatial dimensions $u \times v$ — the height and width of that grid (e.g. $7 \times 7$ in ResNet-50's last conv layer). It is small because pooling has shrunk the image. The number of cells is $Z = u \cdot v$ .
Gradient $\partial S_{c} / \partial (something)$ — "how much does the score $S_{c}$ move if I nudge that something a tiny bit?" One backward pass computes it everywhere at once.

Before we start, one picture fixes all the notation used below.

Figure s01 — The image is a stack of three brightness grids, one per colour channel (magenta = R, violet = G, orange = B). One pixel $I_{x, y, c}$ is highlighted. The orange arrow is the gradient $\partial S_{c} / \partial I_{x, y, c}$ : it carries the question "how much does the class score $S_{c}$ (the box on the right, a raw logit) move if I nudge this one pixel?" Saliency is just the size of that arrow, collapsed to one number per pixel.

Level 1 — Recognition

L1.1 — What does a gradient sign mean?

The gradient of the "cat" logit with respect to one specific pixel — call it $I_{x_{0}, y_{0}, c_{0}}$ , sitting at some fixed row $x_{0}$ , column $y_{0}$ , channel $c_{0}$ — is $\frac{\partial S _{cat}}{\partial I _{x_{0}, y_{0}, c_{0}}} = - 0.8$ . In plain words, what does the sign tell you, and what does the saliency map store for this pixel?

Recall Solution

Sign first. A negative gradient means: if you increase this pixel's brightness, the cat score goes down (and if you decrease it, the cat score goes up). The pixel is currently "fighting" the cat prediction — but it is still influential.

What the saliency map stores. Saliency measures magnitude of influence, not direction, so we take the absolute value: $Saliency (x_{0}, y_{0}) = ∣ - 0.8 ∣ = 0.8$ A pixel with gradient $+ 0.8$ and one with $- 0.8$ are equally important on the map.

L1.2 — Which layer, which object — and why?

Fill the blanks and justify each choice: A saliency map takes gradients at the ______ layer, giving a ______-level map. Grad-CAM takes gradients at the ______ layer, giving a ______ map that must be ______ back to image size. In one sentence each, say why those particular layers are chosen.

Recall Solution

Saliency: gradients at the input layer → pixel-level (high-resolution, noisy) map. Why the input layer? We want per-pixel sensitivity, and the only place with one value per pixel is the input itself — differentiating there gives full image resolution.
Grad-CAM: gradients at the last convolutional layer → coarse / region-level (e.g. $7 \times 7$ ) map → upsampled (bilinear interpolation) to image size. Why the last conv layer? Its feature maps encode high-level object parts (a face, a paw) with spatial layout still intact, so gradients there tell us which region supports the class — something pixel gradients are too noisy to reveal. It's small, so we upsample afterwards.

L1.3 — Reveal cards

Softmax before backward?

Wrong — softmax couples all classes; use raw logits so you isolate one class score.

Grad-CAM uses ReLU because...

we want only positive evidence (regions that raise the class score), not regions that suppress it.

Level 2 — Application

L2.1 — Aggregate three channels into one number

At one pixel the per-channel gradients of $S_{cat}$ are $\frac{\partial S _{cat}}{\partial I _{R}} = 3, \frac{\partial S _{cat}}{\partial I _{G}} = - 4, \frac{\partial S _{cat}}{\partial I _{B}} = 0.$ Compute the pixel's saliency value using (a) the max-abs rule and (b) the L2-norm rule.

Recall Solution

A saliency map needs one number per pixel, but the gradient has three (one per colour). Two standard collapses:

(a) Max of absolute values: $max (∣3∣, ∣ - 4 ∣, ∣0∣) = max (3, 4, 0) = 4.$

(b) L2 norm across channels: $3^{2} + (- 4)^{2} + 0^{2} = 9 + 16 + 0 = 25 = 5.$

Both are valid; L2 blends all channels, max picks the single loudest one.

L2.2 — Global average pooling of gradients

For Grad-CAM, one feature map $A^{k}$ has spatial size $u \times v = 2 \times 2$ . The gradients of $S_{dog}$ flowing into it are $\frac{\partial S _{dog}}{\partial A ^{k}} = [23 - 1 0] .$ Compute the channel weight $α_{k}^{dog}$ .

Recall Solution

The weight is the global average of the gradients over all $Z = u \cdot v = 2 \cdot 2 = 4$ spatial locations (here $u = v = 2$ , the grid's height and width): $α_{k}^{dog} = \frac{1}{Z} \sum_{i, j} \frac{\partial S _{dog}}{\partial A _{i, j}^{k}} = \frac{2 + ( - 1 ) + 3 + 0}{4} = \frac{4}{4} = 1.$ Why average, not sum? Dividing by $Z = u \cdot v$ makes the weight independent of feature-map size, so a $7 \times 7$ and a $14 \times 14$ map are comparable.

L2.3 — Build a Grad-CAM heatmap by hand

Two feature maps, each $2 \times 2$ , with their weights: $A^{1} = [1201], α_{1} = 2; A^{2} = [0113], α_{2} = - 1.$ Compute $L = ReLU (α_{1} A^{1} + α_{2} A^{2})$ .

Recall Solution

Weighted sum first: $2 A^{1} = [2402], - 1 \cdot A^{2} = [0 - 1 - 1 - 3] .$ $α_{1} A^{1} + α_{2} A^{2} = [23 - 1 - 1] .$ Now ReLU (replace every negative with 0): $L = [2300] .$ The bottom-left cell (value 3) is where the model found the strongest positive dog-evidence.

Level 3 — Analysis

L3.1 — Why softmax poisons saliency

A network gives logits $S_{cat} = 2$ , $S_{dog} = 1$ , $S_{fox} = 0$ . Suppose a pixel nudge leaves $S_{cat}$ unchanged but raises $S_{dog}$ by a lot. Explain, using the softmax probability, why differentiating the probability $p_{cat}$ would wrongly flag this pixel as important.

Recall Solution

The softmax probability is $p_{cat} = \frac{e ^{S_{cat}}}{e ^{S_{cat}} + e ^{S_{dog}} + e ^{S_{fox}}} .$ Its numerator depends only on $S_{cat}$ , but its denominator depends on every logit. So if $S_{dog}$ grows while $S_{cat}$ stays put, the denominator grows and $p_{cat}$ falls.

Concretely, before: denominator $= e^{2} + e^{1} + e^{0} \approx 7.389 + 2.718 + 1 = 11.107$ , so $p_{cat} \approx 7.389/11.107 \approx 0.665$ . If $S_{dog}$ jumps to $3$ : denominator $\approx 7.389 + 20.086 + 1 = 28.475$ , $p_{cat} \approx 7.389/28.475 \approx 0.259$ .

The pixel changed $p_{cat}$ enormously ( $0.665 \to 0.259$ ) without touching the cat score at all. Differentiating $p_{cat}$ therefore blames this pixel for something it did to the dog class. Differentiating the raw logit $S_{cat}$ correctly reports zero influence. That is why saliency uses raw logits.

L3.2 — Diagnosing a cheating model

Grad-CAM for the "horse" class lights up a small bright patch in the bottom-left corner of every horse photo, and almost nothing on the horse itself. What is the most likely explanation, and which class of tool from Model debugging does this reveal?

Recall Solution

The corner patch is almost certainly a dataset artifact — a copyright watermark or source tag that co-occurs with horses in the training set. The model learned the shortcut "watermark ⇒ horse" instead of "horse shape ⇒ horse."

This is a textbook spurious-correlation / shortcut-learning bug, exactly the kind Model debugging exists to catch. The fix is data-side (remove/randomise the watermark, augment), not architectural. Grad-CAM did its job: it made the shortcut visible.

L3.3 — Coarse vs fine, and why

Explain why a saliency map is high-resolution but noisy, while Grad-CAM is low-resolution but smooth. Tie each property to where the gradients are taken.

Recall Solution

Saliency differentiates w.r.t. the input grid, so it has one value per pixel — full $224 \times 224$ resolution. But individual pixel gradients are chaotic: tiny, locally-varying, sensitive to noise in a single pixel. High resolution, high noise.
Grad-CAM differentiates w.r.t. the last conv layer, whose spatial grid is tiny (e.g. $7 \times 7$ ) because pooling has shrunk it. Each cell of that grid summarises a whole region of the image and encodes high-level parts (a face, a paw). Averaging gradients over space (GAP) smooths further. Low resolution, low noise.

The trade-off is fundamental: resolution comes from being near the input; semantic smoothness comes from being deep. You cannot have both from one map — hence the two methods coexist.

Level 4 — Synthesis

L4.1 — Full Grad-CAM from raw gradients

Last conv layer has two channels, spatial size $2 \times 2$ . Activations and gradients of $S_{c}$ : $A^{1} = [1021], \frac{\partial S _{c}}{\partial A ^{1}} = [0 - 2 20]; A^{2} = [3102], \frac{\partial S _{c}}{\partial A ^{2}} = [1111] .$ Produce the final (pre-upsample) Grad-CAM heatmap.

Recall Solution

Step 1 — channel weights (GAP of gradients), $Z = 4$ : $α_{1} = \frac{0 + 2 + ( - 2 ) + 0}{4} = 0, α_{2} = \frac{1 + 1 + 1 + 1}{4} = 1.$ Step 2 — weighted sum: $0 \cdot A^{1} + 1 \cdot A^{2} = A^{2} = [3102] .$ Step 3 — ReLU (no negatives here, unchanged): $L = [3102] .$ Notice channel 1 vanished entirely: its gradients averaged to zero, so it contributed nothing to the class evidence — the method automatically ignored an irrelevant feature.

L4.2 — Why the naive Taylor argument justifies saliency

Starting from a first-order Taylor expansion of $S_{c}$ around the current image, show that the coefficient multiplying a pixel perturbation $ϵ$ is exactly the gradient, and explain why this makes the gradient the right "importance" number.

Recall Solution

Perturb one pixel: $I_{p} \to I_{p} + ϵ$ . First-order Taylor expansion of the score: $S_{c} (I_{p} + ϵ) \approx S_{c} (I_{p}) + \frac{\partial S _{c}}{\partial I _{p}} ϵ .$ The change in score is $Δ S_{c} \approx \frac{\partial S _{c}}{\partial I _{p}} ϵ .$ For a fixed small nudge $ϵ$ , the size of $Δ S_{c}$ is directly proportional to $∣ \partial S_{c} / \partial I_{p} ∣$ : a pixel whose gradient is large moves the score a lot per unit nudge, and a pixel whose gradient is near zero barely moves it at all. So the gradient magnitude answers precisely the question "how much does the score respond when I perturb this pixel?" — which is exactly what we mean by a pixel's importance. Because this is a first-order (linear) statement, it is the cheapest possible importance measure: a single backward pass to the input evaluates it for every pixel at once — no per-pixel perturbation loop is needed (contrast LIME and SHAP, which do perturb the input many times to estimate importance). The absolute value appears because both a large positive and a large negative gradient signal strong influence; only the size of $Δ S_{c}$ matters, not its sign.

Level 5 — Mastery

L5.1 — Adversarial trust-break

You have a correctly-classified panda image with a clean Grad-CAM on the panda's face. An adversarial perturbation flips the label to "gibbon" while the image looks unchanged to humans. Predict what happens to (a) the saliency map and (b) the Grad-CAM for the new class, and state the sobering lesson for interpretability.

Recall Solution

(a) Saliency for "gibbon" will look like structured noise concentrated on the perturbed pixels — it will produce a confident-looking map even though nothing gibbon-like is there. Pixel gradients happily explain the adversarial signal.

(b) Grad-CAM for "gibbon" often still lights up plausible object regions (or shifts subtly), because the adversary manipulated the deep features. It can look reasonable and thus falsely reassuring.

Lesson: interpretability tools explain the model's computation, not ground truth. A convincing heatmap does not certify correctness — a fooled model produces fooled explanations. Use Grad-CAM as a sanity check under normal conditions, never as a security or correctness guarantee. Concretely: never sign off on a model just because its heatmap looks sensible on a handful of clean images — pair it with adversarial testing, out-of-distribution checks, and independent explanation methods (LIME and SHAP) before you trust the result.

L5.2 — Design a debugging protocol

Design a minimal 4-step protocol using Grad-CAM to decide whether a skin-cancer classifier is trustworthy before deployment.

Recall Solution

A defensible protocol:

Hold-out Grad-CAM audit. On a labelled test set, generate class- $c$ Grad-CAM for the predicted class of every correctly-classified image.
Region overlap metric. With expert lesion masks, compute the fraction of Grad-CAM mass falling inside the lesion. High mass on lesion = focusing on the right thing.
Confounder sweep. Deliberately hunt for off-lesion hotspots (rulers, ink marks, skin-tone borders). Recurrent off-lesion focus = shortcut learning → flag for data cleaning (Model debugging).
Sanity check (Guided-vs-random). Confirm Grad-CAM changes when you change the target class and degrades on a randomly-initialised network. If the map is identical regardless, the explanation is meaningless.

Only if lesion-overlap is high, confounders are absent, and sanity checks pass do you tentatively trust the model — and even then (see L5.1) not against adversaries.

Recall Self-test summary card

Saliency = gradient at ::: the input layer, absolute value, one map per pixel. Grad-CAM weight $α_{k}^{c}$ = ::: global average of $\partial S_{c} / \partial A^{k}$ over all $Z = u \cdot v$ spatial positions. Grad-CAM heatmap = ::: $ReLU (\sum_{k} α_{k}^{c} A^{k})$ , then upsample. Always differentiate the ::: raw logit, never the softmax probability.