Saliency maps and Grad-CAM: Question bank — AI-ML deep dive

Before we start, three words we'll lean on:

logit = the raw class score $S_{c}$ the network outputs before softmax turns it into a probability.
feature map $A^{k}$ = one 2-D grid of activations from a convolutional layer; a CNN produces many of them stacked in depth.
gradient $\frac{\partial S _{c}}{\partial ( \cdot )}$ = "if I nudge this thing up by a hair, how much does the score move?"

True or false — justify

Is a saliency map computed with respect to the network's weights?

False — it is the gradient of the class score with respect to the input pixels (

\nabla_{I} S_{c}

). Weight gradients are for training; input gradients ask "which pixels matter."

A saliency map takes the absolute value of the gradient — does the sign carry no useful information?

The sign does carry information (whether raising or lowering a pixel raises the score), but the saliency score deliberately discards it because we want magnitude of influence, not direction, for a single "importance" heatmap.

True or false: Grad-CAM and a plain saliency map produce heatmaps of the same resolution.

False — saliency is pixel-resolution (

H \times W

), while Grad-CAM starts at the last conv layer's coarse grid (e.g.

7 \times 7

) and is upsampled, so it is smooth and blocky, not pixel-sharp.

Grad-CAM's weights

α_{k}^{c}

are one number per pixel of the feature map.

False — global average pooling collapses each feature map's spatial grid to a single scalar

α_{k}^{c}

, so there is one weight per feature map

k

, not per pixel.

True or false: applying ReLU inside Grad-CAM is optional cosmetic cleanup.

False — ReLU removes regions with negative contribution (evidence against the class); dropping it mixes "where the model sees the object" with "where it sees the opposite," which is misleading for localization.

The softmax step should be applied before backpropagating for a saliency map.

False — softmax couples every class (

p_{c}

depends on all logits), so its gradient reflects other classes too; you backprop from the raw logit

S_{c}

to isolate that one class's sensitivity.

True or false: a bright Grad-CAM region always means the model reasoned correctly.

False — brightness only says "the model used this region for class

c

." If it lights up a watermark or hospital logo, the model is confidently cheating, which is exactly why we use it for Model debugging.

Saliency maps require a special second network to explain the first.

False — they are just one extra backward pass through the same network, so they are essentially free and model-internal (unlike LIME and SHAP, which fit separate surrogate models).

Spot the error

loss = softmax(logits)[c]; loss.backward() — what breaks?

Backpropagating through softmax couples all classes, so the gradient shrinks wherever another logit rises even if

S_{c}

is unchanged; use loss = logits[c] on the raw logit instead.

Someone computes Grad-CAM gradients with respect to the input image and calls it Grad-CAM.

That is (essentially) a saliency map, not Grad-CAM — Grad-CAM needs gradients flowing into the last convolutional layer's feature maps

A^{k}

, which is what gives it spatial, class-discriminative structure.

cam = (alpha * feature_maps).sum(axis=0) with no ReLU — why is this wrong for localization?

Negative-weighted regions (evidence against the class) survive and get visualized, so the map no longer purely shows where positive evidence for the class lives; wrap it in relu(...).

A student normalizes the saliency map but forgets to take an absolute value or L2/max over the RGB channels — what's the bug?

The three color-channel gradients stay separate (and signed), so there's no single per-pixel importance; aggregate across channels (max or L2 norm) to get one

H \times W

map.

Someone takes Grad-CAM from an early conv layer to get a sharper heatmap.

Early layers encode edges/textures, not objects, so you gain resolution but lose the high-level "this is a dog's face" meaning — Grad-CAM deliberately uses the last conv layer where features are semantic.

"I'll interpret negative saliency pixels as 'safe/ignored' pixels."

Wrong — a large negative gradient means changing that pixel strongly lowers the score, so it is highly influential; only near-zero gradient means "the model ignores it."

Why questions

Why use raw logits rather than probabilities as the target for gradients?

Logits measure that one class independently, while probabilities are a relative quantity across classes, so logit gradients give clean, isolated importance for the chosen class.

Why does Grad-CAM average the gradients over space to form

α_{k}^{c}

instead of keeping them spatial?

The goal is one importance weight per whole feature map

A^{k}

; averaging asks "how much does feature

k

overall help class

c

," letting the feature map's own spatial pattern supply the "where."

Why is Grad-CAM smoother and less noisy than a raw saliency map?

Deep feature maps already aggregate information over large receptive fields, whereas per-pixel input gradients are chaotic and high-frequency, so pixel saliency looks like speckled noise.

Why is Grad-CAM called "class-discriminative" while a plain feature-map heatmap is not?

The weights

α_{k}^{c}

come from the gradient of a specific class score, so the same image yields a different heatmap for "dog" vs "cat"; plain activations don't know which class you asked about.

Why can we say Grad-CAM and Attention mechanisms are related but not the same?

Both highlight where the model "looks," but attention weights are learned and used during the forward pass, whereas Grad-CAM is a post-hoc gradient-based visualization that the network never uses to predict.

Why does the parent note warn that a good-looking heatmap can still hide an adversarial vulnerability?

Grad-CAM shows where the model attends on normal inputs; a tiny crafted perturbation can flip the prediction while barely moving the heatmap, so a clean map is not proof of robustness.

Edge cases

What does a saliency map look like if the class score is (locally) flat in the input — every gradient near zero?

The map is nearly uniform/dark everywhere, meaning no single pixel dominates; this often signals saturation (e.g. the model is very confident and gradients have vanished).

What happens to Grad-CAM if every

α_{k}^{c}

turns out negative for the chosen class?

The weighted sum is entirely negative, and ReLU zeros it out, giving a blank heatmap — a legitimate signal that this layer found no positive evidence for that class.

If two classes ("cat" and "dog") have nearly identical logits, is the saliency map meaningless?

Not meaningless, but interpret cautiously — near a decision boundary the gradient of one logit is very sensitive to tiny changes, so the map can look unstable; comparing per-class Grad-CAMs is more informative here.

For a grayscale (single-channel) image, does the channel-aggregation step still apply?

There is only one channel, so max/L2 over channels is trivial and you can take the absolute gradient directly — no aggregation needed.

What does Grad-CAM produce for an untrained or randomly-initialized network?

A near-random, uninformative heatmap, since

α_{k}^{c}

reflects meaningless gradients — a useful sanity check: a properly trained model should localize the object, random weights should not.

If the last conv layer is

1 \times 1

spatially (fully pooled already), what is the Grad-CAM output?

The heatmap has a single spatial cell, so upsampling produces a flat overlay with no localization — Grad-CAM needs spatial feature maps to point anywhere.

Recall One-line summary of the traps

Saliency = input gradients (noisy, pixel-level, use raw logits, take magnitude); Grad-CAM = last-conv feature maps weighted by pooled class gradients then ReLU'd and upsampled (smooth, class-specific, region-level).