Saliency maps and Grad-CAM

What Are Saliency Maps?

Derivation from scratch:

What we want: Understand which input features (pixels) the model is sensitive to.
How neural networks work: The output $S_c$ is a differentiable function of input $I$ : $S_c = f(I; \theta)$ .
Taylor expansion intuition: If we perturb pixel $I_{x,y}$ by $\epsilon$ , the output changes by approximately $\frac{\partial S_c}{\partial I_{x,y}} \cdot \epsilon$ . Large gradient → large impact.
The saliency score: We take $\left| \frac{\partial S_c}{\partial I_{x,y}} \right|$ (absolute value because both increasing and decreasing a pixel can change the score).
Computation: One backward pass through the network with respect to the input layer (not the weights).

Why absolute value?

A negative gradient means "decreasing this pixel increases the score," which is still important.
We care about magnitude of influence, not direction.

Computing Saliency: Step-by-Step

Worked example Walkthrough: Saliency Map for "Cat" Class

Setup: CNN classifies image → outputs logits $[S_{\text{cat}}, S_{\text{dog}}, \ldots]$ . We want saliency for "cat."

Step 1: Forward pass

Input image $I \in \mathbb{R}^{H \times W \times 3}$ → Network Logits → $S_{\text{cat}}$ (no softmax yet!)
Why no softmax? Softmax squashes gradients. We want raw class scores. Step 2: Backward pass
Compute $\nabla_I S_{\text{cat}}$ (gradient of cat-score w.r.t. input pixels).
Shape: $\mathbb{R}^{H \times W \times 3}$ (same as input).
Why this step? Chain rule gives us $\frac{\partial S_{\text{cat}}}{\partial I}$ by backpropating from output to input. Step 3: Aggregate across color channels
Take max across RGB: $\text{Saliency}(x,y) = \max_c \left| \frac{\partial S_{\text{cat}}}{\partial I_{x,y,c}} \right|$
Or take L2 norm: $\sqrt{\sum_c \left( \frac{\partial S_{\text{cat}}}{\partial I_{x,y,c}} \right)^2}$
Why max/norm? We want one value per pixel for visualization.

Step 4: Visualize

Normalize to [0, 1], apply colormap (hot = important, cold = irrelevant).

Concrete numbers:

Image: 224×224×3
Gradient: 224×224×3 (same shape)
Saliency map: 224×224 (grayscale heatmap)

Grad-CAM: Class Activation Mapping with Gradients

Why this design?

Derivation:

Observation: Deep conv layers capture high-level spatial information (e.g., "cat face here, paws there").
Goal: Weight each feature map by "how much it contributes to class $c$ ."
Gradient as importance: $\frac{\partial S^c}{\partial A^k}$ measures sensitivity of class score to feature map $k$ .
Global average pooling: Instead of pixel-wise weights, average gradients across spatial locations: $\alpha_k^c = \frac{1}{u \cdot v} \sum_{i,j} \frac{\partial S^c}{\partial A_{i,j}^k}$ $α_{k}^{c} = \frac{1}{u \cdot v} \sum_{i, j} \frac{\partial S ^{c}}{\partial A _{i, j}^{k}}$ . This gives one weight per feature map.
- Why average? Feature maps are spatially coarse; we want overall importance of that feature.
Linear combination: $\sum_k \alpha_k^c A^k$ weights each feature map, summing to a single heatmap.
ReLU: Negative values would highlight "what decreases the score" (less interpretable for localization).
Upsample: Feature map is smaller than input (e.g., 7×7 vs. 224×224). Bilinear interpolation to image size.

Worked example Grad-CAM: Classifying "Dog" in 224×224 Image

Architecture: ResNet-50, last conv layer = 7×7×2048 (2048 feature maps, 7×7 spatial). Step 1: Forward pass → $S_{\text{dog}}$ (logit for dog class). Step 2: Compute $\frac{\partial S_{\text{dog}}}{\partial A}$ (gradient w.r.t. last conv layer, shape7×7×2048).

Step 3: Global average pooling over spatial dimensions: $\alpha_k^{\text{dog}} = \frac{1}{49} \sum_{i=1}^{7} \sum_{j=1}^{7} \frac{\partial S_{\text{dog}}}{\partial A_{i,j}^k}$ Now we have2048 scalars (one per feature map). Step 4: Weighted sum of feature maps: $L = \sum_{k=1}^{2048} \alpha_k^{\text{dog}} \cdot A^k$ Result: 7×7 heatmap.

Step 5: ReLU (zero out negative values) → Upsample to 224×224 → Overlay on image.

Interpretation: Bright regions = model focused here for "dog" classification. If it lights up a dog's face: good. If it lights up a watermark: model is cheating.

Saliency vs. Grad-CAM: Key Differences

Aspect	Saliency Maps	Grad-CAM
Granularity	Pixel-level (high-res)	Region-level (coarse, then upsampled)
Layer	Input layer gradients	Last conv layer activations + gradients
What it shows	Which pixels change the score	Which spatial regions activate for class $c$
Noise	Very noisy (pixel gradients are chaotic)	Smother (feature maps aggregate info)
Use case	Fine-grained debugging	Localization, sanity checks

Why Grad-CAM over saliency?

Feature maps in deep layers encode objects/parts (edges → textures → parts → objects). Saliency maps are pixel-level and noisy.
Grad-CAM is class-discriminative: you can visualize "where did the model see the cat vs. the dog?"

Common mistake Common Pitfall: Using Softmax Before Gradients

Wrong approach:

logits = model(image)
probs = softmax(logits)
loss = probs[class_idx]
loss.backward()  # gradient w.r.t. input

Why it feels right: "I want to know what influences the probability."

Why it's wrong: Softmax couples all classes. The gradient of $p_c = \frac{e^{S_c}}{\sum_k e^{S_k}}$ w.r.t. input involves all logits. If another class score increases, $p_c$ decreases even if $S_c$ is unchanged. You get noisy, confusing saliency.

Fix: Use raw logits: loss = logits[class_idx]. This isolates the class score's sensitivity.

Common mistake Forgetting ReLU in Grad-CAM

Wrong:

cam = (alpha * feature_maps).sum(axis=0)  # No ReLU

Why it feels right: "Negative weights might cancel positives, showing the full picture."

Why it's wrong: Negative regions suppress the class score (e.g., "background decreases dog confidence"). For localization, we only care where the model found positive evidence for the class. Fix:

cam = relu((alpha * feature_maps).sum(axis=0))

This zeros out negative contributions, keeping only class-specific positive regions.

Recall Feynman: Explain to a 12-Year-Old

Imagine you trained a robot to recognize cats in photos. You show it a picture, and it says "90% sure that's a cat!" But how did it decide?

Saliency maps: We ask the robot, "If I change this one pixel a tiny bit, how much does your'cat confidence' change?" If changing a pixel in the cat's ear changes the answer a lot, that pixel is important. We color important pixels bright red and unimportant ones dark blue. Now we see the robot was looking at the ears and whiskers!

Grad-CAM: Instead of asking about every single pixel (which gets mesy), we ask the robot's "brain layers" deeper inside. The last layer before the final answer has mini-images showing "I see fur here, pointy ears there." We ask, "Which of these mini-images made you think 'cat'?" and color those regions. Then we blow up that colored map and put it over the original photo. Now we see the robot focused on the cat's face, not the couch behind it.

Why this matters: If the robot only looks at the background (like always seeing cats on blue carpets), it'll fail on a cat on a red carpet. These maps help us catch the robot "cheating."

Connections

Backpropagation: Saliency/Grad-CAM use gradients from standard backprop, just targeting input/features instead of weights.
Convolutional Neural Networks: Grad-CAM leverages spatial structure of conv layers; wouldn't work for dense layers.
Model debugging: Saliency/Grad-CAM reveal spurious correlations (Clever Hans effects).
Attention mechanisms: Attention weights show "what the model attends to" at each step; Grad-CAM shows spatial attention in CNs.
Adversarial examples: High-saliency regions are vulnerable to adversarial perturbations.
LIME and SHAP: Model-agnostic alternatives to gradient-based saliency; Grad-CAM is gradient-specific but faster.

#flashcards/ai-ml

What is a saliency map? :: A visualization showing the magnitude of the gradient of the output w.r.t. each input pixel: $|\frac{\partial S_c}{\partial I_{x,y}}|$ . High values indicate pixels that strongly influence the prediction.

Why do we use raw logits instead of softmax probabilities for saliency?

Softmax couples all classes—changing one class score affects all probabilities. Raw logits isolate the target class's sensitivity to input changes, giving cleaner gradients.

What does Grad-CAM stand for?

Gradient-weighted Class Activation Mapping. It uses gradients flowing into the last convolutional layer to weight feature maps, producing a class-specific localization heatmap.

Write the Grad-CAM formula :: $L_{\text{Grad-CAM}}^c = \text{ReLU}\left( \sum_k \alpha_k^c A^k \right)$ , where $\alpha_k^c = \frac{1}{Z} \sum_{i,j} \frac{\partial S^c}{\partial A_{i,j}^k}$ (global average pooled gradients) and $A^k$ is the $k$ -th feature map.

Why does Grad-CAM apply ReLU to the final heatmap?

ReLU zeros out negative values, keeping only regions that positively contribute to the target class score. Negative regions suppress the class (less interpretable for localization).

What is the key difference between saliency maps and Grad-CAM? :: Saliency maps are pixel-level gradients (noisy, high-res). Grad-CAM uses last conv layer feature maps (coarse, semantic, class-discriminative).

In Grad-CAM, why do we use the last convolutional layer?

Deep conv layers capture high-level spatial features (object parts, semantic regions) while retaining spatial structure. Earlier layers are too low-level; later dense layers lose spatial info.

How do you compute

\alpha_k^c

in Grad-CAM?

Global average pooling of gradients:

\alpha_k^c = \frac{1}{u \cdot v} \sum_{i,j} \frac{\partial S^c}{\partial A_{i,j}^k}

. This gives one importance weight per feature map.

What does a Grad-CAM heatmap tell you about a misclassification?

It shows which spatial regions the model used for the (wrong) class. If it highlights irrelevant features (background, watermarks), the model learned spurious correlations.

Why is Grad-CAM class-discriminative?

The weights

\alpha_k^c

depend on the gradient w.r.t. class

c

's score, so different classes produce different heatmaps showing where the model found evidence for each class.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Saliency maps aur Grad-CAM: Neural network ke andar jhank ke dekhna

Jab ap ek neural network train karte ho (jaise cat-dog classifier), toh wo ek black box ban jata hai. Input daal diya, output aa gaya – par kaise decide kiya, wo pata nahi. Interpretability ka matlab hai kium model ke "dimag" mein jhank sakein aur dekh sakein ki usne kis jagah focus kiya.

Saliency maps ka concept simple hai: hum model se pochhte hain, "Agar main input image kaek pixel thoda sa badal doon, toh teri prediction kitna change hogi?" Jis pixel ko badalne se output bahut change hota hai, wo pixel important hai. Mathematically, yeh gradient hai – $\frac{\partial S_c}{\partial I_{x,y}}$ (class score ka input pixel ke sath derivative). High gradient wale pixels ko bright color mein dikhate hain (jaise red ya yellow), low gradient wale ko dark (blue ya black). Result: ek heatmap jo bata hai ki model ne kahaan-kahaan dhyan diya.

Grad-CAM thoda alag approach leta hai. Pixel-level gradients bahut noisy hote hain (chaos lagta hai). Toh Grad-CAM last convolutional layer ki feature maps use karta hai. Deep CNN mein, last conv layer mein spatial information hoti hai (jaise "yahan pe kaan hain, wahan pe tail hai"), lekin pixels sezyada semantic (meaningful) hoti hai. Grad-CAM compute karta hai ki har feature map ka kitna contribution hai class score mein – yeh hum gradients se nikaalte hain ( $\alpha_k^c = \frac{1}{Z} \sum_{i,j} \frac{\partial S^c}{\partial A_{i,j}^k}$ ). Phir un weights se feature maps ko multiply karke ek heatmap banate hain, aur use original image ke upar overlay kar dete hain. Result: ek smooth, colored region jo dikhata hai ki "model ne yahan dog dekha" ya "yahan cat ka face dekha."

Kyun zaroori hai yeh? Agar apka model galat reasons seahi answer de raha hai (jaise "main hamesha blue background dekhta hoon jab cat hoti hai"), toh production mein fail ho jayega. Grad-CAM aur saliency maps se ap debug kar sakte ho, bias pakad sakte ho (model watermark dekh raha hai ya actual object?), aur medical/legal fields mein explainability provide kar sakte ho (regulatory requirement). Basically, yeh tools trust build karte hain – user ko pata chalta hai ki model randomness nahi, real patterns dekh raha hai.

What Are Saliency Maps?

Computing Saliency: Step-by-Step

Grad-CAM: Class Activation Mapping with Gradients

Saliency vs. Grad-CAM: Key Differences

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Interpretability & Explainability

Connections