Noesis
6.3.4 · D5Interpretability & Explainability

Question bank — Attention visualization and limitations

1,456 words7 min read↑ Back to topic

This is a rapid-fire self-test for the parent topic. Each line hides an answer after ::: — read the prompt, commit to an answer out loud, then reveal. The answers are short arguments, not verdicts, so if your reasoning differs from mine, that gap is exactly what to study next.

True or false — justify

A high attention weight proves token caused the prediction.
False. measures forward-pass routing only; the value vector can be nullified by later LayerNorm/MLP layers, so "looked at" ≠ "used".
Attention weights always sum to 1 across the key positions for a fixed query.
True — that is exactly what the softmax guarantees, which is why they look like a probability distribution over tokens.
Because attention weights sum to 1, they are a genuine probability distribution over "importance".
False. They sum to 1 by construction, but summing to 1 is necessary, not sufficient, for being an importance measure — importance needs value magnitudes and downstream sensitivity too.
Two very different attention maps can never produce the same model output.
False. Jain & Wallace (2019) and Serrano & Smith (2019) built adversarial attention distributions that leave the output essentially unchanged — see Adversarial Attention.
Averaging all heads gives a "consensus" explanation that is safer than any single head.
False. Averaging blends redundant, specialized, and antagonistic heads together, diluting the one head that may carry the real signal.
A token with near-zero attention weight can still strongly influence the loss.
True. Influence can flow indirectly through other tokens, so the gradient can be large even when is tiny.
Attention rollout gives the complete picture of information flow through the model.
False. Rollout only tracks attention edges; it ignores that MLPs and the residual stream also move and transform information.
Gradients and attention weights measure the same thing from different directions.
False. Attention is forward-pass correlation (where the model looked); the gradient is backward-pass causation (what changes the loss). They routinely disagree.
The scaling factor changes which token is judged most important.
Mostly false — it rescales all scores uniformly before softmax to stop dot products exploding in high dimensions; it stabilizes training, not the ranking of a single query's keys (though softmax's temperature-like sharpness does shift).

Spot the error

"I visualized layer 6, head 3, saw 'not' get high attention, so head 3 is the negation detector that drives the sentiment output."
The leap from "attends to 'not'" to "drives the output" is unjustified. You must ablate the head or use gradient attribution to show the classification layer actually reads it.
"To combine per-layer maps I just added them: ."
Wrong operation. Flow through layers is transitive, so it needs matrix multiplication — the product's entry sums over intermediate tokens . Addition destroys the path structure.
"I multiplied the raw attention matrices for rollout and skipped the identity term."
Dropping ignores the residual connection: each layer also passes raw input through unchanged, so without you overstate how much attention (vs. the residual stream) does.
"I did rollout over 24 layers without normalizing between products and the numbers blew up."
Correct symptom, correct cause. Repeated multiplication grows values geometrically; you must row-normalize after each product to keep the probability interpretation.
"Head 10 has smooth dispersed attention, so it's doing careful whole-sentence reasoning."
Dispersed ≈ uniform ≈ uninformative. A flat map often means the head contributes little discriminative signal, the opposite of "careful reasoning".
"Contribution equals the attention weight, so ."
Incomplete. Real contribution scales as — a large times a value vector in the null space of later layers contributes nothing.

Why questions

Why is softmax used to normalize the scores rather than just dividing by their sum?
Raw dot products can be negative; dividing by their sum could give negative or blowing-up "weights". forces positivity, and dividing by the exponentials' sum guarantees a valid distribution.
Why is the dot product the chosen similarity, not (say) Euclidean distance?
The dot product measures directional alignment — large when vectors point the same way — which is exactly "does this key match what this query is asking for", and it's cheap as one matrix multiply.
Why does adding the identity matrix appear in rollout but not in the plain attention formula?
Plain attention describes one layer's routing; rollout models the whole residual stream, and the residual connection literally adds the unchanged input, which represents.
Why can't we trust a single beautiful heatmap as an explanation, even if it "makes sense"?
Because a plausible-looking map is a post-hoc story. Confirmation bias makes almost any pattern seem meaningful; only causal tests (ablation, gradients, adversarial attention) can validate it.
Why do gradient-based methods like Integrated Gradients or Layer-wise Relevance Propagation complement attention maps rather than replace curiosity about them?
Attention tells you the model's internal routing (mechanistic), gradients tell you input→loss sensitivity (functional). You often want both: the how and the whether-it-mattered.
Why does high attention plus a large value-vector norm still not guarantee importance?
Because the product must survive the downstream path — LayerNorm, residual adds, and MLPs can amplify or cancel it. Sensitivity of later layers is the missing third factor.
Why might Probing Classifiers give a different verdict than an attention heatmap about what a layer "knows"?
A probe tests whether information is linearly decodable from representations; a heatmap only shows routing. Information can be present yet unattended, or attended yet unused.

Edge cases

The query attends almost entirely to itself (). What does the visualization tell you?
Very little externally — the token is largely just carrying its own representation forward this layer, so cross-token influence at this layer is minimal.
Every entry in a row is (perfectly uniform attention). Is this "attending to everything"?
Effectively it attends to nothing in particular; a uniform average is the least informative case and usually signals the head isn't discriminating between tokens.
A causal (decoder) mask sets for . Does a blank upper triangle mean the model ignores future tokens?
No — it's architecturally forbidden from seeing them, not a learned choice. Reading meaning into the zeros there is a category error.
A special token like [CLS] or [SEP] soaks up huge attention across most heads. Does it hold the key information?
Often not. These act as attention sinks / no-op parking spots — heads dump excess probability mass there when no other token is relevant, so high weight there is frequently uninformative.
Two heads are exactly redundant (identical maps). Does deleting one hurt the model?
Usually no — redundant heads are wasteful but harmless, and ablation experiments often show such heads can be pruned with negligible effect.
A token has zero attention everywhere but changing it flips the prediction. How is that possible?
Its influence is indirect: it shaped another token's representation in an earlier layer, so the causal path exists through gradients even though the direct attention edge is zero.
You want to explore all of this interactively rather than by hand. Which vault tool fits?
BertViz Tool — it renders per-head, per-layer attention so you can hunt for these traps yourself, but remember it visualizes routing, so pair it with a causal method.
Recall One-sentence summary to lock in

Reveal ::: Attention visualization answers "where did the model look?" — never assume it also answers "what caused the answer?"; that second question always needs ablation, gradients, or adversarial tests.

Connections