Attention visualization and limitations: Worked examples

This is a deep-dive companion to the parent topic. There we argued why attention is a tricky explanation tool. Here we compute — we take every kind of situation the topic can throw at you and grind through the numbers, so that when you meet a real heatmap you already know what each cell of the matrix means.

Before line one, three plain-word reminders so no symbol arrives un-earned:

We will need the softmax repeatedly. In plain words: softmax turns a list of raw scores into a list of positive fractions that sum to 1, and it does so by exponentiating (to force positivity) then dividing by the total (to force the sum-to-one).

We also meet one more symbol repeatedly, so let us earn it now:

The scenario matrix

Every situation this topic produces falls into one of these cells. The examples below are labelled with the cell(s) they hit, and together they fill the whole grid.

Cell	What it stresses	Example
C1 — Basic single-head	one $A$ matrix, read a heatmap	Ex 1
C2 — Softmax scaling / $d_{k}$	why divide scores, what changing $d_{k}$ does	Ex 2
C3 — Degenerate: uniform attention	all weights equal → says nothing	Ex 3
C4 — Degenerate: value in null space	high weight, zero contribution	Ex 4
C5 — Multi-head averaging trap	antagonistic heads cancel	Ex 5
C6 — Attention rollout (limiting/deep)	multiply layers, add identity	Ex 6
C7 — Attention vs gradient disagree	high attention, low gradient & vice-versa	Ex 7
C8 — Adversarial rewrite	two different $A$ , same output	Ex 8
C9 — Real-world word problem	sentiment sentence end-to-end	Ex 9

Forecast: Guess — will "not" (score 2.0) get roughly twice the weight of "good" (score 1.0)? Softmax is exponential, so the gap is bigger than "twice". Hold that thought.

Exponentiate each score. $e^{2} = 7.389$ , $e^{1} = 2.718$ , $e^{0} = 1.000$ . Why this step? Softmax forces positivity through $e^{s}$ ; large scores blow up faster than small ones — the exponential is what makes attention peaky.
Sum them. $7.389 + 2.718 + 1.000 = 11.107$ . Why this step? This total is the denominator that turns scores into fractions summing to 1.
Divide. $A = [0.665, 0.245, 0.090]$ . Why this step? Now each entry reads directly as "fraction of information from that token."

The figure below plots these three weights as bars, with each token's raw score printed underneath it. The point it makes visually is the gap-exaggeration: the "not" bar sits at $0.665$ while "good" sits at $0.245$ — a ratio of about 2.7×, even though the raw scores ( $2.0$ vs $1.0$ ) differ by only 2×. The extra stretch is the exponential at work; reading the raw scores alone would badly under-estimate how sharply the model concentrates on "not".

Figure — Attention visualization and limitations

Figure 1 — Ex1 softmax bars. Three coloured bars (coral "not" $\approx 0.665$ , lavender "good" $\approx 0.245$ , mint "." $\approx 0.090$ ); the raw score is printed under each bar and a butter callout arrow marks how the exponential stretches the not-vs-good gap from 2× to 2.7×.

Verify: $0.665 + 0.245 + 0.090 = 1.000$ ✓ (a valid distribution — every row of an attention matrix must do this).

Forecast: Guess whether the unscaled version is more or less extreme. Bigger score gaps → more extreme softmax, so unscaled should nearly one-hot.

Unscaled softmax of $[64, 32]$ . The gap is $32$ ; $e^{32}$ dwarfs $e^{0}$ , so weight on the first token $\approx \frac{1}{1 + e ^{- 32}} \approx 1.0000$ . Why this step? Shows the danger: in high dimensions raw dot products are huge, and softmax collapses to a single spike — the gradient through it dies.
Scaled softmax of $[8, 4]$ ( $= [64, 32] / 64$ , and $64 = 8 = d_{k}$ ). Gap is $4$ : weight $= \frac{e ^{8}}{e ^{8} + e ^{4}} = \frac{1}{1 + e ^{- 4}} = 0.9820$ . Why this step? Dividing by $d_{k}$ keeps score magnitudes $O (1)$ so softmax stays soft and trainable.
Read the lesson. Same pattern (first > second), but scaling leaves a usable gradient ( $0.982$ , not $1.0000$ ). Why this step? $d_{k}$ is not cosmetic — it is why the model can learn.

Verify: $\frac{1}{1 + e ^{- 4}} = 0.98201$ (rounds to $0.982$ ) ✓.

Forecast: Guess: does uniform attention prove equal importance, or prove nothing?

Compute the entropy (spread) of this distribution. With $n = 4$ equal weights, entropy $= lo g_{2} 4 = 2$ bits — the maximum possible. Why this step? Maximum entropy = the head is not selecting anything; it is the "I have no preference" output.
Interpret. A maximally-uniform head has thrown away all discriminative signal. It could equally be (a) a genuinely useless head or (b) a head whose real work happens in the value vectors, not the weights. Why this step? Uniform weights are a degenerate input to any interpretation — the weights alone can't tell (a) from (b). You must inspect $v_{j}$ or gradients.
Conclusion. Never read "importance" off a flat heatmap. Flatness = information absent, not "everything matters."

Verify: For 4 equal weights, entropy $- \sum 0.25 lo g_{2} 0.25 = 2.0$ bits, and this equals $lo g_{2} 4$ ✓.

Forecast: Weight is $0.9$ — surely token $j$ dominates the output?

The contribution the network sees is (weight) × (projection of the value onto what downstream reads): $α_{ij} (w \cdot v_{j})$ . Why this step? A weight scales the value, but the next layer only "feels" the part of the value along its own reading direction $w$ .
Compute the projection. $w \cdot v_{j} = (1) (2) + (1) (- 2) = 0$ . Why this step? $v_{j}$ is orthogonal to $w$ — it lies in the null space of the downstream read. A dot product of zero means "invisible."
Multiply. Contribution $= 0.9 \times 0 = 0$ . Why this step? Despite the enormous $0.9$ weight, token $j$ contributes nothing to the output. This is the parent note's warning made concrete.

Verify: $(1) (2) + (1) (- 2) = 0$ , so $0.9 \times 0 = 0$ ✓. High attention ⇏ high influence.

Forecast: Will averaging reveal the signal, or erase it?

Average element-wise. $\overset{ˉ}{A} = [\frac{0.9 + 0.1}{2}, \frac{0.1 + 0.9}{2}] = [0.5, 0.5]$ . Why this step? This is the naive "smooth out the noise" move the parent note warns against.
Read the result. The average is perfectly uniform — exactly the degenerate Cell C3! The strong, opposite specializations cancelled. Why this step? Antagonistic heads produce a flat mean that hides both the signal and the confounder.
The fix. Keep heads separate and use gradient attribution to find which head the classifier actually reads. Averaging destroyed the very structure we wanted.

Verify: $\frac{0.9 + 0.1}{2} = 0.5$ and $\frac{0.1 + 0.9}{2} = 0.5$ ✓ — a real signal averaged into meaningless uniformity.

Forecast: Layer 2 sends 0.5 of token-2's information back to token 1; layer 1 sends 0.2 of token-1 to token 2. Does the chain create influence token 1 → 2 that neither layer shows alone?

Why multiply? If layer 1 routes $j \to k$ and layer 2 routes $k \to i$ , then $j$ influences $i$ through $k$ . Matrix product $A^{(2)} A^{(1)}$ sums over every intermediate $k$ automatically: $(A^{(2)} A^{(1)})_{ij} = \sum_{k} A_{ik}^{(2)} A_{k j}^{(1)}$ . Why this step? Composition of two routings = one matrix multiply; that's the whole idea of rollout.
Compute row 2, column 1 of $A^{(2)} A^{(1)}$ . Row 2 of $A^{(2)}$ is $[0.5, 0.5, 0]$ ; column 1 of $A^{(1)}$ is $[0.8, 0, 0]^{T}$ . $0.5 \times 0.8 + 0.5 \times 0 + 0 \times 0 = 0.40$ Why this step? This traces the path: token 1 → (0.8, layer 1) → token 1's slot, then token 2 pulls 0.5 of it in layer 2.
Interpret the limit. The cumulative influence $0.40$ is larger than the single-layer number and reveals a path no one layer's heatmap displays. Why this step? This is exactly why rollout exists — deep flow ≠ local attention.

Caveat (never skip): rollout assumes attention is the only pathway. Residuals (the $+ I$ ) and MLPs also carry information, so rollout is an upper-bound sketch, not truth.

Verify: $(A^{(2)} A^{(1)})_{21} = 0.5 \cdot 0.8 + 0.5 \cdot 0 = 0.40$ ✓.

Forecast: Attention says "the"; gradient says "not". Who wins?

What attention says. "the" is looked at 12× more than "not" ( $0.6$ vs $0.05$ ). Why this step? Establishes the correlation story — where the model's eyes lingered.
What gradient says. Perturbing "not" moves the loss $45 \times$ more than perturbing "the" ( $0.9$ vs $0.02$ ). Why this step? Gradient is the causal test: it measures whether changing the input changes the outcome.
Resolve. "not" is the causal driver despite tiny attention (its influence is indirect, through other tokens); "the" was examined but unused. Trust the gradient for importance claims. Why this step? This is the parent note's slogan made numeric: attention shows correlation, gradients show causation.

Verify: Attention ranks "the" above "not" ( $0.6 > 0.05$ , ratio $12$ ); gradient ranks "not" above "the" ( $0.9 > 0.02$ , ratio $45$ ). The two rankings are opposite ✓.

Forecast: If two value vectors are identical, does the split between them matter?

Compute the original output. $0.5 v_{1} + 0.4 v_{2} + 0.1 v_{3} = (0.5 + 0.4, 0.1) = (0.9, 0.1)$ . Why this step? Establishes the target output we must reproduce.
Try a wildly different $A^{'} = [0.1, 0.8, 0.1]$ . Because $v_{1} = v_{2}$ , only their sum of weights matters: $0.5 + 0.4 = 0.9 = 0.1 + 0.8$ . $0.1 v_{1} + 0.8 v_{2} + 0.1 v_{3} = (0.9, 0.1)$ Why this step? Identical (or collinear) value vectors make the attention split non-identifiable — many weightings yield one output.
Conclusion. The "explanation" heatmap flipped from favouring $v_{1}$ (weight $0.5$ ) to favouring $v_{2}$ (weight $0.8$ ), yet the model's output vector is byte-for-byte unchanged at $(0.9, 0.1)$ . Any observer reading the heatmap would tell two contradictory stories about "what the model focused on," while the network behaves identically. This is the Jain & Wallace adversarial-attention phenomenon in miniature: because attention weights are not uniquely determined by the output, they cannot be the explanation. Why this step? It closes the promised adversarial scenario — different heatmap, same prediction — and points to Adversarial Attention for the full construction on real models.

Verify: $(0.5 + 0.4, 0.1) = (0.9, 0.1)$ and $(0.1 + 0.8, 0.1) = (0.9, 0.1)$ — identical outputs from different attention ✓.

Forecast: Will "not" get the largest weight, and is that enough to justify calling it the explanation?

Scale by $d_{k} = 4 = 2$ . Scaled scores $= [0.25, 0.0, 1.25, 1.0]$ . Why this step? Same $d_{k}$ discipline as Example 2 — divide by $d_{k}$ to keep magnitudes tame before softmax.
Exponentiate and sum. $e^{0.25} = 1.284$ , $e^{0} = 1.000$ , $e^{1.25} = 3.490$ , $e^{1.0} = 2.718$ ; sum $= 8.492$ . Why this step? Standard softmax machinery — positivity via $e^{s}$ , then a denominator to force the sum to 1.
Divide to get the weights. $A = [0.151, 0.118, 0.411, 0.320]$ Why this step? Now we can read the heatmap: "not" ( $0.411$ ) and "great" ( $0.320$ ) dominate — a plausible story that the model looked at the negation and the sentiment word.
The honest caveat. Plausible ≠ proven. To claim "not" caused the negative label, ablate it: rerun with "not" removed and check the label flips, or measure $\partial L / \partial x_{not}$ (cf. Integrated Gradients / SHAP for Deep Learning). As Example 8 showed, a different heatmap could give the same output — so the heatmap is a hypothesis, not a verdict. Why this step? Closes the loop back to the whole page's thesis.

The figure below shows the four weights as bars. Notice "not" and "great" together carry $0.411 + 0.320 = 0.731$ — nearly three-quarters of all the attention mass — while "movie" and "was" split the rest. The visual makes the dominance obvious and reminds us (red note) that dominance in the heatmap is not the same as causal proof.

Figure 2 — Ex9 sentiment attention. Four bars for "movie", "was", "not", "great"; coral "not" ( $\approx 0.411$ ) and butter "great" ( $\approx 0.320$ ) tower over mint "movie" and lavender "was", and a coral caption warns that heatmap dominance must still be confirmed by ablation or gradient.

Verify: $0.151 + 0.118 + 0.411 + 0.320 = 1.000$ ✓, and "not" is the arg-max weight ✓.

Which tool answers which question?

Recall Self-check

Uniform attention over 4 tokens means all four are equally important. ::: False — it means the head expresses no preference; importance is undetermined (Cell C3). A token with attention weight 0.9 must dominate the output. ::: False — if its value vector is orthogonal to what downstream reads, its contribution is 0 (Cell C4). What does the symbol $d_{k}$ stand for? ::: The dimension (number of entries) of each query/key vector; dot-product scores grow with it, which is why we divide by $d_{k}$ . Why divide raw scores by $d_{k}$ before softmax? ::: Raw dot products grow with dimension; without scaling softmax saturates to near one-hot and the gradient dies. What does entry $(i, j)$ of $A^{(2)} A^{(1)}$ represent? ::: Cumulative (rolled-up) information flow from token $j$ to token $i$ summed over all intermediate tokens. Averaging two antagonistic heads $[0.9, 0.1]$ and $[0.1, 0.9]$ gives? ::: $[0.5, 0.5]$ — flat, hiding both the signal and confounder. Attention shows correlation; gradients show causation.

Related tools that measure causal importance where attention cannot: Integrated Gradients, SHAP for Deep Learning, Layer-wise Relevance Propagation, Probing Classifiers. To inspect the raw heatmaps: BertViz Tool. Foundations: Attention Mechanisms, Transformer Architecture.