Attention visualization and limitations: Visual walkthrough

This page rebuilds attention from absolute zero and then shows, picture by picture, why the pretty heatmaps you see can mislead you. We start with nothing but a bag of word-vectors and end at the warning "attention is not explanation." Every symbol is earned before it is used.

This is the visual companion to the parent topic. If you want the raw formula reference, that lives there; here we draw each step.

Step 1 — A sentence is a row of arrows

WHAT. Before any maths, picture the input. Take the sentence "the cat sat". Each word is turned into a list of numbers — call that list a vector, which we draw as an arrow. Three words → three arrows.

WHY. Everything a transformer does is mixing arrows. If you can see the arrows, you can see the mixing. We refuse to write a single Greek letter until you can point at what it stands for on a picture.

PICTURE. Three arrows, one per word, sitting in a plane.

Figure — Attention visualization and limitations

Nothing has been mixed yet. Each arrow only knows about its own word.

Step 2 — "How aligned are two arrows?" — the dot product

WHAT. To decide how much word $j$ should pour into word $i$ , we first ask: how similar are their arrows? The tool that answers "how aligned are two arrows?" is the dot product.

WHY this tool and not another? We need a single number that is big when two arrows point the same way and small (or negative) when they point apart. The dot product does exactly that: $a \cdot b = ∥ a ∥ ∥ b ∥ cos θ$ . The $cos θ$ is the dial — $cos 0^{\circ} = 1$ (same direction, maximum), $cos 9 0^{\circ} = 0$ (unrelated), $cos 18 0^{\circ} = - 1$ (opposite). No other simple operation packages "alignment" into one number so cleanly. See Attention Mechanisms for where this plugs into the full block.

PICTURE. Two arrows and the angle $θ$ between them; the score rises as they close.

Step 3 — Shrink the score so it doesn't blow up: divide by $d_{k}$

WHAT. We replace $s_{ij} = q_{i} \cdot k_{j}$ with $\frac{q _{i} \cdot k _{j}}{d _{k}}$ , where $d_{k}$ is the length of the key vector (how many numbers are in the arrow).

WHY. A dot product adds up $d_{k}$ little products. If each arrow has more numbers, the sum is bigger just because there are more terms, not because the words are more related. Random arrows of length $d_{k}$ give a dot product that typically grows like $d_{k}$ . Dividing by $d_{k}$ cancels that growth, so scores stay in a sane range no matter how big the model. Big raw scores would later make softmax (next step) saturate into a spike, killing learning.

PICTURE. A bar of the raw score growing with $d_{k}$ , and the same bar flattened after dividing.

WHAT. For a fixed word $i$ we have a row of scores $e_{i 1}, e_{i 2}, \dots$ — one per word $j$ . We convert this row into positive numbers that sum to 1 using softmax.

WHY this tool? We want the weights to behave like slices of a pie: never negative, always adding to 100%, so we can read them as "fraction of information." Softmax is the standard way to do that while keeping the ordering (bigger score → bigger slice) and staying smooth (differentiable, so gradients can train it). The $exp$ makes everything positive; dividing by the total makes it sum to 1.

PICTURE. A row of raw scores (some negative) becoming a tidy pie chart of slices.

Step 5 — Pour the arrows together: the weighted average

WHAT. The new arrow for word $i$ is the sum of every word's value arrow $v_{j}$ , each scaled by its share $α_{ij}$ .

WHY. $α_{ij}$ told us how much of each word to take; now we actually take it. Because the shares sum to 1, the result is a genuine weighted average — a blend that lives among the value arrows, never exploding.

PICTURE. Three faint value arrows scaled by their weights, added tip-to-tail into one bold output arrow.

Stacking all rows $i$ into a matrix gives the compact form you saw in the parent: $Z = A, the heatmap softmax (\frac{Q K ^{⊤}}{d _{k}}) V .$ The matrix $A$ (all the $α_{ij}$ ) is exactly what a heatmap draws: row = query word $i$ , column = key word $j$ , brightness = $α_{ij}$ .

Step 6 — Stacking layers: attention rollout

WHAT. One layer's heatmap $A^{(1)}$ moves information one hop. Real models stack $L$ layers. To see word $j$ 's influence on word $i$ across the whole stack, multiply the heatmaps: $A^{(L)} \dots A^{(1)}$ .

WHY. If layer 1 pours $j$ into $k$ (weight $A_{k j}^{(1)}$ ) and layer 2 pours $k$ into $i$ (weight $A_{ik}^{(2)}$ ), then $j$ reaches $i$ through $k$ with strength $A_{ik}^{(2)} A_{k j}^{(1)}$ . Matrix multiplication sums this over every intermediate word $k$ automatically — that is precisely what the entry $(i, j)$ of the product means. We also add the identity $I$ first because the residual connection lets each word keep its own old arrow (a "do nothing" path).

PICTURE. Two stacked heatmaps and the arrows showing $j \to k \to i$ combining into one long-range link.

Step 7 — The trap: a bright cell is not a used cell

WHAT. Here is the whole reason this page exists. A large $α_{ij}$ says word $i$ poured in a lot of value $v_{j}$ . It does not say that pouring mattered to the final answer.

WHY it fails. Two independent things can kill the influence of a high-attention token:

The value arrow may be tiny or point into a dead direction. After attention, $v_{j}$ passes through LayerNorm, more residual adds, and an MLP. If $v_{j}$ lands in the null space of the next layer (a direction that layer maps to zero), it contributes nothing — no matter how bright $α_{ij}$ was.
Attention is not the only pipe. Residual and MLP paths carry information that the heatmap never shows.

So the honest "importance" of $j$ needs the size of the value and the downstream sensitivity, not $α$ alone: $Contribution_{ij} \approx α_{ij} ∥ v_{j} ∥ \cdot (downstream sensitivity) .$

Attention shows correlation (where the model looked); the gradient $\partial L / \partial x_{j}$ shows causation (would changing $x_{j}$ change the answer). These can flatly disagree.

PICTURE. Same bright cell, two fates: value arrow survives → real influence; value arrow squashed by downstream → zero influence.

The one-picture summary

The single diagram: arrows → dot-product score → scale by $d_{k}$ → softmax into shares → weighted-average of values → stacked into rollout → and the red gate reminding you that bright ≠ important.

Recall Feynman retelling — say it like you'd explain to a friend

Every word starts as an arrow. To update a word, the model asks each other word "how well do you match what I'm looking for?" — that match is a dot product. It shrinks that number by $d_{k}$ so bigger models don't cheat, runs the whole row through softmax to get a pie of shares that add to one, then blends the words' value arrows using those shares. Stack this many times and multiply the heatmaps (with a +1 on the diagonal for the residual "keep yourself" path) to see long-range flow. The picture you get — the heatmap — is honest about where the model looked, but it lies if you read it as what mattered, because a looked-at word's value can be crushed by later layers, and other pipes (residual, MLP) never appear on the map. To know what truly mattered, poke the input and watch the loss — that's the gradient, and that's causation.

Recall Quick self-test

Why divide by $d_{k}$ ? ::: Because a dot product of length- $d_{k}$ arrows grows like $d_{k}$ ; dividing keeps scores stable so softmax doesn't saturate. What does $α_{ij} = 1/ n$ for all $j$ mean? ::: Uniform / dispersed attention — the model is spreading its focus evenly (no strong preference). Give one reason a high-attention token can still be unimportant. ::: Its value vector $v_{j}$ can be tiny or fall in the downstream layer's null space, so it contributes ~0 despite the bright weight. Attention shows ____, gradients show ____. ::: correlation ; causation