Attention visualization and limitations: Foundations

Before you can read that story, you need every symbol in it. This page builds them from nothing — no dot products, no softmax, no matrices assumed. Read top to bottom; each block earns the next.

0. A note on notation — italic vectors, capital matrices

Keep this in your pocket: small italic = one arrow, capital = a stack of arrows. Every symbol below obeys it.

1. A vector — a list of numbers that is also an arrow

Figure — Attention visualization and limitations

Why the topic needs it. Every word (token) in a sentence is turned into a vector — a little arrow living in space. "Cat" is one arrow, "sat" is another. All of attention is arrows being compared and blended. The number $d_{k}$ (typically 64 in real models) will come back in §5.

2. The dot product — a number that measures "same direction and how big"

Before we go further we must earn one symbol we are about to write: the length of an arrow, written with vertical bars.

WHY this tool and not another? We want to ask "do these two arrows point the same way?" The dot product answers exactly that — but with an important twist below:

Arrows pointing the same way → large positive number.
Arrows perpendicular → dot product is zero.
Arrows pointing opposite ways → large negative number.

Why this matters. Because size sneaks into the score, two words can look "very related" just because their arrows are long, not because they truly align. This is one more reason attention scores get squashed by softmax (§4) into honest fractions rather than trusted as raw truth.

WHAT IT LOOKS LIKE. Look at the figure: the same arrow $q$ compared against three keys. The green $k$ (aligned) gives a big score, the gray $k$ (perpendicular) gives zero, the red $k$ (opposed) gives negative.

Why the topic needs it. The parent computes $q_{i} \cdot k_{j}$ — the query of word $i$ dotted with the key of word $j$ . That number is a raw similarity / relevance score: how much word $i$ "cares about" word $j$ .

3. Query, Key, Value — three roles every word plays

The picture: a library search.

Your query is your search request.
Each book's key is its title on the spine (used for matching).
Each book's value is the actual text inside (what you take away).

You match your query against every spine (key), and then read the insides (values) of the books in proportion to how well they matched.

Why the topic needs it. The parent's formula uses capital $Q, K, V$ . By our §0 convention those are just all the query vectors stacked into a grid (one row per word), all the keys stacked, all the values stacked. Once you see them as "stacks of the arrows above", the big equation stops being scary. See Attention Mechanisms for how these roles are learned.

4. Matrix multiplication — how one product makes every pairwise dot product

Before the headline formula we must earn one more symbol: $Q K^{T}$ . This is the promised "no matrices assumed" step.

WHAT IT LOOKS LIKE. Stack queries as horizontal rows on the left, keys as vertical columns on top. Slide row $i$ across column $j$ , multiply-and-add — you drop $q_{i} \cdot k_{j}$ into cell $(i, j)$ .

WHY one product does it all. Instead of computing $q_{i} \cdot k_{j}$ separately for every pair of words, the single operation $Q K^{T}$ produces the whole grid of pairwise similarity scores at once. Row $i$ of that grid is "how much word $i$ scored against every other word." That grid is what softmax then turns into attention weights.

5. Softmax — turning any scores into fractions that sum to 1

WHY exponential and not just "divide by the sum"? Two reasons, both visible in the figure:

Raw scores can be negative; plain division by a sum could give negative or blown-up "fractions". $e^{x}$ is always positive, so every output is a valid fraction.
$e^{x}$ exaggerates gaps — a slightly higher score gets a disproportionately bigger share. This makes the model able to "mostly focus on one word" when it wants to.

WHAT IT LOOKS LIKE. The top bars are raw scores (some negative). The bottom bars are after softmax: all positive, all between 0 and 1, and together they add up to exactly 1 — a probability distribution.

Why the topic needs it. This is the step that turns the similarity grid from §4 into the attention weights $α_{ij}$ . Because they sum to 1, we can honestly call them "the fraction of attention word $i$ gives to word $j$ " — the number every heatmap in the parent is coloring.

6. Scaling by $d_{k}$ — keeping scores from exploding

WHY divide? A dot product adds up $d_{k}$ separate products. The more terms you add, the bigger the total tends to be. How fast it grows depends on the vectors:

For fixed vectors whose entries are all a similar size, the sum of $d_{k}$ products grows roughly linearly in $d_{k}$ (twice as many equal terms ≈ twice the total).
For the case people usually quote — entries that are random and independent with mean $0$ — the typical size of the dot product grows only like $d_{k}$ , because random $+$ and $-$ terms partly cancel. This random-vector assumption is exactly why the divisor is $d_{k}$ and not $d_{k}$ .

Either way the point is the same: as $d_{k}$ grows, raw scores drift larger. If scores get huge, softmax's exponential (§5) collapses almost all weight onto a single word (over-sharpened, hard to train). Dividing by $d_{k}$ counteracts that drift under the standard random-initialization assumption, keeping scores in a sane range whatever the vector length.

The picture. Same directions, longer lists of numbers → naturally bigger dot products. The $\div d_{k}$ rescales them all back to a comparable size. (This is a normalization housekeeping step, not a modeling choice.)

7. Weighted average — the actual output of attention

The picture. Imagine mixing paint: if you use 70% blue value-vector and 30% orange value-vector, you get a blend that is mostly blue. The $α_{ij}$ are the mixing proportions; the $v_{j}$ are the paints.

Why the topic needs it — and its FIRST limitation. Notice the output uses the value vectors $v_{j}$ , not the keys. So even if word $i$ pays huge attention $α_{ij}$ to word $j$ , if that word's value $v_{j}$ is tiny (a small arrow) or later gets crushed by downstream layers, its real contribution is small. This is the seed of the parent's central warning: high attention ≠ high importance. Tools like Integrated Gradients and SHAP for Deep Learning exist precisely to measure real importance instead.

8. Putting the symbols together

Now the parent's headline formula reads in plain English:

$Attention (Q, K, V) = weights A softmax (\frac{Q K ^{T}}{d _{k}}) V$

$Q K^{T}$ — flip the keys with the transpose $^{T}$ (§4), then multiply so every query dots with every key → a grid of similarity scores.
$\div d_{k}$ — keep scores sane (§6).
$softmax$ — turn each row of scores into fractions that sum to 1 (§5).
$\times V$ — weighted-average the value vectors using those fractions (§7).

The grid $A$ , with entries $A_{ij}$ , is the thing every attention picture displays. Rows = the word doing the looking (query $i$ ); columns = the word being looked at (key $j$ ).

9. One more: the matrix and $A_{ij}$

The picture. A heatmap: rows down the side (queries), columns across the top (keys), each cell colored by its weight. Bright = big fraction, dark = near zero. This is literally the parent's 7×7 "cat sat on the mat" figure. Explore live examples with the BertViz Tool.

Prerequisite map

Everything on the left is built on this page; the arrows show how they feed the parent topic. See Transformer Architecture for where these attention blocks sit inside the full model.

Equipment checklist

Test yourself — reveal only after answering out loud.

What is a vector, in two words?

An ordered list of numbers (an arrow).

What is the notation rule for small italic vs capital letters?

Lowercase italic = one vector (one arrow); capital = a matrix (a stack of vectors, one per row).

What does

d_{k}

stand for?

The dimension — how many numbers are in each vector.

How is the magnitude

∣ q ∣

computed?

Square every entry, add them, take the square root:

q_{1}^{2} + q_{2}^{2} + \dots + q_{d_{k}}^{2}

(Pythagoras).

What does the dot product

q \cdot k

measure, fully?

∣ q ∣∣ k ∣ cos θ

— both the alignment (angle) and the lengths of the two arrows.

What does the transpose

^{T}

do, and why is

Q K^{T}

used?

It flips rows into columns;

Q K^{T}

then produces the whole grid of every pairwise dot product

q_{i} \cdot k_{j}

at once.

What two properties does softmax guarantee for its outputs?

Every output is between 0 and 1, and they all sum to 1.

Why do we subtract the max score before exponentiating in softmax?

To avoid numerical overflow of

e^{s_{j}}

; it gives the identical result because the shared factor cancels top and bottom.

Why do we divide scores by

d_{k}

, and under what assumption?

Under the random-initialized-vector assumption the typical dot-product size grows like

d_{k}

; dividing counteracts that drift so softmax is not over-sharpened.

What is an attention weight

α_{ij}

in words?

The fraction of attention word

i

pays to word

j

(a softmaxed similarity score).

Which vectors are actually blended to form the output — keys or values?

Values, weighted by the attention fractions.

Why is "high attention = high importance" a mistake?

The output uses value vectors and downstream layers; a highly-attended word can still contribute little, so attention shows where the model looked, not what it used.