Scaled dot-product attention: Visual walkthrough

This page rebuilds the parent result Scaled Dot-Product Attention from absolute zero, one picture at a time. We start with nothing but arrows on a page, and end holding the full formula

$Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$

Every symbol here is earned before it is used. If you have never seen a dot product, a softmax, or a matrix, start reading at line one — you lose nothing.

Step 1 — A word is an arrow

WHAT. Before any attention, we must agree what a "word" even is to the machine. Each word (or token) is turned into a list of numbers — a vector. Draw that list as an arrow from the origin: the direction the arrow points is the meaning.

WHY. We need meaning to live somewhere we can do arithmetic. Arrows can be added, stretched, and — crucially for Step 2 — compared by direction. Plain words can't.

PICTURE. Two arrows below. The amber arrow $q$ is a query — "the thing currently looking for information". The cyan arrows $k_{1}, k_{2}$ are keys — "things on the shelf that might answer". Notice $q$ leans toward $k_{1}$ and away from $k_{2}$ .

Step 2 — "How well do you match?" is a dot product

WHAT. We need a single number that says how aligned the query arrow is with a key arrow. That number is the dot product:

$q \cdot k = 1st coords multiplied q_{1} k_{1} + 2nd q_{2} k_{2} + \dots + last q_{d_{k}} k_{d_{k}}$

Read it term by term: pair up matching coordinates, multiply each pair, add all the products. One number falls out.

WHY this tool and not another? We could have measured distance between arrow tips instead. But we want alignment, not closeness, and we want it cheap and smooth (differentiable, so Gradient Descent can push on it). The dot product costs only $d_{k}$ multiplies and has a beautiful geometric meaning:

$q \cdot k = ∥ q ∥ ∥ k ∥ cos θ$

where $θ$ is the angle between the arrows and $∥ q ∥$ means the length of $q$ . So the dot product literally reads off the angle:

arrows point the same way ( $θ = 0°$ , $cos θ = 1$ ) → large positive number → strong match,
arrows are perpendicular ( $θ = 90°$ , $cos θ = 0$ ) → zero → unrelated,
arrows point opposite ( $θ = 180°$ , $cos θ = - 1$ ) → large negative number → anti-match.

PICTURE. The same arrows, now with the angle $θ$ marked and the three sign-cases spelled out. This is the "compatibility score" the parent note calls $S_{ij}$ .

Step 3 — Doing all comparisons at once: $Q K^{T}$

WHAT. Real inputs have many queries and many keys. Stack the query arrows as rows of a grid $Q$ (that grid is a matrix), and the key arrows as rows of $K$ . We want every query dotted with every key. That whole table of dot products is written

$S = Q K^{T}, S_{ij} = q_{i} \cdot k_{j}$

Reading the symbols. $Q$ has shape $n \times d_{k}$ ( $n$ queries, each of length $d_{k}$ ). $K$ has shape $m \times d_{k}$ . The little $T$ in $K^{T}$ means transpose — flip the key grid on its side so its columns become the arrows, making the multiplication line up. The result $S$ has shape $n \times m$ : one score per (query, key) pair.

WHY. Two reasons. (1) A single formula covers the whole sentence. (2) GPUs are built to multiply matrices blindingly fast, so computing thousands of dot products at once costs almost the same as one.

PICTURE. A grid: rows are queries, columns are keys, each cell is a match-score. Bright amber cells = strong match, dark = weak/negative. This grid is the raw attention before any softening.

Step 4 — The size problem, and why we divide by $d_{k}$

WHAT. Here is a fact that surprises everyone. If the query and key numbers are ordinary random values (average $0$ , spread $1$ ), then a dot product over $d_{k}$ dimensions has:

$E [q \cdot k] = 0, Var (q \cdot k) = d_{k}$

The average stays $0$ , but the spread grows with $d_{k}$ . Since spread (standard deviation) is the square root of variance, the typical size of a score is about $d_{k}$ . For $d_{k} = 64$ that is $\approx 8$ — scores routinely swing between $- 8$ and $+ 8$ .

WHY this is dangerous. Step 5 feeds these scores into softmax, which involves $e^{score}$ . A gap of $8$ versus $7.5$ becomes $e^{8}$ versus $e^{7.5}$ — one term utterly dominates, softmax collapses to "100% on one key, 0% on the rest". Once it's that lopsided, the gradient (the slope Gradient Descent climbs) is almost flat, so learning stalls. We are stuck.

THE FIX. Shrink the scores back to spread $1$ by dividing by $d_{k}$ :

$Var (\frac{q \cdot k}{d _{k}}) = \frac{d _{k}}{d _{k}} = 1$

Why $d_{k}$ and not $d_{k}$ ? Because we are fixing the spread, and spread scales as $d_{k}$ , not $d_{k}$ . Dividing by $d_{k}$ would over-correct and squash everything to zero.

PICTURE. Two bell curves: the wide one is the unscaled score distribution (std $d_{k}$ ), the narrow one is after dividing (std $1$ ). Shaded band shows softmax's healthy "sensitive zone" roughly $[- 3, 3]$ — only the scaled curve sits inside it.

Step 5 — Turning scores into percentages: softmax

WHAT. Take one row of scaled scores, say $z = (z_{1}, z_{2}, \dots, z_{m})$ , and convert it into positive numbers that add up to $1$ — a set of percentages. The Softmax Function does exactly this:

$A_{ij} = \frac{e ^{z_{j}}}{j ^{'} = 1 \sum m e ^{z_{j^{'}}}}$

Term by term: $e^{z_{j}}$ makes every score positive (even negatives become small positives), and dividing by the sum of all exponentials makes the whole row add to $1$ .

WHY this tool. We want three things at once: outputs that are positive, that sum to $1$ (so they read as "how much attention"), and that are smooth so gradients flow. The exponential also creates competition — pushing one key up automatically pushes the others down, because they share one denominator.

PICTURE. Left: a row of raw scores (some negative). Right: the same row after softmax — all bars positive, tallest bar where the score was highest, and every bar's height is now a percentage summing to 100%.

Step 6 — Blending the answers: multiply by $V$

WHAT. Behind each key sits a value $v_{j}$ — the actual content to hand back. The output for query $i$ is the attention-weighted blend of all values:

$output_{i} = \sum_{j = 1}^{m} A_{ij} v_{j}$

Each value $v_{j}$ is scaled by its attention weight $A_{ij}$ and the results are summed. In matrix form the whole sentence at once:

$Attention (Q, K, V) = A V \in R^{n \times d_{v}}$

WHY. The query asked, the keys matched, and now the values answer. Because the weights are percentages summing to $1$ , the output is a genuine weighted average — it lands inside the cloud of value arrows, leaning hardest toward whichever value the query attended to most.

PICTURE. Value arrows $v_{1}, v_{2}$ in cyan, the output arrow in amber sitting between them but closer to $v_{1}$ (because $A$ favoured key 1). A slider-bar shows the $73%/27%$ blend from the worked example.

Step 7 — Watch it run on real numbers

Let's push the parent note's Example 1 through every step so no move stays abstract. Take $d_{k} = 4$ , one query, two keys, two values:

$q = (1, 0, 1, 0), k_{1} = (1, 0, 1, 0), k_{2} = (0, 1, 0, 1), v_{1} = (5, 0, 0), v_{2} = (0, 3, 0)$

Dot products (Step 2/3): $q \cdot k_{1} = 1 + 0 + 1 + 0 = 2$ , $q \cdot k_{2} = 0$ . So $S = [2, 0]$ — the query is $k_{1}$ , hence perfect match; it's perpendicular to $k_{2}$ .

Scale (Step 4): $d_{k} = 4 = 2$ , so scaled scores $= [1, 0]$ .

Softmax (Step 5): $A = [\frac{e ^{1}}{e ^{1} + e ^{0}}, \frac{e ^{0}}{e ^{1} + e ^{0}}] \approx [0.731, 0.269]$ .

Blend (Step 6): $output = 0.731 (5, 0, 0) + 0.269 (0, 3, 0) = (3.655, 0.807, 0)$

PICTURE. The full pipeline in one strip: arrows → score grid → scaled → softmax bars → output arrow, with these exact numbers annotated at each arrow.

The one-picture summary

Everything on this page compressed: arrows become a score grid ( $Q K^{T}$ ), the grid is shrunk ( $\div d_{k}$ ) into softmax's sweet spot, softmax makes each row a set of percentages ( $A$ ), and those percentages blend the values ( $A V$ ) into the output.

Recall Feynman retelling (say it out loud, no symbols)

I turn every word into an arrow. To ask "who's relevant to me?", I measure how much my arrow points the same way as each other arrow — that alignment number is big when they agree, zero when they're unrelated, negative when they clash. When there are lots of dimensions, those alignment numbers naturally grow huge, so I shrink them by the square root of the dimension count to keep them tame. Then I squash the row of numbers into percentages that add to 100% — big alignment gets a big slice, negatives get almost nothing. Finally I mix the answer-vectors in exactly those percentages. The word I agreed with most contributes most. That mixed vector is my answer. Nothing more.

Recall

Why divide by $d_{k}$ and not $d_{k}$ ? ::: The dot product's spread (standard deviation) grows like $d_{k}$ ; to reset spread to $1$ we divide by that spread, $d_{k}$ . Dividing by $d_{k}$ would over-shrink. What single job does softmax do here? ::: Turn a row of real-valued scores into positive percentages that sum to $1$ , smoothly and competitively. Where does a masked position go? ::: Its score is set to $- \infty$ , so $e^{- \infty} = 0$ and it receives exactly $0%$ attention. In Example 1, what is the output vector? ::: $(3.655, 0.807, 0)$ .

See also: Self-Attention · Cross-Attention · Multi-Head Attention · Positional Encoding