Scaled dot-product attention: Foundations — AI-ML deep dive

The parent note opens with one dense line of notation. We will not write it yet — that would use symbols before they are earned. Instead we build every brick from absolute zero, and only at the very end, once each symbol has a picture and a meaning, do we assemble them into the full formula. Nothing is used before it is drawn.

1. A number list is a vector

The smallest brick is a vector: just an ordered list of numbers, like $[1, 0, 1, 0]$ . "Ordered" means position matters — the first slot and the third slot mean different things.

The picture: an arrow from the origin. A 2-number list $[q_{1}, q_{2}]$ is an arrow that goes $q_{1}$ steps right and $q_{2}$ steps up. Longer lists are arrows in more directions than we can draw, but the idea — a direction and a length — stays.

Why the topic needs it. Each word in a sentence is turned into such a list of numbers (its embedding). That is the raw material attention operates on. See Query-Key-Value Model for where these lists come from.

2. The length (norm) of a vector

Before we combine vectors, we need to say how long an arrow is — its size, ignoring direction.

Why the topic needs it. Length is one of the two things (length and direction) that decide how strongly two vectors align — we use it in the very next step to explain what a dot product really measures.

3. The subscript and summation notation

We keep writing $q_{1} + q_{2} + \dots$ ; there is a compact symbol for that.

Plain example: $\sum_{i = 1}^{3} q_{i} = q_{1} + q_{2} + q_{3}$ . That's the whole trick — it is shorthand for "write a plus sign between every slot and add."

Why the topic needs it. Every similarity score is a sum over the slots of two vectors. Without this shorthand the formulas would be pages long.

4. The dot product — measuring "do these point the same way?"

Now we combine two vectors. We want ONE number that says how aligned two arrows are. Both vectors must have the same number of slots; call that count $L$ (for Length of the list). Wherever a later section fixes this length to be the query/key length, it will be renamed $d_{k}$ — but for now it is just "however many slots the two lists share."

Why this tool and not another? We need a similarity score that is (a) a single number, (b) cheap to compute, and (c) smooth so we can later nudge it with Gradient Descent. The dot product is all three. And it has a beautiful geometric meaning that answers exactly our lookup question:

The dot product behaves like a similarity meter:

Arrows pointing the same way → large positive number.
Arrows at a right angle (perpendicular) → zero.
Arrows pointing opposite ways → large negative number.

5. Matrices — stacking many vectors into a grid

We rarely have one vector; we have a whole sentence of them. Stack them into a rectangle.

The picture: a spreadsheet where each row is one word's number-list.

Why the topic needs it. A real sentence is many word-vectors at once, and modern hardware (GPUs) is built to crunch whole grids in one shot rather than one arrow at a time. Bundling every query and every key into a matrix lets us compute all the similarity scores in a single multiplication — the batching the parent note relies on.

Now we can name the three matrices the topic revolves around. Each is defined in the Query-Key-Value Model:

$Q \in R^{n \times d_{k}}$ — the Query matrix: $n$ query vectors (the "questions"), each of length $d_{k}$ , stacked as rows.
$K \in R^{m \times d_{k}}$ — the Key matrix: $m$ key vectors (the "labels to match"), each also length $d_{k}$ .
$V \in R^{m \times d_{v}}$ — the Value matrix: $m$ value vectors (the "content to retrieve"), each length $d_{v}$ .

The letters mean: $n$ = how many questions, $m$ = how many stored items, $d_{k}$ = length of a query/key list (this is the shared slot-count $L$ from the dot product, now given its proper name), $d_{v}$ = length of a value list.

6. Transpose and matrix multiply — the score grid $Q K^{T}$

To score every query against every key in one stroke, we need two more pieces of notation.

Why flip $K$ ? We want the score between query $i$ and key $j$ to be their dot product. Rows of $Q$ are queries (length $d_{k}$ ); we need keys sitting as columns of length $d_{k}$ so a row-times-column dot product lands. Transposing $K$ ( $m \times d_{k} \to d_{k} \times m$ ) puts each key in a column.

7. Softmax — turning raw scores into a set of weights that add to 1

The scores $S_{ij}$ are arbitrary real numbers. We want them to become weights — non-negative and summing to 1 — so we can take a weighted average. That converter is softmax.

Why this tool? We need a smooth, differentiable way to go from "scores" to "a probability distribution." Exponentials guarantee positivity and exaggerate the winner (bigger score → much bigger weight), while the division normalizes. See Softmax Function for the full treatment.

Case check — all cases the reader might hit:

All scores equal → equal weights $1/ m$ each. (Perfect indecision.)
One score huge → weight $\approx 1$ on it, $\approx 0$ elsewhere (near one-hot).
A very negative score → weight near 0, never exactly 0. Nothing is ever fully ignored unless forced by an Attention Mask.
Negative scores are fine: $e^{- 3}$ is still positive, so softmax handles any sign.

This softness is precisely why the parent worries about scaling — push scores too big (Example 2) and softmax freezes into a near-one-hot with a flat, near-zero gradient.

8. Variance — why a $d_{k}$ knob appears at all

The parent's Step 3 argues the scores' spread grows with $d_{k}$ . One symbol — and one honest assumption — to define.

The picture: a dartboard. Small variance = darts clustered tight at the center; large variance = darts scattered wide.

Why the topic needs it. That $d_{k}$ spread is exactly what we divide by: $\frac{Q K ^{T}}{d _{k}} ⟹ spread back to \approx 1,$ keeping softmax in its lively, learnable middle range instead of its frozen peak. The scale factor is a variance-cancelling knob, nothing more mysterious.

9. Putting it together — the full formula, symbol by symbol

Every piece is now earned. Assemble them in the order the data flows:

Score every query against every key (Sections 4–6): $S = Q K^{T} \in R^{n \times m}$ .
Calm the spread (Section 8): divide by $d_{k}$ so the scores have standard deviation $\approx 1$ .
Turn scores into weights (Section 7): apply softmax to each row, giving a weight grid $A \in R^{n \times m}$ whose rows each sum to 1.
Blend the values (this last step): multiply the weight grid by the value matrix $V$ . This is exactly a matrix multiplication (Section 6) — for query $i$ it forms $\sum_{j = 1}^{m} A_{ij} v_{j}$ , a weighted average of the value rows using the attention weights. That is the "pull out a blend of the stored content" from our core idea.

You can now read the parent note's formula line by line without meeting a single unexplained symbol. From here you are ready for Self-Attention, Cross-Attention, and Multi-Head Attention, which all reuse this exact machinery.

How these feed the topic

Every arrow is a "you must own the tail before the head." Master the leaves (vector, variance) and the trunk (scaled attention) becomes readable line by line.

Equipment checklist

Test yourself — cover the right side and answer out loud.

What is a vector, in plain words?

An ordered list of numbers; pictured as an arrow with a direction and a length.

What does

q_{i}

mean?

The single number sitting in slot

i

of the list

q

How do you compute the norm

∥ q ∥

of a vector?

Square every slot, add them, take the square root — the Pythagorean length of the arrow.

Read

\sum_{i = 1}^{L} q_{i} k_{i}

in words.

Multiply matching slots of

q

and

k

, then add all

L

products.

What does the dot product

q \cdot k

tell you geometrically?

How aligned the two arrows are — positive if same direction, zero if perpendicular, negative if opposite; equals

∥ q ∥ ∥ k ∥ cos θ

What does

R^{r \times c}

denote?

The set of all grids (matrices) with

r

rows and

c

columns of real numbers.

Why must we transpose

K

before multiplying?

So keys sit as columns of length

d_{k}

, letting row-times-column dot products form the score grid; also so shapes align (

n \times d_{k}

times

d_{k} \times m

What is the shape of

S = Q K^{T}

when

Q

n \times d_{k}

and

K

m \times d_{k}

n \times m

What does softmax do to a row of scores?

Turns them into non-negative weights that sum to 1, exaggerating larger scores.

Why does softmax use

e^{x}

To force positivity and be smooth/differentiable, so bigger scores get exponentially more weight.

What happens to softmax when all scores are equal?

Every weight becomes

1/ m

— a perfectly even split.

Under what assumption does

Var (q \cdot k) = d_{k}

hold?

When each slot

q_{i}, k_{i}

is independent with mean 0 and variance 1; then the

d_{k}

products' variances add to

d_{k}

Why divide the scores by

d_{k}

To cancel that

d_{k}

spread, resetting the scores' standard deviation to about 1 so softmax stays sensitive and its gradients stay healthy.

What is the final step that turns the weight grid into the attention output?

Multiply the softmax weight grid

A

by the value matrix

V

: each output row is

\sum_{j} A_{ij} v_{j}

, a weighted blend of the values.