Visual walkthrough — The original - Attention is All You Need - architecture

Before line one, the only thing you must accept: a word can be turned into a list of numbers — an arrow in space. That's it. Everything else we construct.

Step 1 — A word is an arrow

WHAT. Take three words: "The", "cat", "sat". A machine cannot add letters, so we hand each word a short list of numbers — say 2 numbers so we can draw it on paper. That list is a point, and the arrow from the origin to that point is the word's vector (its "embedding").

WHY. We need something we can do arithmetic on. Arrows can be measured, compared, and pointed in similar or different directions. "Similar meaning" will become "arrows pointing the same way" — a geometric idea we can compute.

PICTURE. Three arrows leaving the origin. "cat" and "sat" lean close together (they often appear together); "The" points off elsewhere. The angle between arrows is the whole game.

Step 2 — How "aligned" are two arrows? The dot product

WHAT. To measure whether two arrows point the same way, multiply them coordinate-by-coordinate and add up the results. For arrows $a=[a_1,a_2]$ and $b=[b_1,b_2]$: $a\cdot b=\underset{\text{1st coords multiplied}}{\underbrace{a_1{b}_1}}+\underset{\text{2nd coords multiplied}}{\underbrace{a_2{b}_2}}$

WHY this tool and not another? We want ONE number that grows when two arrows agree and shrinks (even goes negative) when they disagree. The dot product is exactly that: $a\cdot b=\Vert a\Vert \Vert b\Vert \cos \theta$, where $\theta$ is the angle between them. Same direction ($\theta =0$) → $\cos \theta =1$ → big. Perpendicular ($\theta =90^{\circ }$) → $\cos \theta =0$ → zero. Opposite → negative. No other simple operation packs "alignment" into a single number this cleanly.

PICTURE. Two arrows with the angle $\theta$ marked; a dial reads high when they overlap, drops to zero at a right angle, goes negative when they oppose.

Step 3 — Every word scores every word: the matrix $Q{K}^T$

WHAT. For now pretend every word plays three roles with itself, so let queries $Q$, keys $K$, and values $V$ all equal our stacked word matrix $X$ (we split them apart in Step 6). Compute the dot product of every row of $Q$ with every row of $K$. Collect them in a grid $S=Q{K}^T$.

• $Q$ = the words asking ("who is relevant to me?").
• $K$ = the words offering an identity to be matched against.
• $S_{ij}=Q_i\cdot K_j$ = how much word $i$ (the asker) aligns with word $j$ (the offerer).

WHY. Attention is "every token looks at every token." That is exactly an $n\times n$ table of pairwise alignment scores. Matrix multiply $Q{K}^T$ computes all $n^2$ dot products in one shot — this is why the Transformer parallelizes: no loops, just one big multiply.

PICTURE. A $3\times 3$ heat grid. Row = the asking word, column = the offered word. The diagonal glows (a word aligns with itself); "cat"–"sat" is warm; "The" is cool everywhere.

Step 4 — Why we shrink the scores: dividing by $\sqrt{d_k}$

WHAT. Before doing anything else, divide every score by $\sqrt{d_k}$, where $d_k$ is the number of coordinates in each key vector: $S^{\prime }=\frac{Q{K}^T}{\sqrt{d_k}}$

WHY this exact number? A dot product adds up $d_k$ little products. If each coordinate is random with spread $1$, the sum of $d_k$ of them has spread $\sqrt{d_k}$ (variances add; standard deviation grows like the square root). So with $d_k=64$, raw scores swing around $\pm 8$ just from noise. Feed those into the next step (softmax) and it saturates — one score wins everything, gradients die. Dividing by $\sqrt{d_k}$ cancels exactly that growth, keeping scores at a healthy spread of about $1$ regardless of dimension. That's why the square root and not the number itself: we're undoing a standard-deviation growth, and standard deviation grows like $\sqrt{d_k}$.

PICTURE. Two histograms of scores: the raw one is wide and spiky (spread $\sqrt{d_k}$); after dividing, it's a tidy narrow bell of spread $\approx 1$.

Step 5 — Turning scores into weights: softmax

WHAT. Each row of $S^{\prime }$ is a list of raw scores for one asking word. We want those to become positive weights that add to 1 — a recipe of "how much of each other word to mix in." That is the job of softmax. For a row of scores $s_1,\dots ,s_n$: $\text{softmax}(s{)}_j=\frac{e^{s_j}}{\sum _{k=1}^n{e}^{s_k}}$

WHY the exponential? Three demands: (1) weights must be positive — $e^s$ is always positive; (2) bigger score → bigger weight, sharply — the exponential amplifies leads; (3) weights must sum to 1 — dividing by the total of all $e^{s_k}$ guarantees it. No plain "score / total" would do, because raw scores can be negative. The exponential is the natural tool that satisfies all three at once.

PICTURE. A row of three bars (raw scores) feeding into a machine that outputs three shorter bars summing to a full unit height — a probability recipe.

Step 6 — Mix the values: the final output

WHAT. Now use the recipe. Each word carries a value vector $V_j$ (the content it contributes). The output for asking-word $i$ is the weighted blend: ${\text{Output}}_i={\sum }_j{A}_{ij}{V}_j$ Stacked for all rows, this is just $AV$.

WHY split $Q$, $K$, $V$ apart now? In real Transformers each word is multiplied by three learned matrices $W^Q,W^K,W^V$ before any of this: $Q=X{W}^Q,K=X{W}^K,V=X{W}^V$. The asking-identity ($Q$), the matchable-identity ($K$), and the delivered-content ($V$) are then free to be different — a word can advertise one thing to be found by, and deliver another thing once chosen. Learning lets the model decide what each role should be. (See Scaled Dot-Product Attention and Multi-Head Self-Attention.)

PICTURE. The recipe row for "cat" $[0.31,0.37,0.32]$ pouring three value arrows into a mixing bowl; out comes one blended arrow — "cat" now carries a pinch of "The", a big scoop of itself, a scoop of "sat".

Step 7 — The edge cases we must not skip

Every input the reader could hit, drawn out:

Case A — a word attends only to itself. If one score dominates (say after training a word's query aligns almost perfectly with just its own key), softmax pushes that weight to $\approx 1$ and the rest to $\approx 0$. Output $\approx V_i$: the word passes through nearly unchanged.

Case B — total indifference (all scores equal). If every $S_{ij}^{\prime }$ is the same, softmax gives every weight $=1/n$. The output is the plain average of all values — attention "gives up" and blends everyone equally. This is the degenerate, no-information case.

Case C — the mask (used in the decoder). To stop a word from peeking at future words, we add $-\infty$ to forbidden scores before softmax. Since $e^{-\infty }=0$, those weights become exactly $0$ — future words contribute nothing. Word $i$ can only blend words $1\dots i$. This gives the staircase pattern of Attention Masking.

Case D — zero-length value. If some $V_j=0$, it contributes nothing to any blend no matter how much attention it receives — a word can be attended to yet deliver empty content. The score and the content are independent knobs.

PICTURE. Four mini attention grids side by side: sharp diagonal (A), flat gray uniform (B), lower-triangular staircase (C), and one with a zeroed value column (D).

The one-picture summary

One diagram, left to right: three word-arrows → the $Q{K}^T$ score grid → shrink by $\sqrt{d_k}$ → softmax into recipe rows → multiply by $V$ → three refreshed output arrows, each a blend.

Reveal-yourself checks: