The original - Attention is All You Need - architecture: Foundations

This page assumes you have seen none of the notation. We build each symbol from a picture before the parent note is allowed to use it.

1. A vector — an arrow of numbers

Picture it. In the figure below the arrow has two numbers, so it lives on a flat board. Real Transformers use arrows with $512$ numbers — impossible to draw, but the idea is identical: an arrow pointing somewhere in a high-dimensional space.

Figure — The original - Attention is All You Need - architecture

Why the topic needs it. A word like "cat" cannot be added, scaled, or compared as a raw symbol. Turn it into an arrow of numbers and suddenly all of arithmetic is available. Every box in the parent note is moving arrows around.

2. Symbols you will keep seeing

Before any formula, here is the full glossary the parent note leans on. Read top to bottom — each builds on the one above. Every symbol here is pictured or drawn in the sections that follow; the map at the end shows exactly which figure defines which.

Definition Core symbols

$x$ — one input token turned into an arrow (a chunk of text; see Tokenization). Pictured in §2a as one row of the token grid.
$n$ — how many tokens are in the sentence right now. The number of rows in that grid.
$pos$ — the position of a token in the sentence ( $0, 1, 2, \dots$ ). Which row you are looking at.
$i, j$ — plain counters. $i$ points at a row (a token/position), $j$ at a column (an entry, or the "other" token). Drawn as the row/column pointers in §2a.
$d_{model}$ — the length of every arrow inside the model ( $= 512$ in the paper). One fixed width for all vectors so they can be added. The number of columns in the grid.
$d_{in}, d_{out}$ — generic labels for "how long the arrow going in to a reshape is" and "how long the arrow coming out is" (§3). They are not fixed numbers; each reshape sets its own.
$E$ (embedding) and $PE$ (positional encoding) — two arrows added together to make a token's starting vector (§2b).
$W$ — a weight matrix: a learned reshaping machine (§3). $b$ — a bias vector: a learned shift added after reshaping (§3).
$⊤$ (transpose) — flipping a grid's rows and columns (§3a).
$Q, K, V$ — three grids called Queries, Keys, Values (§6), with shapes stated there. $W^{O}$ — the final mix matrix that glues heads back together (§6a).
$d_{k}, d_{v}$ — the width of each query/key arrow and each value arrow ( $= 64$ ). Shown in §6.
$h$ — the number of parallel attention "heads" ( $= 8$ ); they split $d_{model}$ into $h$ blocks of size $d_{k}$ (§6a).
$\cdot$ — the square-root sign; $a$ is "the number that, times itself, gives $a$ " (used to tame growth in §8).
$μ, σ$ — the mean and spread of a set of numbers (§7). $γ, β$ — learned rescale/shift knobs of layer norm (§7).

Do not memorise these — each is drawn below.

2a. The token grid — where $x$ , $n$ , $d_{model}$ , $i$ , $j$ , $pos$ live

Every symbol about shape lives in one picture: the grid $X$ . Each row is one token's arrow $x$ (there are $n$ of them, indexed by $pos$ or $i$ ); each column is one of the $d_{model}$ entries (indexed by $j$ ).

2b. Where the arrows come from — embedding $E$ and positional encoding $PE$

A raw token is just a symbol ("cat"). Two steps turn it into a starting arrow of length $d_{model}$ .

3. Multiplying an arrow by a matrix — a reshaping machine

We use the row-vector convention throughout, matching the parent note ( $Q = X W^{Q}$ , with $x$ written as a row and the matrix on its right).

Why the topic needs it. The projections $Q = X W^{Q}$ , the feed-forward $x W_{1} + b_{1}$ , the output mix $W^{O}$ — all of them are "reshape this arrow through a learned machine, then nudge". Training tunes the numbers inside $W$ and $b$ .

3a. Transpose $⊤$ — flipping a grid

4. The dot product — how aligned are two arrows?

This is the heart of attention, so we slow down.

Why this tool and not another? We need a single number saying "how relevant is token A to token B?". Distance would work but grows with vector length and ignores direction. The dot product answers exactly "do these two arrows agree in direction?" — which is precisely what "these two words are related" should mean once meaning lives in direction. That is why the parent's score matrix is $Q K^{⊤}$ (every query dotted with every key).

5. Softmax — turning scores into a fair vote

Why the exponential $e^{s}$ and not just "divide by the total"? Two reasons. First, raw scores can be negative; $e^{s}$ is always positive, so every weight is a valid share. Second, $e^{s}$ exaggerates the winner — a slightly higher score grabs a much larger slice, which lets attention focus sharply on the most relevant token instead of spreading itself thin.

6. Q, K, V — three costumes for the same tokens

Analogy. A library search: your typed query meets each book's index card (key); the match strength decides how much of each book's contents (value) you carry away.

Why the topic needs it. Splitting one arrow into these three roles lets the same token behave differently as a seeker versus as a target. Without it, attention could only measure raw similarity, never learned, task-specific relevance. This is the core of Scaled Dot-Product Attention and Multi-Head Self-Attention.

6a. $h$ heads and the final mix $W^{O}$

7. Mean, variance, and layer normalization — keeping numbers tame

Why the topic needs it. Deep stacks make numbers drift huge or tiny, and softmax + dot products misbehave at extremes. Layer Normalization uses exactly this to keep every arrow at a sane scale before the next layer; $γ$ and $β$ give it a way to undo the squashing when helpful.

8. Why $d_{k}$ and $d_{model}$ appear

9. How it all connects

Read it top-down: tokens become embedding-plus-position arrows in a grid, arrows get reshaped into Q/K/V, transpose lets dot products score them, scaling + softmax turn scores into a vote, heads split the width, the vote blends the values, $W^{O}$ mixes the heads, and normalization (with $γ, β$ ) keeps everything stable enough to stack six times.

Equipment checklist

Cover the right side and answer aloud — if you can, you are ready for the parent note.

What is a vector, in one picture?

An arrow of numbers pointing to a spot in

d

-dimensional space.

What does the symbol

R

mean?

The set of all real numbers — any ordinary number on the number line.

In the token grid

X

, what is a row, and what is a column?

A row is one token

x

at position

pos = i

; a column is one feature slot

j

shared across all tokens.

What does

R^{n \times d}

describe?

A grid with

n

rows and

d

columns — one arrow per row.

Where does a token's starting arrow come from?

Add the meaning arrow

E (x)

to the position arrow

PE (pos)

In the row-vector convention, what shapes must

W

and

b

have for

x W + b

to work?

W \in R^{d_{in} \times d_{out}}

and

b \in R^{d_{out}}

;

b

shifts the reshaped arrow.

What do

d_{in}

and

d_{out}

stand for?

The lengths of the arrows going into and out of a reshape.

What does the transpose

K^{⊤}

do and why is it in

Q K^{⊤}

Swaps rows and columns so key arrows become columns, letting one multiply compute all query–key dot products.

What single number does a dot product give, and when is it large?

A relevance/alignment score; large and positive when the two arrows point the same direction.

Why is attention built on dot products rather than distances?

We need one number for "how related?"; the dot product measures directional agreement, which is exactly relatedness once meaning lives in direction.

What does softmax output, why the exponential, and what stability trick is used?

Positive weights summing to 1; the exponential keeps them positive and sharpens focus; subtract

max

before exponentiating to avoid overflow (same result).

What are the shapes of

Q

K

V

Q, K \in R^{n \times d_{k}}

and

V \in R^{n \times d_{v}}

What does

h

do to the width

d_{model}

, and what does

W^{O}

do afterwards?

Splits it into

h

blocks of size

d_{k} = d_{model} / h

;

W^{O} \in R^{d_{model} \times d_{model}}

mixes the concatenated heads back into one output.

What does

\cdot

mean and why divide scores by

d_{k}

a

is the number whose square is

a

; dot products grow like

d_{k}

, so dividing cancels that and keeps softmax in its high-gradient range.

What do

μ

σ

γ

β

do in layer norm?

μ, σ

recenter and rescale to mean 0 / spread 1;

γ, β

are learned scale and shift applied afterward.