The original - Attention is All You Need - architecture: Exercises

Before we begin, let us pin down every symbol so nothing is used unexplained.

Definition Symbol dictionary (read once)

$d_{model}$ ::: the width of every token vector as it flows through the model (the paper uses $512$ ).
$h$ ::: the number of parallel attention heads (the paper uses $8$ ).
$d_{k}$ ::: the width of one head's key/query vectors, equal to $d_{model} / h$ .
$d_{v}$ ::: the width of one head's value vectors; in the paper it equals $d_{k} = d_{model} / h = 64$ .
$Q, K, V$ ::: matrices of queries, keys, values — three different linear "views" of the same token vectors.
$Q K^{T}$ ::: a table of dot products; entry $(i, j)$ measures how similar query $i$ is to key $j$ .
softmax ::: turns a row of raw scores into positive weights that add to $1$ (a probability distribution).
$N$ ::: how many identical encoder (or decoder) layers are stacked (the paper uses $6$ ).
$d_{f f}$ ::: the hidden width of the feed-forward network inside each layer (the paper uses $2048$ ).
$Embed (x)$ ::: the embedding function — it looks up token $x$ (a discrete symbol like a word-piece) in a learned table and returns its continuous vector of width $d_{model}$ .
$PE (pos)$ ::: the positional-encoding function — a fixed (non-learned) rule that maps a position number $pos$ to a vector of width $d_{model}$ built from sines and cosines; $PE (pos, j)$ is its $j$ -th entry.
$W^{Q}, W^{K}, W^{V}$ ::: learned projection matrices that turn a token vector into its query, key, and value; each has shape $d_{model} \times d_{k}$ (per head) or $d_{model} \times d_{model}$ (all heads stacked).
$W^{O}$ ::: the learned output-projection matrix ( $d_{model} \times d_{model}$ ) applied after all heads are concatenated back together.
$W_{1}, b_{1}, W_{2}, b_{2}$ ::: the two weight matrices and two bias vectors of the feed-forward network; $W_{1}$ is $d_{model} \times d_{f f}$ , $W_{2}$ is $d_{f f} \times d_{model}$ .
$γ, β$ ::: learned per-feature scale and shift in layer norm — after standardizing, the model may stretch ( $γ$ ) and offset ( $β$ ) each feature.
$ϵ$ ::: a tiny constant added inside the square root of layer norm so we never divide by zero.

Level 1 — Recognition

Problem 1.1

The encoder has $N = 6$ layers, each with two sub-layers; the decoder has $N = 6$ layers, each with three sub-layers. How many sub-layers are there in the whole model?

Recall Solution

WHAT: count sub-layers per stack, then add. Encoder: $6 \times 2 = 12$ . Decoder: $6 \times 3 = 18$ . Total $= 12 + 18 = 30$ sub-layers. WHY this matters: every one of these $30$ sub-layers is separately wrapped in a residual connection + layer norm, so there are also $30$ "Add & Norm" blocks. Knowing this count tells you exactly how many places gradients must flow through cleanly — it's why residual connections (a shortcut past each block) are non-negotiable in a stack this deep.

Problem 1.2

With $d_{model} = 512$ and $h = 8$ heads, what is $d_{k}$ (the dimension of one head)?

Recall Solution

By definition $d_{k} = d_{model} / h = 512/8 = 64$ . Same for $d_{v}$ (the value width per head is also $64$ in the paper). WHY this matters: because $8$ heads each of width $64$ recombine to $8 \times 64 = 512 = d_{model}$ , running $8$ heads costs roughly the same compute as one full-width head — you get diverse "viewpoints" essentially for free. This budget-splitting is the whole trick of multi-head attention.

Problem 1.3

Name the three sub-layers of a decoder layer in the order the data flows through them.

Recall Solution

Masked multi-head self-attention (each position sees only itself and earlier positions).
Encoder–decoder (cross) attention ( $Q$ from decoder, $K, V$ from encoder).
Position-wise feed-forward network. Each is followed by Add & Norm.

Level 2 — Application

Problem 2.1

For $d_{model} = 4$ , compute the four positional-encoding values at position $pos = 1$ .