The original - Attention is All You Need - architecture: Question bank

Definition Symbols used on this page (read first)

Everything below reuses a handful of names. Pin them down once here so no trap depends on a symbol you haven't seen.

Token — one chunk of the input (roughly a word, see Tokenization).
Sequence dimension $n$ — how many tokens are in the input (the "how long" axis).
Feature dimension $d_{model}$ — the length of the vector representing one token (the "how wide" axis). The original paper uses $d_{model} = 512$ . These two axes are different: $n$ counts tokens, $d_{model}$ counts numbers inside one token.
$Q, K, V$ (query, key, value) — three vectors made from each token by multiplying it with learned matrices $W^{Q}, W^{K}, W^{V}$ . A query asks "what am I looking for?", a key advertises "what do I offer?", and a value is "what I hand over if chosen." Attention matches queries to keys, then mixes values — see Scaled Dot-Product Attention.
$d_{k}$ — the width of each query/key vector. With $h = 8$ heads, $d_{k} = d_{model} / h = 64$ . It appears in the $d_{k}$ scaling because dot-product magnitude grows with this width.
$d_{f f}$ — the hidden width of the feed-forward network (the FFN); the paper uses $d_{f f} = 2048$ , larger than $d_{model}$ so the FFN can expand then compress.
$ε$ (epsilon) — a tiny constant (typically $1 0^{- 5}$ to $1 0^{- 6}$ ) added inside Layer Normalization's square root so we never divide by zero.
PE — short for Positional Encoding (see Positional Encoding); the vector added to a token embedding to tell the model where the token sits.

True or false — justify

Every item is a claim. Decide true or false, then say why — the reveal gives the reasoning, not just the verdict.

The Transformer removes recurrence, so it has no notion of word order at all.

False. Attention itself is order-blind, but the Positional Encoding (PE) from Positional Encoding is added to embeddings before the first layer, injecting order back in. Remove that and it truly loses order.

Layer normalization normalizes across the batch dimension like batch norm does.

False. Layer Normalization computes

μ, σ^{2}

across the

d_{model}

feature dimensions of a single token — so it works identically at batch size 1 and needs no running statistics.

The encoder uses masked self-attention just like the decoder.

False. Only the decoder's first sub-layer is masked. The encoder sees the whole input at once, so every token may attend to every other token — no Attention Masking.

Multi-head attention with

h = 8

heads costs roughly 8× the compute of single-head attention of the same

d_{model}

False. Each head uses

d_{k} = d_{model} / h = 64

dims, so the total projected width stays

d_{model}

. The cost is comparable — heads split the dimensions, they don't multiply them.

Residual connections are optional polish; the architecture works fine without them.

False. With 6 encoder + 6 decoder layers (many sub-layers deep), Residual Connections are what let gradients flow back without vanishing. Removing them typically breaks training entirely.

The

d_{k}

scaling in

softmax (Q K^{T} / d_{k})

is there to keep the output values small.

False. It keeps the softmax input in a well-conditioned range. Dot products grow like

d_{k}

in variance; without scaling, softmax saturates and gradients vanish — see Scaled Dot-Product Attention.

Positional encodings are learned parameters trained by backprop.

False (in the original paper). The sine/cosine Positional Encoding is a fixed function of position — no parameters. The paper tested learned ones and found them roughly equal, choosing sinusoids for length extrapolation.

Cross-attention in the decoder uses the decoder's own output as keys and values.

False. In encoder–decoder attention,

Q

comes from the decoder but

K

and

V

come from the encoder output. That's exactly how the decoder reads the source sentence.

The FFN sub-layer mixes information between different token positions.

False. The position-wise FFN is applied independently to each position with shared weights — no cross-position interaction. Only attention mixes positions.

Softmax attention weights in one row can be negative if the scores are negative.

False. Softmax outputs are always non-negative and sum to 1 per row. Negative scores just become small positive weights, never negative ones.

Spot the error

Each line contains a deliberately wrong statement. Name the error and correct it.

"We scale by

d_{model}

inside the attention softmax."

The scale inside softmax is

d_{k} = 64 = 8

, not

d_{model} = 512

. The

d_{model}

factor scales the embeddings, a different place entirely.

"Masking sets future scores to 0 before softmax."

It sets them to

- \infty

, not 0. After softmax

- \infty

becomes weight 0; a raw 0 would still get positive weight — see Attention Masking.

"The decoder mask blocks position

i

from seeing positions

j \leq i

Backwards. It blocks

j > i

(the future) while allowing

j \leq i

(past and present). Autoregressive generation needs the past, not blocks it.

"Each encoder layer has three sub-layers."

The encoder has two (self-attention + FFN). The decoder has three (masked self-attn + cross-attn + FFN). Easy to swap them.

"Layer norm is applied, then the residual is added:

x + LayerNorm (Sublayer (x))

In the original post-norm design it is

LayerNorm (x + Sublayer (x))

— the residual is added first, then normalized together.

d_{f f} = 2048

means the FFN outputs 2048-dimensional vectors."

d_{f f}

is the hidden width. The FFN expands from

d_{model} = 512

up to 2048, applies ReLU, then projects back to

d_{model} = 512

on output.

"Different heads are forced to learn different relationships by an explicit loss term."

There is no such loss. Heads differ only because their projection matrices

W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}

start from different random inits and specialize during training — see Multi-Head Self-Attention.

Why questions

Answer the "why" — the reveal gives the mechanism.

Why multiply embeddings by

d_{model}

before adding PE?

Embeddings start with variance

\sim 1

but the Positional Encoding is bounded in

[- 1, 1]

. Scaling by

512 \approx 22.6

lets the semantic content dominate early so PE nudges position without drowning meaning.

Why use sine and cosine rather than just one?

The sin/cos pair lets

PE (pos + k)

be written as a fixed linear (rotation) transform of

PE (pos)

, so the model can learn relative offsets easily.

Why give each dimension a different frequency (the

1000 0^{2 i / d}

term)?

It creates a spectrum of wavelengths: high-frequency dims flip fast (fine position), low-frequency dims change slowly (coarse position). Together they encode position uniquely across scales.

Why does the decoder need masking during training but not (in the same way) at inference?

In training with Teacher Forcing the whole target is fed at once, so masking prevents peeking at future tokens. At inference tokens are generated one at a time, so the future doesn't exist yet.

Why attention instead of recurrence for long-range dependencies?

Any two tokens connect in a single attention step (path length 1), whereas an RNN needs

O (n)

steps, degrading the signal. Attention makes distant relations as cheap as adjacent ones.

Why an FFN at all if attention already mixes information?

Attention is a weighted sum — essentially linear per output. The FFN's ReLU adds the non-linearity and extra capacity needed to transform each token's representation.

Edge cases

Boundary and degenerate scenarios — the answer explains what actually happens.

What does self-attention output for a sequence of length

n = 1

With one token, its query attends only to itself, softmax gives weight 1, and the output is just its own value vector (transformed). Attention becomes an identity-like pass.

What about a totally empty sequence,

n = 0

There are no tokens, so

Q, K, V

are empty (

0 \times d_{k}

) matrices and

Q K^{T}

is a

0 \times 0

matrix — attention is dimensionally undefined and has nothing to output. Real pipelines guard against this: an all-padding batch is either dropped or a start token is required, so

n \geq 1

always holds.

What happens to the causal mask at the very first decoder position?

Position 1 may attend only to position 1 — every other entry in its row is

- \infty

. Its softmax row is a single 1, so it depends on nothing but itself, as it should.

If two tokens have identical embeddings and identical positions, can attention tell them apart?

No. Identical query/key/value vectors produce identical attention behaviour. In practice positions differ, so Positional Encoding breaks the tie — that's part of its job.

What if all scaled scores in a row are equal?

Softmax then gives a uniform distribution (

1/ n

each), so the output is the plain average of all value vectors — attention "abstains" from focusing.

Does the sinusoidal PE break for a position longer than any seen in training?

No — that's the point. Since PE is a fixed continuous function of

pos

, it extrapolates to unseen lengths, unlike a learned table which has no entry beyond its trained range.

What happens if

ε = 0

in LayerNorm and a token's features are all equal?

Then

σ^{2} = 0

and you divide by zero. The small

ε \approx 1 0^{- 5}

inside

σ^{2} + ε

exists precisely to keep this degenerate constant-feature case numerically safe.

Recall Fast self-test

One decoder layer has how many sub-layers, and which one is masked? ::: Three sub-layers; only the first (self-attention) is causally masked. Where do $K$ and $V$ come from in cross-attention? ::: From the encoder output, while $Q$ comes from the decoder. What do $n$ and $d_{model}$ count, respectively? ::: $n$ counts tokens (sequence length); $d_{model}$ counts numbers inside one token's vector (feature width).