Foundations — Encoder vs decoder vs encoder-decoder

This page is the toolbox. Before you can read the parent note, you need to know what every squiggle means. We build each symbol from nothing, anchor it to a picture, and say why the topic needs it. Read top to bottom — each item stands on the one before it.

0. What is a "token" and a "sequence"?

Before any math, we need the raw material.

Picture a row of boxes, each holding one word:

We write a sequence as $\mathbf{x}=[x_1,x_2,\dots ,x_n]$.

• The bold $\mathbf{x}$ means "the whole list at once."
• Each $x_i$ (read "x-sub-i") is the token sitting in box number $i$.
• $n$ is simply how many tokens there are — the length of the sentence.

Why the topic needs it: everything a transformer does is "take a sequence in, produce a sequence out." $n$ tells us how many boxes we are juggling.

1. The subscript $i$, $j$, $t$ — pointing at a box

We use different letters only to keep two fingers apart:

• $i$ usually means "the box I am currently thinking about."
• $j$ usually means "some other box I am comparing against."
• $t$ means "the current time step" — used when we generate one word after another.

Why the topic needs it: attention is all about pairs of tokens ("how much should box $i$ care about box $j$?"). You cannot talk about a pair without two pointers.

2. The embedding vector ${\mathbf{h}}_i$ — a word becomes a list of numbers

A computer cannot add the word "bank." So we turn each token into a list of numbers.

Read ${\mathbb{R}}^d$ as: "$\mathbb{R}$" is the set of all ordinary (real) numbers, and the little $d$ means "$d$ of them in a row."

The symbol ${\mathbf{h}}_i$ is the meaning of token $i$ expressed as numbers. The superscript in ${\mathbf{h}}_i^{(L)}$ (read "h-sub-i, layer L") just says "this is the version of that vector after layer number $L$."

Why the topic needs it: the whole point of a transformer is to improve these number-lists layer by layer, so that by the end ${\mathbf{h}}_{\text{"bank"}}$ already "knows" whether we mean a river or money.

3. The dot product $q\cdot k$ — measuring "how similar?"

To decide how much box $i$ should care about box $j$, we need a similarity score between two vectors. The tool for that is the dot product.

Why this tool and not another? We need a single number that says "how aligned are these two meanings?" Distance would also compare vectors, but the dot product is cheap (one multiply-add per slot) and it grows when directions agree — exactly the "relevance" signal attention wants.

4. $q_i$, $k_j$, $v_i$ — the three roles a token plays

Each token vector is copied into three different vectors by three learned recipes:

Think of a library. Your query is your search request; each book's key is its title on the spine; the value is the book's contents. You match your query against every spine, then read the books that matched best.

The dot product $q_i\cdot k_j$ therefore reads: "how well does what box $i$ wants match what box $j$ offers?"

Why the topic needs it: splitting each token into these three roles is what lets attention be selective — a token can ask for one thing while offering another.

5. $W_Q$, $W_K$, $W_V$ — the recipes (matrices)

Where do $q$, $k$, $v$ come from? From the embedding $\mathbf{h}$ multiplied by a matrix.

• $W_Q$ turns a token into its query.
• $W_K$ turns a token into its key.
• $W_V$ turns a token into its value.

These grids are exactly the numbers the network learns during training. Everything else in attention is fixed arithmetic; the intelligence lives in these matrices.

Why the topic needs it: without learnable matrices, every token would ask and offer the same thing. The matrices let the model discover useful notions of relevance from data.

6. Softmax — turning scores into a "how-much-I-care" distribution

We now have raw scores $s_{ij}=q_i\cdot k_j$ for every other box $j$. We want to turn them into weights that are all positive and add up to 1 (so they behave like "percent of attention").

Read it piece by piece:

• $e^{s_j}$ (the exponential) makes every score positive — even negative scores become small positive numbers, never zero.
• The bottom, ${\sum }_{j^{\prime }}{e}^{s_{j^{\prime }}}$ (read "sum over all $j^{\prime }$"), adds up all those positive numbers so we can divide by the total. This is the normalising step that forces the outputs to sum to 1.

The result ${\alpha }_{ij}$ (Greek letter alpha) is the attention weight: the fraction of box $i$'s focus spent on box $j$. All the $\alpha$'s for a fixed $i$ add to 1.

7. Putting it together — one attention output

Now the payoff. Box $i$'s new vector is a weighted blend of everyone's values: ${\mathbf{h}}_i={\sum }_{j=1}^n{\alpha }_{ij}{v}_j$

The ${\sum }_{j=1}^n$ (read "sum as $j$ goes from 1 to $n$") means: walk through every box $j$, take a ${\alpha }_{ij}$-sized scoop of its value $v_j$, and pour them all together. Boxes you care about a lot contribute most.

8. The scale $\sqrt{d_k}$ — why we divide

$d_k$ is just the dimension of the key vectors — how long each $k$ is. The square root is the exact amount that cancels the natural growth of a sum of $d_k$ random products.

9. The mask $M$ — deciding who may look at whom

This is the symbol that separates the three architectures.

Why $-\infty$? Because $e^{-\infty }=0$: after softmax, a forbidden pair gets weight exactly zero. It is completely blocked.

Two shapes matter:

• All-ones (bidirectional): every box may see every box. Used by the encoder. Good for understanding a whole sentence at once.
• Lower-triangular (causal): box $i$ may see only boxes $j\le i$ (itself and the past), never the future. Used by the decoder. Needed for generation, because a word can't peek at words not yet written.

Why the topic needs it: change the mask and the same attention machine becomes an understander or a generator. The mask is the whole plot of the parent note. See Attention Masking.

10. The product $\prod$ and conditional $P(x_t\mid x_{<t})$ — probability of a sentence

Decoders generate one token at a time, so we need to talk about probabilities of sequences.

The big $\prod$ (capital Greek pi) means "multiply all these together," just as $\sum$ means "add all together."

11. Cross-attention — a second sentence enters

Encoder–decoder models have two sequences (e.g. English → French). Cross-attention is attention where the query comes from one sequence and the keys/values from another.

It is the same attention formula from §7 — only the source of $Q$ differs from the source of $K,V$. This is how the French decoder "reads" the English encoding. See Cross-Attention.

Prerequisite map

Related building blocks in the vault: Self-Attention Mechanism, Positional Encoding, Multi-Head Attention, Masked Language Modeling, Sequenceto-Sequence Models, Autoregressive Generation, T5.

Equipment checklist

Read the question, answer in your head, then reveal.