Visual walkthrough — Encoder vs decoder vs encoder-decoder

This page builds the whole story of Encoder vs Decoder vs Encoder-Decoder from a single idea: who is allowed to look at whom. Everything else — BERT, GPT, T5, translation, generation — falls out of that one choice.

We start from zero. No prior notation is assumed. Every symbol is earned before it is used.

---

Step 1 — The row of tokens and their three vectors

WHAT. We line up the input as a sequence of tokens. A token is just one chunk of text (a word or word-piece). We write the sequence as

$\mathbf{x}=[x_1,x_2,\dots ,x_n]$

• $x_1,\dots ,x_n$ — the tokens in order, position $1$ first.
• $n$ — how many tokens there are (the length).

Each token is turned into a list of numbers (a vector). From that vector we compute three new vectors by multiplying with three learned number-grids (matrices) $W_Q,W_K,W_V$:

$q_i=W_Q{e}_i,k_i=W_K{e}_i,v_i=W_V{e}_i$

• $e_i$ — the embedding vector of token $x_i$ (its raw meaning as numbers).
• $q_i$ — the query: "what is token $i$ looking for?"
• $k_i$ — the key: "what does token $i$ advertise about itself?"
• $v_i$ — the value: "what does token $i$ hand over if you attend to it?"

WHY three? Because looking is a matching game. A query from one token is compared against the keys of others; wherever query and key match well, that token's value gets pulled in. Splitting the roles (asking / advertising / delivering) is what lets attention be selective. See Self-Attention Mechanism.

PICTURE.

---

Step 2 — Scoring: how strongly should $i$ look at $j$?

WHAT. For token $i$ looking at token $j$, we measure how well the query $q_i$ matches the key $k_j$ using the dot product:

$s_{ij}=q_i\cdot k_j={\sum }_d(q_i{)}_d(k_j{)}_d$

• $s_{ij}$ — the raw score (a single number) for "$i$ attending to $j$".
• $q_i\cdot k_j$ — the dot product: multiply the vectors component-by-component, add it all up.
• large positive $\Rightarrow$ they point the same way $\Rightarrow$ strong relevance; near zero $\Rightarrow$ unrelated.

WHY the dot product and not something else? We need a single number that grows when two vectors agree in direction. The dot product is exactly that: it is large when the query and key are aligned. It answers the question "how relevant is $j$ to $i$?" cheaply, with one multiply-and-add.

Then we scale by $\sqrt{d_k}$:

$s_{ij}^{\prime }=\frac{s_{ij}}{\sqrt{d_k}}$

• $d_k$ — the length (number of components) of each key vector.
• $\sqrt{d_k}$ — the shrink factor.

WHY divide by $\sqrt{d_k}$? Adding up $d_k$ products makes the score grow roughly like $\sqrt{d_k}$ just by chance. If scores get huge, the next step (softmax) collapses to "all attention on one token" and the gradient dies. Dividing by $\sqrt{d_k}$ keeps the numbers in a healthy range.

PICTURE.

---

Step 3 — Softmax: turning scores into a peek-budget

WHAT. Each token has a fixed amount of "attention" (total $1$) to spread over the tokens it may look at. Softmax converts the raw scores into that spread:

${\alpha }_{ij}=\frac{\exp (s_{ij}^{\prime })}{\sum _{j^{\prime }}\exp (s_{i{j}^{\prime }}^{\prime })}$

• $\exp (\cdot )$ — the exponential; it makes every value positive and blows up big scores.
• numerator $\exp (s_{ij}^{\prime })$ — the un-normalised weight for token $j$.
• denominator ${\sum }_{j^{\prime }}\exp (s_{i{j}^{\prime }}^{\prime })$ — the sum over all tokens $i$ is allowed to see; dividing forces the weights to add to $1$.
• ${\alpha }_{ij}$ — the final attention weight: the fraction of $i$'s peek-budget spent on $j$.

WHY softmax? We want a set of positive numbers that sum to $1$ — a probability distribution — so "attention" behaves like a budget. Softmax is the smooth, differentiable way to do this, and the exponential rewards the best-matching token most.

The new representation of token $i$ is the budget-weighted blend of values:

${\mathbf{h}}_i={\sum }_j{\alpha }_{ij}{v}_j$

• ${\sum }_j{\alpha }_{ij}{v}_j$ — mix each value $v_j$ by its weight; tokens with big $\alpha$ dominate the blend.
• ${\mathbf{h}}_i$ — token $i$'s new, context-aware vector.

PICTURE.

---

Step 4 — The mask: the ONE knob that makes three architectures

WHAT. The word "all $j^{\prime }$" in the softmax denominator hides a choice. We control which $j$ are allowed by adding a mask matrix $M$ before softmax:

${\alpha }_{ij}=\frac{\exp (s_{ij}^{\prime }+M_{ij})}{{\sum }_{j^{\prime }}\exp (s_{i{j}^{\prime }}^{\prime }+M_{i{j}^{\prime }})},M_{ij}\in \{0,-\infty \}$

• $M_{ij}=0$ — token $i$ may look at token $j$ (score unchanged).
• $M_{ij}=-\infty$ — token $i$ may not; because $\exp (-\infty )=0$, that weight becomes exactly zero.

Two choices of $M$ give us two rules (see Attention Masking):

$$\underset{\text{bidirectional — encoder}}{\underbrace{M_{ij}=0\forall i,j}}\underset{\text{causal — decoder}}{\underbrace{M_{ij}=\{\begin{array}{ll}0& j\le i\\ -\infty & j>i\end{array}}}$$

WHY $-\infty$ and not just $0$-weighting afterwards? If we zeroed weights after softmax, the denominator would still have counted the forbidden tokens and the budget wouldn't sum to $1$. Injecting $-\infty$ before softmax removes them cleanly, so the remaining weights still add to exactly $1$.

WHY causal at all? When a decoder generates word $t$, words $t+1,t+2,\dots$ do not exist yet. Letting position $i$ peek right would be cheating — training on answers it won't have at generation time. The lower-triangular mask enforces "left-only". This is Causal Language Modeling; the all-ones mask powers Masked Language Modeling.

PICTURE.

---

Step 5 — Encoder-only: everyone peeks at everyone (BERT)

WHAT. Set $M_{ij}=0$ everywhere. Now token $3$ blends values from tokens $1,2,4,5,\dots$ — past and future. Stack $L$ such layers; each refines the blend:

${\mathbf{h}}_i^{(L)}={\text{Block}}^{(L)}({\mathbf{h}}_i^{(L-1)},\{{\mathbf{h}}_j^{(L-1)}{\}}_{j=1}^n)$

• ${\mathbf{h}}_i^{(\ell )}$ — token $i$'s vector after layer $\ell$.
• $\{{\mathbf{h}}_j^{(\ell -1)}\}$ — all tokens from the previous layer feed into the next.
• $L$ — total number of stacked layers.

WHY bidirectional? To understand, a word needs its whole surroundings. "bank" in "river bank" versus "bank account" only disambiguates if it sees words on both sides at once. See BERT.

PICTURE.

---

Step 6 — Decoder-only: peek left only (GPT)

WHAT. Use the causal mask. Position $t$ blends only $v_1,\dots ,v_t$. To predict the next token:

$P(x_t\mid x_{<t})=\text{softmax}\left(\mathbf{W}{\mathbf{h}}_t^{(L)}\right)$

• $x_{<t}$ — all tokens before position $t$.
• ${\mathbf{h}}_t^{(L)}$ — depends only on $x_1,\dots ,x_t$ (never the future).

The whole-sequence probability factors by the chain rule:

$P(x_1,\dots ,x_n)={\prod }_{t=1}^{n}P(x_t\mid x_{<t})$

• $\prod$ — multiply the per-step probabilities.
• Each factor is one causal prediction; multiplying them gives the joint probability, exactly like $P(A,B,C)=P(A)P(B\mid A)P(C\mid A,B)$.

WHY this shape? Autoregressive Generation: generate one token, append it, feed it back, repeat. The left-only rule is what makes training match generation. See GPT.

PICTURE.

---

Step 7 — Encoder-decoder: two rows joined by cross-attention (T5)

WHAT. Now run both rules together. The source $\mathbf{x}$ goes through a bidirectional encoder; the target $\mathbf{y}$ goes through a causal decoder — but the decoder adds a third kind of attention, cross-attention, that lets it read the encoder's outputs:

${\text{CrossAttn}}_t={\sum }_{i=1}^n\frac{\exp (q_t\cdot k_i/\sqrt{d_k})}{{\sum }_{j=1}^n\exp (q_t\cdot k_j/\sqrt{d_k})}{v}_i$

Here the roles are split across rows:

• $q_t=W_Q{\mathbf{z}}_t$ — query from the decoder (what the output word is asking for). ${\mathbf{z}}_t$ is the decoder's causal self-attention output at step $t$.
• $k_i=W_K{\mathbf{h}}_{\text{enc},i}$ — key from the encoder (what each source word offers).
• $v_i=W_V{\mathbf{h}}_{\text{enc},i}$ — value from the encoder (the source information delivered).

WHY cross-attention? In translation the decoder must look at the whole source while still not peeking at its own future. Self-attention (causal) handles the growing target; cross-attention (unmasked, over the source) handles the input. See Cross-Attention, Sequenceto-Sequence Models, and T5.

One decoder layer, in order: (1) causal self-attention on $\mathbf{y}$, (2) cross-attention into the encoder, (3) a feed-forward mix. Prediction:

$P(y_t\mid y_{<t},\mathbf{x})=\text{softmax}\left(\mathbf{W}{\mathbf{h}}_{\text{dec},t}\right)$

PICTURE.

---

Step 8 — Degenerate and edge cases (never leave a gap)

WHAT / WHY / PICTURE, four corner cases:

1. $n=1$ (single token). Softmax over one item gives ${\alpha }_{11}=1$. In any architecture the token just returns its own value — nothing to attend to. Causal and bidirectional coincide here.
2. First decoder position ($t=1$, causal). The mask allows only $j\le 1$, so position $1$ attends to itself alone: ${\alpha }_{11}=1$. A decoder's very first step has no left context — it leans entirely on the prompt/cross-attention.
3. All scores equal. If every $s_{ij}^{\prime }$ is the same, softmax gives uniform weights ${\alpha }_{ij}=1/(\text{count of allowed }j)$ — attention degenerates to a plain average. This is the "no preference" baseline before training sharpens it.
4. Empty source in encoder-decoder ($n=0$). No keys/values exist, so cross-attention has nothing to sum over — the decoder collapses to a pure decoder-only model. This shows the three architectures are one family: turn the source off and encoder-decoder is a decoder.

---

The one-picture summary

Everything above is one pipeline — score, scale, mask, softmax, blend — with only the mask changed, plus an optional cross-attention bridge.

Recall Feynman retelling — say it like a story

Line up the words. Give each word three name-tags: a question (query), a label (key), and a gift (value). To update a word, hold up its question and compare it against every label you're allowed to read — the closeness score is just a dot product. Divide the scores by $\sqrt{d_k}$ so they don't explode, then softmax them into a budget of attention that adds to one. Blend the gifts by that budget: that's the word's new, context-rich vector.

Now the only real choice: who may you read? If everyone may read everyone, you understand (BERT-style encoder). If you may only read to your left, you can generate one word at a time without cheating on the future (GPT-style decoder). If you want to translate, run both: an encoder reads the whole source, and a left-only decoder writes the target while a cross-attention bridge lets each output word peek at the entire source. Turn the source off and the translator becomes a plain generator — proof they're all the same machine wearing different masks.

Recall Quick self-test

Why divide the score by $\sqrt{d_k}$? ::: Scores grow like $\sqrt{d_k}$ by chance; dividing keeps softmax from saturating and gradients from dying. Why put $-\infty$ in the mask before softmax, not zero the weight after? ::: So the softmax denominator never counts forbidden tokens — the remaining weights still sum to exactly $1$. What single thing turns an encoder into a decoder? ::: The mask: all-zeros (bidirectional) → lower-triangular (causal). In encoder-decoder cross-attention, where do $Q$, $K$, $V$ come from? ::: $Q$ from the decoder, $K$ and $V$ from the encoder outputs. What happens to cross-attention if the source is empty? ::: Nothing to attend to — the model reduces to decoder-only.