Worked examples — Encoder vs decoder vs encoder-decoder

The parent note told you what the three architectures are. This page makes you compute them. We will hand-crank real numbers through attention masks, softmax, and cross-attention — one scenario at a time — until no case can surprise you.

Everything here rests on one machine you must already picture: the attention score. Before we open the matrix, let's re-earn every symbol.

---

The scenario matrix

Every situation this topic can hand you is one row below. The examples that follow are labelled with the cell they cover, so by the end no blank cell remains.

Cell	Scenario class	Architecture	What makes it tricky
A	Bidirectional attention, all-1s mask	Encoder (BERT)	every token sees every token
B	Causal attention, lower-triangular mask	Decoder (GPT)	future tokens blocked with $-\infty$
C	Zero / degenerate input: sequence length 1	Decoder	softmax over a single element
D	Limiting case: two equal scores	Encoder	ties → uniform split
E	Sign / negative scores	Any	negative dot products still valid
F	Cross-attention: decoder queries encoder	Encoder-Decoder (T5)	different masks stacked
G	Real-world word problem	Encoder-Decoder	translation alignment
H	Exam twist: scaling by $\sqrt{d_k}$ actually changes the answer	Any	why the $\sqrt{d_k}$ is not decoration

Prerequisites you can revisit: Self-Attention Mechanism, Attention Masking, Cross-Attention, Causal Language Modeling, Masked Language Modeling, Autoregressive Generation.

---

Example 1 — Cell A: full bidirectional softmax (encoder)

Forecast: which token should token $2$ listen to most? Guess before computing.

1. Scale each score by $\sqrt{d_k}=\sqrt{4}=2$. $s_{21}=\frac{2}{2}=1,s_{22}=\frac{4}{2}=2,s_{23}=\frac{2}{2}=1$ Why this step? Long vectors make raw dot products large; dividing by $\sqrt{d_k}$ keeps softmax out of its flat, gradient-killing zone.

2. No masking added — encoder sees everything, so $M_{ij}=0$ everywhere. Scores unchanged. Why this step? Cell A is defined by the all-1s mask: nothing is blocked, both past and future keys stay.

3. Exponentiate: $e^1=2.71828$, $e^2=7.38906$, $e^1=2.71828$. Sum $=12.82562$. Why this step? Softmax turns any real numbers into positive weights that sum to $1$.

4. Normalise: ${\alpha }_{21}=\frac{2.71828}{12.82562}=0.21194,{\alpha }_{22}=\frac{7.38906}{12.82562}=0.57612,{\alpha }_{23}=0.21194$

Verify: $0.21194+0.57612+0.21194=1.0000$ ✓. Highest score ($k_2$) got the biggest weight, and the two equal scores got equal weights — matches the forecast.

---

Example 2 — Cell B & E: causal mask with a negative score (decoder)

Forecast: token $4$ has the biggest raw score — will it dominate?

1. Scale by $\sqrt{d_k}=2$: $s=(-1,0,1,3)$ before masking. Why this step? Same health reason as before; note negative scores are perfectly legal — a negative dot product just means the query and key arrows point apart.

2. Apply the causal mask. Token $3$ may only attend to $j\le 3$. So $M_{34}=-\infty$: $s=(-1,0,1,-\infty )$ Why this step? During Autoregressive Generation position $4$ does not exist yet when we compute position $3$; letting it leak would be cheating. This is Attention Masking in action.

3. Exponentiate (and $e^{-\infty }=0$): $e^{-1}=0.36788$, $e^0=1$, $e^1=2.71828$, $e^{-\infty }=0$. Sum $=4.08616$. Why this step? The $-\infty$ is the switch: after exponentiation the blocked token contributes exactly $0$.

4. Normalise over the three allowed tokens: ${\alpha }_{31}=0.09003,{\alpha }_{32}=0.24473,{\alpha }_{33}=0.66524,{\alpha }_{34}=0$

Verify: $0.09003+0.24473+0.66524+0=1.0000$ ✓. Despite token $4$ having the largest raw score, it gets weight $0$ — the mask overruled it. This is exactly Cell B + Cell E (negative scores handled correctly).

---

Example 3 — Cell C: degenerate input, sequence length 1

Forecast: with only one thing to look at, what must the weight be?

1. Scale: $s_{11}=5/\sqrt{1}=5$. Why this step? Consistency — even a lone token goes through the same pipeline.

2. Causal mask: $j=1\le i=1$, allowed. $M_{11}=0$. Why this step? The token attends to itself; nothing to block.

3. Softmax over one element: ${\alpha }_{11}=\frac{e^5}{e^5}=1$ Why this step? A pie with one slice is the whole pie. The value of the score ($5$) is irrelevant — this is the degenerate case.

4. Mix: ${\mathbf{h}}_1={\alpha }_{11}{v}_1=1\cdot [3,-1]=[3,-1]$.

Verify: ${\alpha }_{11}=1$ exactly, independent of the score $5$ ✓. Sanity: a one-token sequence just passes its own value through, which is why the very first generated token depends only on the prompt embedding.

---

Example 4 — Cell D: limiting case of two exactly equal scores

Forecast: ties should split how?

1. Equal case, exponentiate: $e^3=e^3$, sum $=2e^3$. Why this step? Identical inputs to softmax must give identical outputs — softmax is symmetric.

2. Normalise: ${\alpha }_{i1}=\frac{e^3}{2e^3}=0.5,{\alpha }_{i2}=0.5$ Why this step? A perfect tie → an even split. This is the uniform limit.

3. Nudge the second score up by $\epsilon$: ${\alpha }_{i2}=\frac{e^{3+\epsilon }}{e^3+e^{3+\epsilon }}=\frac{e^{\epsilon }}{1+e^{\epsilon }}$ Why this step? We want the behaviour near the tie, not just at it — softmax is smooth, no jump.

4. Take $\epsilon \to 0$: $\frac{e^0}{1+e^0}=\frac{1}{2}=0.5$. Why this step? Confirms the split is continuous: infinitesimally breaking a tie moves the weight infinitesimally, never discontinuously.

Verify: ${\alpha }_{i1}={\alpha }_{i2}=0.5$ and the limit equals $0.5$ ✓. No sharp jump — softmax never teleports probability.

---

Example 5 — Cell H: the exam twist, does $\sqrt{d_k}$ change the answer?

Forecast: does dividing both scores by the same $4$ leave the weights unchanged? (Trap!)

1. Unscaled softmax: exponentiate $e^8=2980.958$, $e^4=54.598$, sum $=3035.556$. ${\alpha }_1^{\text{unscaled}}=0.98201,{\alpha }_2^{\text{unscaled}}=0.01799$ Why this step? This is what happens if you forget the scale — the gap of $4$ becomes enormous after exponentiation.

2. Scaled scores: $8/4=2$, $4/4=1$. Exponentiate $e^2=7.38906$, $e^1=2.71828$, sum $=10.10734$. ${\alpha }_1^{\text{scaled}}=0.73106,{\alpha }_2^{\text{scaled}}=0.26894$ Why this step? Dividing by $\sqrt{d_k}$ shrinks the gap, and softmax is nonlinear — so the weights genuinely change.

3. Compare: unscaled says "listen 98% to key 1"; scaled says "73%". Very different. Why this step? The exam trap is believing softmax is scale-invariant. It is not: $\text{softmax}(cs)\ne \text{softmax}(s)$ for $c\ne 1$.

Verify: $0.98201+0.01799=1$ ✓ and $0.73106+0.26894=1$ ✓, and the two ${\alpha }_1$ values differ ✓. Scaling is not decoration — it controls how sharp the attention is.

---

Example 6 — Cell F: cross-attention, decoder queries the encoder

Forecast: which encoder value dominates the output?

1. Scale by $\sqrt{4}=2$: $s=(3,1,1)$. Why this step? Cross-attention uses the same score→normalise→mix pipeline; only the source of $K,V$ changes (encoder, not decoder).

2. No causal mask here. The decoder may look at all encoder positions (they are the given source, not future output). $M=0$ across the three. Why this step? This is the crucial difference from Cell B: cross-attention over encoder outputs is bidirectional over the source, even inside a causal decoder.

3. Softmax: $e^3=20.0855$, $e^1=2.71828$, $e^1=2.71828$, sum $=25.52210$. ${\alpha }_1=0.78698,{\alpha }_2=0.10651,{\alpha }_3=0.10651$

4. Mix the values: ${\mathbf{h}}_{\text{dec},t}=0.78698[1,0]+0.10651[0,1]+0.10651[0,-1]=[0.78698,0]$ Why this step? The output is a weighted blend of encoder information; here $v_2$ and $v_3$ (opposite arrows) cancel in the second coordinate.

Verify: weights sum to $1$: $0.78698+0.10651+0.10651=1.0000$ ✓. Output second coordinate $=0.10651-0.10651=0$ ✓. Encoder token $1$ dominates as forecast.

---

Example 7 — Cell G: real-world word problem (translation alignment)

Forecast: a French noun should align to the English noun — verify the numbers agree.

1. Softmax the two scores $(0,4)$: $e^0=1$, $e^4=54.598$, sum $=55.598$. ${\alpha }_{\text{the}}=0.01799,{\alpha }_{\text{cat}}=0.98201$ Why this step? This turns the alignment scores into a soft "who does 'chat' translate?" distribution.

2. Interpret: $98.2\%$ of attention lands on "cat". Why this step? Machine-translation cross-attention is soft word alignment — the peak reveals the source word being translated.

3. Mix: $\mathbf{h}=0.01799[1,0]+0.98201[0,1]=[0.01799,0.98201]$. Why this step? The output vector is almost purely "cat"'s information, feeding the head that emits "chat".

Verify: $0.01799+0.98201=1.0000$ ✓; the noun aligns to the noun ✓. Matches human intuition and the parent note's translation example.

---

Example 8 — Cell B (full picture): computing an autoregressive product probability

Forecast: joint probability of three independent-looking factors — bigger or smaller than any single one?

1. Write the chain rule. $P(x_1,x_2,x_3)=P(x_1)P(x_2\mid x_1)P(x_3\mid x_{<3})$ Why this step? Causal Language Modeling decomposes a joint distribution into a product of conditionals — that is precisely why the mask must block the future.

2. Multiply: $0.5\times 0.4\times 0.25=0.05$. Why this step? Probabilities in $[0,1]$ multiplied together shrink — this is why long generations use log-probabilities to avoid underflow.

Verify: $0.5\cdot 0.4\cdot 0.25=0.05$ ✓, and $0.05<$ every factor, as expected for a product of probabilities.

---

Recall Quick self-test

Why does a blocked (future) token get weight exactly 0? ::: We add $-\infty$ to its score; $e^{-\infty }=0$, so its softmax weight is $0$. In a length-1 sequence, what is the self-attention weight? ::: Exactly $1$, regardless of the score — softmax over one element. Does dividing all scores by $\sqrt{d_k}$ leave softmax weights unchanged? ::: No — softmax is nonlinear; scaling changes how sharp the distribution is. In cross-attention inside a causal decoder, can the decoder see all encoder positions? ::: Yes — the causal mask only restricts decoder self-attention, not cross-attention over the source. How do two exactly-equal scores split their weight? ::: Evenly (uniform), e.g. $0.5/0.5$ for a tie of two.