Encoder vs decoder vs encoder-decoder: Exercises

This page is a graded ladder. Each rung is harder than the last, and every problem carries a full worked solution hidden inside a collapsible callout so you can quiz yourself. Try each problem cold, then open the solution.

Before we start, one promise: every symbol used below is defined the moment it appears. If a term feels new, it was built for you in the parent note — this page assumes only that you have read it.

Figure — Encoder vs decoder vs encoder-decoder

Above: the mask grid. A row $i$ is the token doing the looking (the query). A column $j$ is the token being looked at (the key). A filled cell $(i, j)$ means "token $i$ is allowed to attend to token $j$ ". Keep this picture beside you for the whole page.

Level 1 — Recognition

Exercise 1.1

For a 4-token sequence, an encoder uses a bidirectional mask (all cells filled) and a decoder uses a causal mask (a cell $(i, j)$ is filled only when $j \leq i$ ). How many filled cells does each mask have? What fraction of the encoder's cells does the decoder keep?

Recall Solution 1.1

The grid is $n \times n$ with $n = 4$ , so $4 \times 4 = 16$ cells total.

Encoder (bidirectional): every cell filled $\Rightarrow 16$ cells.
Decoder (causal): cell $(i, j)$ filled when $j \leq i$ . Row $i$ has exactly $i$ filled cells (it can see itself and everything before). So the count is $1 + 2 + 3 + 4 = 10.$ That is the triangular number $\frac{n ( n + 1 )}{2} = \frac{4 \cdot 5}{2} = 10$ .
Fraction kept: $\frac{10}{16} = 0.625 = 62.5%$ .

Look at the right panel of the figure: the decoder mask is exactly the lower triangle (including the diagonal) of the full square.

Exercise 1.2

Match each model to its architecture: BERT, GPT, T5.

Recall Solution 1.2

BERT → encoder-only (bidirectional; reads and understands).
GPT → decoder-only (causal; generates left-to-right).
T5 → encoder-decoder (reads a source two-way, writes a target one-way).

Level 2 — Application

Exercise 2.1

A decoder computes raw attention scores $s_{ij} = q_{i} \cdot k_{j} / d_{k}$ for query token $i = 2$ (the third token, 0-indexed... here we use 1-indexed, so $i = 2$ is the second token) over a 3-token sequence. The scaled scores are $s_{2, 1} = 1.0, s_{2, 2} = 2.0, s_{2, 3} = 3.0.$ Apply the causal mask by adding $M_{2, j}$ (where $M = 0$ if $j \leq i$ , else $- \infty$ ), then compute the attention weights $α_{2, j}$ via softmax. Give the numbers.

Recall Solution 2.1

Step 1 — what the mask does. For query $i = 2$ , cell $j = 3$ is a future token ( $3 > 2$ ), so $M_{2, 3} = - \infty$ . Cells $j = 1, 2$ are allowed, $M = 0$ .

Masked scores: $s_{2, 1}^{'} = 1.0, s_{2, 2}^{'} = 2.0, s_{2, 3}^{'} = 1.0 + (- \infty) = - \infty$ .

Step 2 — why $- \infty$ . Softmax exponentiates, and $e^{- \infty} = 0$ . So the future token gets weight exactly zero and drops out — the query never leaks information from the future.

Step 3 — softmax over the survivors. $α_{2, j} = \frac{e ^{s_{2, j}^{'}}}{e ^{1.0} + e ^{2.0}} (the e^{- \infty} term is 0) .$ $e^{1.0} = 2.71828, e^{2.0} = 7.38906, sum = 10.10734.$ $α_{2, 1} = \frac{2.71828}{10.10734} = 0.26894, α_{2, 2} = \frac{7.38906}{10.10734} = 0.73106, α_{2, 3} = 0.$ The weights over the visible tokens sum to $1$ , and the future token is silenced.

Exercise 2.2

An encoder (bidirectional) sees the same three scaled scores for query $i = 2$ : $s_{2, 1} = 1.0, s_{2, 2} = 2.0, s_{2, 3} = 3.0$ , no mask. Compute $α_{2, j}$ . Compare $α_{2, 1}$ to the decoder's answer above and explain the difference in one sentence.

Recall Solution 2.2

No mask, so all three compete: $e^{1.0} = 2.71828, e^{2.0} = 7.38906, e^{3.0} = 20.08554, sum = 30.19288.$ $α_{2, 1} = \frac{2.71828}{30.19288} = 0.09003, α_{2, 2} = \frac{7.38906}{30.19288} = 0.24473, α_{2, 3} = \frac{20.08554}{30.19288} = 0.66524.$ Comparison: the encoder gives token 1 only $0.090$ , while the decoder gave it $0.269$ . Adding the strong future token 3 into the denominator steals probability mass from token 1 — the decoder, blind to the future, hands that mass back to the past.

Level 3 — Analysis

Exercise 3.1

In an encoder-decoder translating a source of length $n = 5$ into a target of length $m = 4$ , count the attention connections of each type when computing the whole forward pass once (ignore the $d_{k}$ constant, count query–key pairs):

Encoder self-attention pairs.
Decoder causal self-attention pairs.
Cross-attention pairs (decoder queries over encoder keys).

Recall Solution 3.1

1. Encoder self-attention (Self-Attention Mechanism, bidirectional): every source token attends to every source token $\Rightarrow n \times n = 5 \times 5 = 25$ pairs.

2. Decoder causal self-attention: target token $t$ sees only $y_{\leq t}$ , so it's the triangular count over $m$ : $\frac{m ( m + 1 )}{2} = \frac{4 \cdot 5}{2} = 10$ pairs.

3. Cross-attention (Cross-Attention): each of the $m$ decoder queries attends to all $n$ encoder keys — no mask here, because the entire source is known before generation begins. So $m \times n = 4 \times 5 = 20$ pairs.

Total: $25 + 10 + 20 = 55$ attention pairs.

Key insight: cross-attention is a full rectangle ( $m \times n$ ), never triangular — the source has no "future" to hide from the target.

Exercise 3.2

Explain, using the mask picture, why a decoder-only model can be trained on a whole sentence in one forward pass (all positions at once) even though at inference it generates one token at a time.

Recall Solution 3.2

Training: you already have the full target sentence. Feed it in. The causal mask guarantees that the prediction at position $t$ is computed using only positions $1.. t$ — the mask physically blocks positions $t + 1.. n$ . So all $n$ next-token predictions are computed simultaneously and independently, yet each obeys the "no peeking ahead" rule. This is Causal Language Modeling: one pass, $n$ supervised predictions.

Inference: you do not have the future tokens — they are what you are trying to produce. So you must loop: predict $x_{t}$ , append it, predict $x_{t + 1}$ , and so on. This is Autoregressive Generation.

The mask is what makes these two regimes consistent: training with the mask simulates the information-starvation of inference, so the model never learns to depend on tokens it won't have.

Level 4 — Synthesis

Exercise 4.1

You must build a system that reads a legal contract and answers "Is clause 7 enforceable? (yes/no)". The answer is always contained in the input. Which of the three architectures is most efficient, and why is a decoder-only model wasteful here?

Recall Solution 4.1

Choice: encoder-only (BERT-style).

Why. The task is understanding, not generating — the output is a single label, and the evidence lives entirely inside the input. Bidirectional attention lets "clause 7" attend simultaneously to text before and after it, which matters because enforceability may hinge on a definition that appears later in the contract. A Masked Language Modeling-pretrained encoder is designed exactly for this two-way evidence gathering.

Why a decoder is wasteful. A decoder-only model would read the contract left-to-right under a causal mask, so early tokens could never see later clauses — it would have to re-encode the relevant context by generating it, spending compute on autoregressive generation you don't need. You'd pay for a generation machine to do a classification job. Even if it worked, throwing away the future context is a strict information handicap.

Exercise 4.2

Design a training recipe (in words) for an English→French translator using an encoder-decoder, and state exactly which mask each of the three attention blocks uses.

Recall Solution 4.2

Data: aligned pairs (English source $x$ , French target $y$ ).

Forward pass, per block:

Encoder self-attention — mask = all 1s (bidirectional). The full English sentence is known, so every source word gets full context. Uses Positional Encoding so word order survives.
Decoder self-attention — mask = lower-triangular causal. The French target is fed in shifted-right (teacher forcing), and position $t$ may see only $y_{< t}$ , matching inference.
Cross-attention — mask = all 1s ( $m \times n$ rectangle). Each French query reads the entire encoded English.

Loss: cross-entropy of $P (y_{t} ∣ y_{< t}, x)$ summed over all target positions, computed in a single pass thanks to the causal mask. This is the seq2seq objective. Multi-Head Attention runs all of this in parallel heads so different heads can specialise (e.g. one head aligns articles, another aligns verbs).

Level 5 — Mastery

Exercise 5.1

A decoder generates over a vocabulary and outputs, for a 3-token sequence, the conditional probabilities $P (x_{1}) = 0.5, P (x_{2} ∣ x_{1}) = 0.4, P (x_{3} ∣ x_{1}, x_{2}) = 0.25.$ Compute the full joint probability $P (x_{1}, x_{2}, x_{3})$ and its per-token perplexity. (Perplexity $= exp (- \frac{1}{n} \sum_{t} ln P (x_{t} ∣ x_{< t}))$ ; it measures "on average, how many equally-likely choices the model felt it faced" — lower is more confident.)

Recall Solution 5.1

Step 1 — joint via chain rule. A decoder factorises the sequence probability as a product (Causal Language Modeling): $P (x_{1}, x_{2}, x_{3}) = P (x_{1}) P (x_{2} ∣ x_{1}) P (x_{3} ∣ x_{1}, x_{2}) = 0.5 \times 0.4 \times 0.25 = 0.05.$ Why a product? The causal mask means each factor conditions only on the past, and the chain rule of probability $P (A, B, C) = P (A) P (B ∣ A) P (C ∣ A, B)$ turns the joint into exactly this ordered product.

Step 2 — average log-likelihood. $\frac{1}{3} (ln 0.5 + ln 0.4 + ln 0.25) = \frac{1}{3} (- 0.69315 - 0.91629 - 1.38629) = \frac{- 2.99573}{3} = - 0.99858.$ Step 3 — perplexity. $PPL = exp (0.99858) = 2.71412.$ So on average the model behaved as if choosing among about $2.71$ equally likely tokens per step.

Exercise 5.2

Two heads in a Multi-Head Attention cross-attention layer produce, for one French query token, these attention weight vectors over 4 English source tokens:

Head A: $[0.7, 0.1, 0.1, 0.1]$
Head B: $[0.25, 0.25, 0.25, 0.25]$

Compute the Shannon entropy $H = - \sum_{i} p_{i} lo g_{2} p_{i}$ (bits) of each head's distribution and interpret: which head is "sharply aligned" to one source word and which is "spread out"?

Recall Solution 5.2

Entropy measures how concentrated a distribution is: $0$ bits = all mass on one option, $lo g_{2} 4 = 2$ bits = perfectly uniform over 4.

Head A: $H_{A} = - (0.7 lo g_{2} 0.7 + 3 \times 0.1 lo g_{2} 0.1) .$ $lo g_{2} 0.7 = - 0.51457$ , so $0.7 \times (- 0.51457) = - 0.36020$ . $lo g_{2} 0.1 = - 3.32193$ , so $0.1 \times (- 3.32193) = - 0.33219$ , times 3 $= - 0.99658$ . $H_{A} = - (- 0.36020 - 0.99658) = 1.35678 bits.$

Head B (uniform): $H_{B} = - 4 \times (0.25 lo g_{2} 0.25) = - 4 \times (0.25 \times - 2) = 2.0 bits.$

Interpretation: Head B is at the maximum ( $2$ bits) — perfectly spread, no alignment preference. Head A ( $1.357$ bits) is much lower — it has locked onto the first English word (weight $0.7$ ), a sharp alignment typical of a head that has learned word-to-word correspondence. Different heads specialising this way is precisely why multi-head attention beats a single averaged head.

Recall Quick self-test (reveal answers)

Encoder mask on 4 tokens has how many filled cells? ::: 16 (all of $4 \times 4$ ). Decoder mask on 4 tokens has how many filled cells? ::: 10 (the triangular number $4 \cdot 5/2$ ). Cross-attention on source $n = 5$ , target $m = 4$ has how many pairs? ::: 20 (a full $m \times n$ rectangle). Which mask does cross-attention use? ::: All 1s — the whole source is known, nothing to hide. Joint prob of $0.5, 0.4, 0.25$ conditionals? ::: $0.05$ (chain-rule product).