Foundations — Encoder-decoder architecture

This page assumes nothing. Before you can read the parent note, you need to be fluent in a handful of symbols and pictures. We build each one from scratch, in an order where every item leans on the one before it.

---

0. What is a "sequence"? — the picture behind everything

Before any math, look at what we are even feeding the machine.

Figure 1 — Two rows of boxes. The cyan row (top) is the input sequence "I love AI", labelled $x_1,x_2,x_3$. The amber row (bottom) is the output "J'aime l'IA", labelled $y_1,y_2$. What to observe: three boxes go in, two come out — the white arrow and caption "3 in → 2 out" show that the two rows have different lengths. That mismatch is the whole reason this architecture exists.

A sequence is just an ordered list of things. "I love AI" is a sequence of three words. Order matters: "AI love I" is a different sequence even though it uses the same words.

WHY the topic needs this: the whole reason encoder-decoders exist is that the input list and the output list can have different lengths ($T_x\ne T_y$). Translation turns a 3-word English sequence into a 2-word French one. No fixed-size machine could do that directly — hence two machines.

---

1. Subscripts and time steps: $x_1,x_2,\dots ,x_{T_x}$

The parent note writes $\mathbf{x}=(x_1,x_2,\dots ,x_{T_x})$. Let's earn every piece.

• $x$ = "an input token."
• The little number below, the subscript, says which position in the line. $x_1$ is the first token, $x_2$ the second.
• $x_t$ with a letter $t$ means "the token at some general position $t$" — a placeholder so we can write one rule that works for every position.
• $x_{T_x}$ is the last input token, because $T_x$ is the total count.

---

2. Embedding: $\text{emb}(\text{"I"})$ — turning a word into numbers

A neural network cannot add up the letters "l-o-v-e." It only does arithmetic on numbers. So each token must first become a list of numbers, called a vector.

Figure 2 — Word to number-column. On the left, the cyan word "love"; the amber arrow labelled emb( ) feeds it into the white column of four numbers on the right. What to observe: the word does not stay a word — it becomes a fixed list of numbers (here ${\mathbb{R}}^4$; in real code ${\mathbb{R}}^{512}$). This is the only form the network's arithmetic can touch, which is why every token must pass through this arrow first.

WHY the topic needs this: when the parent writes "$\mathbf{c}\in {\mathbb{R}}^{512}$", the bold letter and the ${\mathbb{R}}^{512}$ are saying "$\mathbf{c}$ is a vector of 512 real numbers." You need the vector idea to even read that line.

---

3. The hidden state $h_t$ and the recurrent function $f$

This is the heart of the encoder. The parent writes:

$h_t^{enc}=f_{enc}(x_t,h_{t-1}^{enc})$

Read it as a sentence: "the new memory ($h_t$) is made by a function $f$ that looks at the current token ($x_t$) and the old memory ($h_{t-1}$)."

• $h$ = hidden state = the machine's running memory / notepad.
• $h_{t-1}$ = the memory one step ago (subtract 1 from the seat number → the previous seat).
• $f$ = the fixed rule that mixes "what I just read" with "what I remembered." The same $f$ is reused at every step (that is what "recurrent" means — the loop reuses one rule).

Figure 3 — The recurrence unrolled. Read it left to right. The amber boxes at the bottom are the tokens "I", "love", "AI"; each feeds upward (white arrow) into a cyan memory circle $h_1,h_2,h_3$. The horizontal white arrows carry the old memory forward into the next circle — that is the "same rule $f$ reused each step" (noted top-left). What to observe: the last circle $h_3$ is pulled out (amber arrow) and renamed $\mathbf{c}$, the context pill. Nothing new is invented for $\mathbf{c}$; it is literally the final memory.

WHY the topic needs this: the context vector $\mathbf{c}$ is nothing but this final snowball, $\mathbf{c}=h_{T_x}^{enc}$. The whole "squeeze the sentence into one pill" idea is this recursion run to the end.

The actual $f$ can be a plain RNN cell, an LSTM, or a GRU — the parent note lets you swap in any of these. They differ only in how carefully they decide what to keep and what to forget.

---

4. tanh: the squasher inside $f$

Example 3 of the parent uses:

$h_2^{dec}=\tanh (W_h{h}_1^{dec}+W_{hy}\text{emb}(\text{"I"})+b_h)$

Two new pieces here.

$W$ and $b$ — a weight matrix $W$ multiplies a vector to reshape/rescale it; a bias $b$ is a vector added on. Together $Wv+b$ is the network's adjustable "knobs" (the parameters it learns). Different subscripts ($W_h$, $W_{hy}$, $W_o$) are just different knob-sets for different jobs.

$\tanh$ — the hyperbolic tangent. It is a function that takes any number and gently squashes it into the range $(-1,1)$.

Figure 4 — The tanh curve. The horizontal axis is the raw input $z$ (any number); the vertical axis is the output. What to observe: the cyan S-curve is nearly a straight line through the origin (small signals pass through almost unchanged) but flattens toward the two amber dashed lines at $+1$ and $-1$ (huge signals get clamped). No input, however large, escapes the band $(-1,1)$ — that is exactly why the memory cannot blow up.

---

5. From scores to probabilities: exp and softmax

The decoder ends with:

$P(y_t=v\mid \dots )=\frac{\exp (z_v)}{{\sum }_{u\in V}\exp (z_u)}$

Let's earn each symbol.

• $z$ = logits = raw, unbounded scores the network outputs, one per possible word in the vocabulary $V$ (the full list of words the model knows). $z_v$ is the score for word $v$.
• $\exp (z)$ means $e^z$, the exponential. Why exponentiate? Because $\exp$ turns any number (even negative) into a positive one, and it stretches gaps: a slightly higher score becomes a much higher weight. This lets the model express confident preferences.
• ${\sum }_{u\in V}$ = "add up over every word $u$ in the vocabulary." The big $\Sigma$ is just a compact "add all of these."
• Dividing each $\exp (z_v)$ by that total sum forces all the values to add up to exactly $1$.

$P(y_t\mid y_{<t},\mathbf{x})$ in words: "the probability of the next output token $y_t$, given ($\mid$) all the tokens generated before it ($y_{<t}$, meaning $y_1,\dots ,y_{t-1}$) and the input $\mathbf{x}$."

---

6. Products, logs, and the loss $\mathcal{L}$

The training objective chains all those per-step probabilities together.

The product $\prod$: the expression ${\prod }_{t=1}^{T_y}P(y_t^{\ast }\mid y_{<t}^{\ast },\mathbf{x})$ means "multiply these probabilities for every step $t$ from $1$ to $T_y$." The star superscript $y_t^{\ast }$ marks the ground-truth correct token — the answer key we are trying to match (it is not an exponent). We multiply because the probability of the whole correct sentence is the chance of getting step 1 right and step 2 right and so on.

Why the log? Multiplying many numbers each less than $1$ gives a tiny number the computer can't store safely (underflow). The logarithm $\log$ turns a product into a sum:

$\log \left({\prod }_t{P}_t\right)={\sum }_t\log P_t$

Sums are stable and easy to differentiate. That single identity is why the parent's loss suddenly switches from $\prod$ to $\sum$.

The minus sign and $\frac{1}{N}$: training minimizes a cost, but we want to maximize likelihood — so we flip the sign (maximize $L$ = minimize $-L$) and divide by the number of examples $N$ to get an average.

$\mathcal{L}(\theta )=-\frac{1}{N}{\sum }_{i=1}^N{\sum }_{t=1}^{T_y^{(i)}}\log P_{\theta }\left(y_t^{(i)}\mid y_{<t}^{(i)},{\mathbf{x}}^{(i)}\right)$

Here $\theta$ (theta) is the bundle of all the $W$'s and $b$'s — every knob in both networks. Training = nudging $\theta$ to make this number small. The nudging itself is done by backpropagation through time.

---

7. Autoregressive generation and decoding search

The decoder feeds its own last output back in as the next input — this is autoregressive ("self" + "regress on past outputs"). At each step it faces a choice of which word to emit. Two strategies appear in the parent:

• Greedy: always take the single highest-probability word. Fast, but can paint itself into a corner.
• Beam search: keep the best few partial sentences alive at once. See beam search for the full mechanism.

---

Prerequisite map

---

Equipment checklist

Cover the answers and test yourself; if any line is fuzzy, reread its section.