Foundations — Sequence-to-sequence models

This page assumes you have seen nothing. We name every squiggle on the Sequence-to-sequence models page, draw the picture it stands for, and say why the topic cannot live without it. Read top to bottom — each item is a rung, and every rung stands on the one below it.

0. What a "sequence" even is

Before symbols, a mental image.

A sequence is just an ordered list of things where order matters. "I am happy" is a sequence of three words. Reverse it — "happy am I" — and it means something different (or nothing). That "order matters" is the whole reason we need special machinery: a normal calculator adds numbers where order doesn't change the answer, but language does.

1. Subscripts and the time index $t$

The parent writes $x_1,x_2,\dots ,x_{T_x}$. Let's earn every piece.

• The letter $x$ means "an input token."
• The little number below — the subscript — is a position label. $x_1$ = the first token, $x_2$ = the second. It is not multiplication and not a power; it is an address, like seat numbers in a row.
• The parent also uses $t$ as a subscript ($x_t$). Here $t$ stands for time step: we imagine reading the sentence left to right, one token per "tick" of a clock. $t=1$ is the first tick.

2. Vectors, and bold letters like $\mathbf{h}$

A single number can't hold the meaning of a word — "happy" needs many shades (positive? emotional? adjective?). So we use a vector: an ordered list of numbers stacked together, like $[0.1,0.6,-0.3,0.4]$.

• Bold letters (like $\mathbf{h}$, and later $\mathbf{c}$) signal "this is a vector, not a single number."
• Each slot in the vector is a dimension. A vector with 4 slots "lives in 4-dimensional space."

2b. Embeddings — turning a token index into a vector

We just said each token becomes an index (a single whole number like $0,1,2$). But §2 said meaning needs a vector, not one number. How do we bridge the two? With an embedding.

Picture a lookup table with one row per vocabulary word. Row 0 holds a vector for "I", row 1 for "am", and so on. Feeding token index $2$ ("happy") means "go fetch row 2" — out pops a vector.

3. The hidden state $h_t$ — a running memory

Now the star of the show. As the encoder reads token by token, it keeps a running summary of everything seen so far. That summary is a vector called the hidden state, written $h_t$.

Picture a person reading a sentence aloud while jotting notes on a single sticky note. After each word they update the sticky note — old notes plus the new word give new notes. The sticky note is $h_t$.

• $h_0$ = the blank sticky note before reading (all zeros).
• $h_t=\text{LSTM}(x_t,h_{t-1})$ reads: "the new note is made from the new token $x_t$ and the previous note $h_{t-1}$."
• Every hidden state is itself a vector of the same size: $h_t\in {\mathbb{R}}^{d_h}$. So $h_1,h_2,\dots$ all share the same $d_h$ slots — the sticky note never changes size, only its contents.
• The superscript in $h_t^{(enc)}$ vs $h_t^{(dec)}$ just says which machine's note it is — encoder or decoder. It's a name tag, not math.

The names $\text{LSTM}$ and $\text{GRU}$ are two specific recipes for this update-the-sticky-note step, designed so early words aren't forgotten too fast. You can treat them as black boxes here; the details live in LSTM networks and GRU networks.

4. The context vector $\mathbf{c}$

After the encoder reads the last token, its final sticky note $h_{T_x}^{(enc)}$ has "seen" the whole sentence. We give it a new name:

$\mathbf{c}=h_{T_x}^{(enc)}$

That's it — $\mathbf{c}$ (for context, or "the thought") is simply the last hidden state, promoted to the role of "the compressed meaning of the entire input." It's the single vector that gets handed from the reader machine to the writer machine. Because it is a hidden state, $\mathbf{c}\in {\mathbb{R}}^{d_h}$ too — same $d_h$ slots as every $h_t$.

4b. The decoder recurrence — how the writer machine steps

The encoder's job is done: it handed over $\mathbf{c}$. Now the decoder — a second LSTM — writes the output one token at a time, keeping its own running memory $h_t^{(dec)}$.

Two things must happen to start the writer:

• Initialize its memory from the meaning vector. We set the decoder's very first hidden state to the context: $h_0^{(dec)}=\mathbf{c}.$ This is literally "load the thought into the writer's head before it speaks."
• Give it a starting word. It has no real previous word yet, so we feed a special "start" token as $y_0$ — the <SOS> token defined just below.

Then, at each output step $t$, the decoder updates its memory from its previous memory and the previous output token:

$h_t^{(dec)}=\text{LSTM}(E[y_{t-1}],h_{t-1}^{(dec)})$

Read it exactly like the encoder recurrence, but the "input token" is now the previous output word $y_{t-1}$ (looked up through the embedding table $E$), and the memory it builds on is the decoder's own $h_{t-1}^{(dec)}$.

5. Probability $P(\cdot \mid \cdot )$ and the softmax

The decoder must choose the next word. It never says "the word is X" flatly — it gives a probability to every possible word.

To turn raw scores into probabilities we need a gadget that (a) makes everything positive and (b) makes the total add up to 1. That gadget is the softmax.

• $V$ = the vocabulary, the full list of possible tokens. $|V|$ = how many there are.
• $W_o$ and $b_o$ (the parent's $\text{softmax}(W_o{h}_t^{(dec)}+b_o)$) are just learned numbers that reshape the hidden vector into one score per vocabulary word. $W_o$ is a matrix ("weights"), $b_o$ a vector ("bias") — think of them as a tunable lens.

6. Product $\prod$, logarithm, and the loss $\mathcal{L}$

A whole sentence's probability is each word's probability multiplied together:

$P(y_1,\dots ,y_{T_y}\mid x)={\prod }_{t=1}^{T_y}P(y_t\mid y_{<t},\mathbf{c})$

Multiplying many small probabilities gives a tiny number that computers round to zero. The fix: take the logarithm.

Applying it to the whole-sentence product. Take $\log$ of both sides of the product above. By the rule "$\log$ of a product = sum of the logs," applied $T_y$ times:

$\log {\prod }_{t=1}^{T_y}P(y_t\mid y_{<t},\mathbf{c})={\sum }_{t=1}^{T_y}\log P(y_t\mid y_{<t},\mathbf{c})$

7. The special tokens <SOS> and <EOS> — and when to stop

We met <SOS> in §4b as the "go" signal. Its partner answers a different question: how does the decoder know when to stop writing? A sentence has no fixed length, so we need an explicit "I'm done" signal.

8. $\arg \max$ — picking the winner

Greedy decoding uses: $y_t=\arg {\max }_{v}P(v\mid y_{<t},\mathbf{c})$

Considering several candidate sentences at once instead of one greedy pick is beam search.

Prerequisite map

The arrows say "you must understand the tail before the head." Notice every path funnels into the seq2seq node, and it in turn points onward to attention — the very next topic. Backprop through both machines is backpropagation through time; the full modern successor is Transformers.

Equipment checklist

Reveal each line only after you can answer it in your own words.