T5 and text-to-text framework: Foundations — AI-ML deep dive

Before you can trust that idea, you need to own a handful of symbols and pictures the parent note throws at you. This page builds each one from nothing, in an order where each rests on the one before it.

0. What is a "token"? (the atom of everything)

Every symbol on this page is ultimately about tokens, so we start here.

The fixed list of all possible tokens is called the vocabulary.

Look at the figure: the sentence gets chopped into boxes (tokens), and each box points to its ID number in the vocabulary list on the right. That arrow — text → list of IDs — is the very first thing that happens to any input.

1. The sequences $x$ and $y$

Now that a token is defined, two whole sequences of them appear constantly in the parent note.

Notice $n$ and $m$ can differ — the answer is usually a different length from the question. That is exactly why T5 is text-to-text: both the question and the answer are just sequences of tokens.

2. The notation $y_{< t}$ (everything before now)

The parent note writes $p (y_{t} ∣ y_{< t}, x)$ . That little $y_{< t}$ is a compact and crucial idea.

The figure shows the decoder writing token by token. The red box is the token being written right now ( $y_{t}$ ); the black boxes to its left are $y_{< t}$ , the history it is allowed to look at. It may never peek at boxes to its right — those aren't written yet.

3. Probability $p (\cdot)$ and the conditional bar $∣$

Two output tokens are possible; which does the model pick? It assigns each a probability.

4. The product $\prod$ and the chain rule

The parent note writes $p (y ∣ x) = \prod_{t = 1}^{m} p (y_{t} ∣ y_{< t}, x)$ . Unpack that big $\prod$ .

5. The logarithm $lo g$ and why we take it

Multiplying many tiny probabilities gives a vanishingly small number that computers handle badly. Enter the log.

The figure shows the curve of $lo g p$ for $p$ between $0$ and $1$ . Notice: when $p$ is close to $1$ (confident and correct), $lo g p$ is near $0$ . When $p$ is tiny (the model was almost sure it was something else), $lo g p$ plunges toward $- \infty$ . So $- lo g p$ is small when you're right and huge when you're wrong — a perfect "surprise / penalty" meter.

6. Cross-entropy: the single loss

Now every piece assembles into the one objective the parent note lives by.

See Cross-entropy loss for this same object studied on its own.

7. Encoder, decoder, and cross-attention (the model's two halves)

The parent says T5 is an encoder–decoder. Here is what those words picture.

In the figure the left block (encoder) has arrows going both ways between input tokens — it reads bidirectionally. The right block (decoder) has arrows only pointing forward, and the red arrows are cross-attention: the bridge letting the output look at the input.

8. Task prefix and sentinel tokens (special strings)

Two more pieces of vocabulary the parent leans on.

Prerequisite map

Everything funnels into the parent topic T5 and the text-to-text framework. The loss branch (left) explains how it learns; the architecture and string branches explain what it is and how you talk to it. Related build-ups: Transfer learning and fine-tuning.

Equipment checklist

I can say what a token and the vocabulary $V$ are ::: A token is one small text chunk (word/subword); $V$ is the finite set of all tokens the model can read or write.
I can read $x = (x_{1}, \dots, x_{n})$ and $y = (y_{1}, \dots, y_{m})$ ::: $x$ is the input token sequence of length $n$ ; $y$ is the target token sequence of length $m$ ; the two lengths may differ.
I know what $y_{< t}$ means ::: All target tokens before position $t$ , i.e. $(y_{1}, \dots, y_{t - 1})$ — the output written so far.
I can read the bar $∣$ ::: $p (A ∣ B)$ = probability of $A$ given that $B$ holds; it means "the answer depends on this context."
I know why the chain rule uses $\prod$ ::: The joint probability of a sequence factorizes exactly into a product of per-token conditional probabilities.
I know why $lo g$ appears ::: It turns the product of tiny probabilities into an additive, numerically stable sum.
I know why the loss has a minus sign ::: $lo g p$ is negative for $p < 1$ ; the minus makes the penalty positive, small when correct, huge when wrong.
I can write the cross-entropy loss ::: $L = - \sum_{t = 1}^{m} lo g p (y_{t} ∣ y_{< t}, x)$ .
I can distinguish encoder, decoder, cross-attention ::: Encoder reads input bidirectionally; decoder writes output left-to-right; cross-attention lets each output token consult the encoder's reading.
I know what a task prefix and a sentinel token are ::: A prefix is the instruction glued to the front of the input; a sentinel is a special labelled blank token marking a corrupted span.