T5 and text-to-text framework: Visual walkthrough

Step 1 — A sentence is a list of numbered slots

WHAT. Before any maths, we need to agree what "text" even is to a computer. A piece of text is chopped into small pieces called tokens (roughly: word-chunks). We line them up and number them left to right.

WHY. Everything T5 does is "read a string, write a string." If we cannot describe a string precisely, we cannot describe the model. So the very first tool we introduce is just indexing — giving each slot a position number so we can talk about "the 3rd token."

PICTURE. In the figure, the input string sits in the top row of boxes labelled $x_{1}, x_{2}, \dots, x_{n}$ (the amber row). The output string sits below as $y_{1}, y_{2}, \dots, y_{m}$ (the cyan row).

Notice $n$ and $m$ can differ — the input "translate English to German: dog" is long, the output "Hund" is short. That is fine; strings have no fixed length.

Step 2 — Every task becomes the same pair of rows

WHAT. We take four totally different jobs — translation, grammar-checking, similarity-scoring, summarising — and write each one as a top row (input) and a bottom row (target).

WHY. This is the whole idea of T5, and we want to see it before we prove anything. If all four jobs look identical on paper (two rows of boxes), then whatever formula scores one row can score them all.

PICTURE. Four stacked pairs of rows. On the left, the top row always starts with a task prefix (highlighted amber) telling the model which job it is. On the right, the bottom row is the answer — and crucially, even the number 3.8 and the label positive are just boxes of tokens, no different in shape from Hund.

The key visual takeaway: the target row is always just text. The label "positive" is tokens. The number "3.8" is tokens 3, ., 8. So if we can score "how good is a predicted target row," we have scored every task at once.

Step 3 — Reading a string one box at a time (the chain rule)

WHAT. We want a single number: how likely is this exact target row $y$ , given the input row $x$ ? Call it $p (y ∣ x)$ (read: "probability of $y$ given $x$ "). We build it by predicting one box at a time, left to right.

WHY THIS TOOL. Why not predict the whole sentence in one shot? Because the number of possible sentences is astronomically large — you could never list them. Instead we use the chain rule of probability, a fact that is always exactly true, with no independence assumption whatsoever: the joint probability of a sequence equals the product of "each next box, given all the boxes before it." The conditioning on earlier tokens is precisely what lets the factors capture dependence between tokens. This turns one impossible question ("which of infinitely many sentences?") into $m$ small, answerable questions ("which single token comes next?").

PICTURE. The decoder is drawn as a machine that, at each step $t$ , looks at (a) the full input row $x$ via cross-attention (the long cyan arrows going up to the encoder), and (b) the target boxes it has already committed to, $y_{1} \dots y_{t - 1}$ (the short cyan arrows). It then outputs a probability for the next box $y_{t}$ .

Formula Factorising the sentence probability

$p (y ∣ x) = \prod_{t = 1}^{m} p (y_{t} ∣ y_{< t}, x)$ Reading it term by term, right where each symbol sits:

$\prod_{t = 1}^{m}$ — "multiply the following, once for each output slot $t$ from $1$ up to $m$ ." This product is the chain rule of probability — it is an exact identity that holds for any sequence; it does not assume the tokens are independent. The dependence is carried inside each factor by the conditioning $y_{< t}$ .
$p (y_{t} ∣ \dots)$ — the probability the model assigns to the actual correct token sitting in slot $t$ .
$y_{< t}$ — shorthand for "all target tokens before slot $t$ ", i.e. $y_{1}, \dots, y_{t - 1}$ . Because each factor conditions on this whole history, the product needs no independence assumption — earlier tokens genuinely influence later ones.
$x$ — the whole input row, always fully visible (the encoder read it bidirectionally). This visibility of all of $x$ is exactly what the encoder–decoder split buys you.

Each factor $p (y_{t} ∣ y_{< t}, x)$ is produced by one softmax over the vocabulary — a recipe that turns raw scores into probabilities.

Definition Softmax — how each factor becomes a probability

The decoder first outputs a raw score (a "logit") $z_{k}$ for every token $k$ in the vocabulary. Let the vocabulary have $V$ tokens, so $k$ and $j$ below both range over all $V$ vocabulary entries, $j = 1, 2, \dots, V$ . Softmax squashes those scores into a probability distribution: $p_{k} = \frac{e x p ( z _{k} )}{\sum _{j = 1}^{V} e x p ( z _{j} )}$

$z_{k}$ — the raw, unbounded score for candidate token $k$ (one of the $V$ vocabulary tokens).
$exp (z_{k})$ — makes every score positive (so it can be a probability) and amplifies the largest.
$\sum_{j = 1}^{V} exp (z_{j})$ — the total over every candidate token $j$ in the whole vocabulary ( $j$ runs from $1$ to $V$ ); dividing by it forces the numbers to sum to 1.

So $p (y_{t} ∣ y_{< t}, x)$ is just $p_{k}$ evaluated at $k =$ the correct token. That is the concrete number Step 4 takes the log of.

Step 4 — Turning "probability of the row" into a loss

WHAT. We now convert $p (y ∣ x)$ (bigger = better) into a loss (smaller = better) that training can push downhill.

WHY LOG, WHY MINUS. Two problems with the raw product from Step 3:

Multiplying $m$ numbers each below 1 gives a tiny value (e.g. $0. 3^{40}$ ) — the computer underflows to zero. The fix is the logarithm, because $lo g (ab) = lo g a + lo g b$ turns the fragile product into a robust sum.
Probabilities live in $[0, 1]$ , so their log is $\leq 0$ (negative). We want a loss we minimise (a positive "badness"), so we flip the sign with a minus.

Maximising $p (y ∣ x)$ is exactly the same as minimising $- lo g p (y ∣ x)$ — $lo g$ is increasing, so it preserves the ranking; the minus just flips "up is good" into "down is good."

PICTURE. Left panel: the raw product shrinking to a near-zero sliver. Right panel: after $- lo g$ , each token contributes a positive bar you add up; a confident-and-correct token gives a short bar (small loss), a surprised token gives a tall bar (big loss).

Step 5 — Teacher forcing: why we feed the true previous tokens

WHAT. During training, at slot $t$ the decoder needs to see $y_{< t}$ . We feed it the ground-truth previous tokens, not the model's own guesses. This is called teacher forcing.

WHY. If early in training the model guessed slot 1 wrong, and we fed that wrong guess into slot 2, the error would snowball and no slot would ever get a clean learning signal. By feeding the true prefix, every slot is judged fairly ("given the correct history, did you predict the next token?"). It also lets us compute all $m$ factors in parallel — we already know the whole target row, so we can score every slot at once.

PICTURE. Two lanes. Top lane "teacher forcing (training)": the true target row is shifted right by one and fed back in as the decoder's input; green check-arrows show each slot getting the real history. Bottom lane "free-running (inference)": the model's own output is fed back — one wrong box can drift the rest. The contrast is the point.

Step 6 — The collapse: one loss, every task

WHAT. Now we cash in. We drop three different jobs into the exact same loss from Step 4 and watch nothing change.

WHY. This is the parent note's central result — "why can one loss train all tasks?" We prove it by substitution: since every target $y$ is a string of tokens (Step 2), and $L$ only cares about "the tokens of $y$ " (Step 4), the machinery is identical regardless of what the string means.

PICTURE. Three funnels — a translation target Hund, a classification target positive, a regression target 3.8 — all pouring into one shared box labelled $L = - \sum lo g p (y_{t} ∣ y_{< t}, x)$ . No per-task head, no per-task loss. One pipe out.

Step 7 — Edge & degenerate cases (the ones that could bite you)

WHAT. We check the corners the smooth story skipped: a one-token target, the empty target, an off-vocabulary answer, and the pretraining (span-corruption) case.

WHY. The contract: the reader must never meet a scenario we did not show. Losses and formulas have to survive $m = 1$ , a zero-length target, weird outputs, and the training-time span-corruption objective.

PICTURE. Four small panels, each a mini row-pair with a note on what the loss does.

Definition Corner case 2 — the "empty" target (

$m = 0$ )? Could a target be genuinely empty, making the sum $\sum_{t = 1}^{m}$ have no terms (an undefined, empty sum)? No — by construction it cannot. Every T5 target string is terminated by a special end-of-sequence token written </s>. So even the shortest possible answer — an "empty" string — is emitted as the single row $y = (‘</s>’)$ , which has $m = 1$ , not $m = 0$ . Consequences:

Every target sequence contains at least the </s> token, so $m \geq 1$ always.
The sum $\sum_{t = 1}^{m}$ therefore always has at least one term — it is never the empty (undefined) sum, and $L$ is always well-defined. No divide-by-nothing, no ambiguity.
Because </s> is a real token in $y$ , "when do I stop writing?" is itself scored by the same $- lo g p$ . Sequence length is learned, not hard-coded.

Definition Corner case 3 — an off-vocabulary answer

What if the correct target contains some exotic string the model has never seen as a word? Recall Step 1: T5 uses a subword vocabulary of $V$ fixed pieces, and any string decomposes into those pieces (down to single bytes if necessary). Consequences:

Every $y_{t}$ is a real vocabulary entry, so its probability $p (y_{t} ∣ y_{< t}, x)$ is always one of the $V$ softmax outputs and is always defined. There is no true out-of-vocabulary target token.
A word-level model without this property would need a stand-in UNK token for unseen words (and would lose information doing so); T5 sidesteps that entirely.
Separately, at inference the model might print a malformed string (e.g. "posative" for a label task). Training is unaffected — we always score against the true target. At scoring time a non-matching label simply counts as wrong: no crash, no special case.

Definition Corner case 4 — pretraining (span corruption) is the

same loss Before any labelled task, T5 is pretrained with no human labels at all — yet it still uses the identical $L$ . The trick is to manufacture an (input, target) string pair from raw text by span corruption: delete a few short contiguous spans, replace each deleted span in the input with a unique sentinel token (<X>, <Y>, …), and make the target the sentinels followed by the text that was removed. Concretely: $input x = (Thank you ⟨ X ⟩ me to your party ⟨ Y ⟩ week.)$ $target y = (⟨ X ⟩ for inviting ⟨ Y ⟩ last ⟨ Z ⟩)$

Each sentinel (<X>, <Y>) is a pointer telling the decoder which blank it is filling.
The trailing <Z> marks the end of the reconstructed target.
Crucially, $y$ here is still just a token row, so the loss is still $L = - \sum_{t = 1}^{m} lo g p (y_{t} ∣ y_{< t}, x) .$ Nothing about the loss changes — only how we built the pair changes. This is exactly why pretraining and fine-tuning share one recipe, and why instruction tuning slots in so cleanly later. (Contrast with the decoder-only recipe of GPT, which reads the prompt in the same left-to-right stream, and with BERT, which scores masked tokens as independent per-position classification rather than as one generated sequence.)

The one-picture summary

Everything above, on one page: the input row flows into the bidirectional encoder; the decoder emits target boxes one at a time, each box's probability read off a softmax, each box's surprise $- lo g p$ stacked into a bar; the bars sum to $L$ ; and three example targets (word, label, number) all feed the same $L$ .

Recall Feynman retelling — the whole walkthrough in plain words

Imagine a fill-in-the-blank machine. You slide in a strip of word-boxes (the input), and it writes back word-boxes one at a time (the answer). Every time it's about to write a box, it makes a bet: "I think the next word is X, and I'm this-percent sure" — and the way it turns its gut scores into percentages that add to 100 is called softmax (it sums over every word in its whole vocabulary). If the real word matches its bet and it was confident, that's a tiny "oops" score; if the real word blindsides it, that's a huge "oops." We add up all the oops-scores across the answer — that total is the loss (for one example this is the negative log-likelihood; averaged over all your data it's the cross-entropy). The bets chain together honestly: each guess is allowed to see everything written before it, so we never pretend the words are unrelated. And here's the magic: it doesn't matter whether the answer is a German word, the label "positive", or the number "3.8" spelled out — they're all just word-boxes chopped into pieces the machine already knows, so the same oops-adding rule trains every job. Even the warm-up game — blank-the-spans, where we erase a few words, tag the gaps with pointers, and have it write the missing pieces back — is scored by that very same rule. During practice we always show it the true previous words (so one early slip can't ruin the rest), and every answer ends in a stop-mark so there's always at least one box to score. One machine, one scoring rule, a hundred jobs.

Active Recall

Does the chain-rule product assume independent tokens?

No — it is an exact identity for any sequence; dependence is carried by conditioning each factor on the full history

y_{< t}

Why

- lo g

on the product?

lo g

turns the underflow-prone product into a stable additive sum; the minus flips "bigger is better" into a loss we minimise.

Write the softmax formula.

p_{k} = exp (z_{k}) / \sum_{j = 1}^{V} exp (z_{j})

, where

j

runs over all

V

vocabulary tokens;

exp

makes scores positive, the denominator forces them to sum to 1.

Why is

m \geq 1

always?

Every target ends in the </s> token, so even an empty answer is the one-token row </s>; the loss sum always has at least one term.

L

cross-entropy or NLL?

Per example it is the negative log-likelihood of the target sequence; cross-entropy is its expectation over the data distribution.

Why is there no out-of-vocabulary target in T5?

Subword tokenisation splits any string into in-vocabulary pieces (down to bytes), so every target token has a defined probability.

How does span-corruption pretraining reuse the loss?

It builds an (input, target) pair by deleting spans and reconstructing them via sentinels; the target is still a token row, so the same

- \sum lo g p (y_{t} ∣ y_{< t}, x)

applies.

Connections

Cross-entropy loss — the loss we built in Step 4.
Transformer encoder-decoder architecture — the encoder/decoder machine drawn in Step 3.
Multi-task learning with prefixes — why the amber prefix in Step 2 lets one model do many jobs.