T5 and text-to-text framework: Worked examples

Warm-up: the three tools we will reuse

Definition Why

$lo g$ enters at all We want a score of pain: small when the model is confident-and-right, large when it is confident-and-wrong. The plain probability $p$ won't do — it rises with goodness. We want the opposite, and we want products to become sums (a sequence's probability is a product of per-token probabilities, and products of tiny numbers underflow to zero on a computer).

The function $- lo g p$ does exactly this: $- lo g (1) = 0 (perfect, no pain), - lo g (0.5) \approx 0.693, - lo g (0.01) \approx 4.605 (big pain) .$ As $p \to 0$ , $- lo g p \to + \infty$ — being confidently wrong is punished without limit. That is the whole reason $lo g$ shows up in the loss and no other tool (like $p$ itself, or $1 - p$ ) is used. All logs here are natural logs (base $e \approx 2.718$ ), which is the convention for cross-entropy in "nats".

Related tools: Cross-entropy loss, Transfer learning and fine-tuning, Multi-task learning with prefixes.

The scenario matrix

Every T5 example lives in one of these cells. The worked examples below are labelled with the cell they cover, and together they hit all of them.

Cell	Case class	What's special / the "edge"	Example
A	Classification, model right	short label string, high $p$	Ex 1
B	Classification, model wrong	off-vocabulary / wrong label → still just CE	Ex 2
C	Regression as text (STS-B)	number `"3.8"` emitted as digit-tokens; rounding to 0.2	Ex 3
D	Translation, multi-token target	full product over several tokens	Ex 4
E	Pretraining span corruption	sentinels `<X>,<Y>`, build input+target from raw text	Ex 5
F	Degenerate input/output	empty target = just `</s>`; $m = 1$ limiting case	Ex 6
G	Multi-task prefix decides task	same input string, two prefixes → two answers	Ex 7
H	Exam twist: greedy vs. beam	greedy can pick a locally-good but globally-wrong string	Ex 8

We will reuse two tiny fixed "sureness" numbers so the arithmetic is clean and checkable.

Ex 1 — Cell A: classification, model is right

Forecast: guess first — will the loss be closer to $0$ , to $1$ , or above $2$ ? (The model is very sure and correct, so bet on small.)

Write the product of confidences. $p (y ∣ x) = 0.90 \times 0.99 = 0.891$ . Why this step? The chain rule (parent note) factorizes the whole-string probability into one factor per token; multiply them.
Take $- lo g$ of each factor and add. $L = - lo g (0.90) - lo g (0.99)$ . Why this step? $- lo g$ of a product = sum of $- lo g$ s; this is exactly $L = - \sum_{t} lo g p_{t}$ .
Evaluate. $- lo g (0.90) = 0.10536$ , $- lo g (0.99) = 0.01005$ , so $L = 0.11541$ . Why this step? Plug the numbers into the natural log.
Per-token loss. $L / m = 0.11541/2 = 0.05771$ . Why this step? Divides the total pain over the 2 tokens so it's comparable across examples.

Verify: $- lo g (0.891) = 0.11541$ ✓ (same as summing the two logs — a good consistency check that the product/sum identity held). Small loss, as forecast: the model was confident and correct.

Ex 2 — Cell B: classification, model is wrong

Forecast: the model is confidently wrong — expect the loss to be large compared to Ex 1.

Loss on the true token. $- lo g (0.05) = 2.99573$ . Why this step? We always score the probability the model gave to the correct target, no matter what it preferred.
Add the </s> term. $- lo g (0.99) = 0.01005$ , so $L = 3.00578$ . Why this step? Sum over all target tokens as before.
Compare. $3.006/0.115 \approx 26 \times$ the loss of Ex 1. Why this step? Confidence in the wrong answer is punished hard — that's the $- lo g p \to \infty$ behaviour from the warm-up.

Verify: $- lo g (0.05) - lo g (0.99) = 3.00578$ ✓.

Ex 3 — Cell C: regression emitted as text (STS-B)

Forecast: rounding $3.83$ to a multiple of $0.2$ — is it $3.8$ or $4.0$ ? And is the loss moderate (the model is only 60% sure)?

Round to nearest $0.2$ . Compute $3.83/0.2 = 19.15$ , round to $19$ , multiply back: $19 \times 0.2 = 3.8$ . Why this step? T5 discretizes the continuous range into a finite vocabulary of targets ( $0.0, 0.2, \dots, 5.0$ = 26 values) so cross-entropy — a tool for choosing among finitely many options — even applies. A raw real number has no "probability of exactly $3.83$ ".
Check the boundary case. The two candidates are $3.8$ and $4.0$ ; $3.83$ is $0.03$ from $3.8$ and $0.17$ from $4.0$ , so $3.8$ wins. Why this step? Always confirm which side of the midpoint ( $3.9$ ) you're on — the exam trap is rounding the wrong way.
Compute the loss. $L = - lo g (0.6) - lo g (0.95) = 0.51083 + 0.05129 = 0.56212$ . Why this step? Identical machinery — the number is just a string.

Verify: rounded target $= 3.8$ ✓; $L = 0.56212$ ✓.

Ex 4 — Cell D: translation, several target tokens

Forecast: four factors each below 1, so the product shrinks — will $p (y ∣ x)$ be around $0.5$ or nearer $0.1$ ?

Multiply confidences. $p = 0.8 \times 0.7 \times 0.9 \times 0.99 = 0.49896$ . Why this step? Chain rule: whole-sentence probability is the product of per-token probabilities.
Total loss = sum of $- lo g$ . $L = - lo g 0.8 - lo g 0.7 - lo g 0.9 - lo g 0.99.$ $= 0.22314 + 0.35667 + 0.10536 + 0.01005 = 0.69523.$ Why this step? Turns the shrinking product into a friendly additive score.
Per-token loss. $0.69523/4 = 0.17381$ . Why this step? A 4-token sentence and a 2-token label can now be compared on equal footing.

Verify: $- lo g (0.49896) = 0.69523$ ✓ (product-vs-sum agree); per-token $= 0.17381$ ✓.

Ex 5 — Cell E: build a span-corruption training pair

Forecast: how many sentinels appear in the input? in the target? (Hint: one per dropped span, plus a closing sentinel at the very end of the target.)

Replace span 1 with <X>, span 2 with <Y>. Input: Thank you <X> me to your party <Y> week . Why this step? Sentinels are unique pointers so the decoder knows which blank each generated chunk fills.
Target = each sentinel followed by its original text, then a final <Z>. Target: <X> for inviting <Y> last <Z> Why this step? The decoder must generate the missing text (keeping the text-to-text shape), not classify masked positions like BERT does. The trailing <Z> marks end-of-target.
Count target tokens. <X>, for, inviting, <Y>, last, <Z> = $m = 6$ tokens. Why this step? $m$ is the number of terms in the loss sum — here 6.

Verify: target token count $m = 6$ ✓ (checked in VERIFY by counting the listed tokens).

Ex 6 — Cell F: the degenerate / limiting case

Forecast: with a single token the loss is just $- lo g p$ — no sum to spread it over. Bigger or smaller than the multi-token cases?

One-term sum. $L = - lo g (0.4) = 0.91629$ . Why this step? The sum $\sum_{t = 1}^{m}$ with $m = 1$ has exactly one term — the shortest possible target still has a well-defined loss.
Per-token loss = total loss. $0.91629/1 = 0.91629$ . Why this step? Dividing by $m = 1$ changes nothing; the degenerate case doesn't break the formula.
Limiting sanity check. If the model were sure to stop, $p \to 1$ and $L \to - lo g 1 = 0$ . If it refused to stop, $p \to 0$ and $L \to \infty$ . Why this step? Confirms both extreme ends behave sensibly — the reader never hits an undefined scenario.

Verify: $L = 0.91629$ ✓.

Ex 7 — Cell G: the prefix chooses the task

Forecast: which single change flips the output from a label to a German sentence? (It is not a new head.)

Prepend the task prefix to the encoder input. Why this step? The prefix is ordinary text the encoder reads; it conditions the decoder on which task to perform — this is multi-task-by-prefix.
Decode until </s>; read the string as the answer. Why this step? Same architecture and same cross-entropy for both — only the target string differs.
Loss is identical machinery. For the CoLA target ["unacceptable","</s>"] with $p = 0.7, 0.98$ : $L = - lo g 0.7 - lo g 0.98 = 0.35667 + 0.02020 = 0.37688$ . Why this step? Demonstrates the same loss formula scores a label string, just as it scored digits and German words.

Verify: $L = 0.37688$ ✓.

Ex 8 — Cell H: exam twist, greedy vs. beam

Forecast: greedy commits to $0.6$ at step 1, but path Q's second token is much stronger. Guess who wins overall.

Sequence probabilities. $p_{P} = 0.6 \times 0.3 = 0.18$ ; $p_{Q} = 0.5 \times 0.8 = 0.40$ . Why this step? Chain rule again — a string's score is the product of its tokens, not just the first token.
Losses. $L_{P} = - lo g 0.18 = 1.71480$ ; $L_{Q} = - lo g 0.40 = 0.91629$ . Why this step? Lower loss = higher-probability string. $Q$ wins ( $0.916 < 1.715$ ).
Conclusion. Greedy chose $P$ (higher first token) but $Q$ is the better sequence. Beam search keeps several partial paths alive, so it can recover $Q$ . Why this step? Shows the classic exam trap: locally best ≠ globally best. This is a decoding choice at inference; it does not change the training loss.

Verify: $p_{P} = 0.18$ , $p_{Q} = 0.40$ ✓; $L_{P} = 1.71480$ , $L_{Q} = 0.91629$ , and $L_{Q} < L_{P}$ ✓.

Recall

Recall Did every cell get covered?

Which example handled a wrong classification? ::: Ex 2 (Cell B). How does Ex 3 turn a real number into something cross-entropy can score? ::: Round to nearest 0.2 → one of 26 string labels → CE over its tokens. In Ex 5, why does the target end in <Z>? ::: <Z> is the closing sentinel marking end-of-target. In Ex 6, what is the shortest possible target and its loss when $p (</s>) = 0.4$ ? ::: ["</s>"], $m = 1$ , $L = - lo g 0.4 \approx 0.916$ . Why did greedy lose in Ex 8? ::: It maximized the first token ( $0.6$ ) but path Q had higher total product ( $0.40 > 0.18$ ).

Connections

Cross-entropy loss — the single objective every example above used.
Multi-task learning with prefixes — Ex 7's mechanism.
Prompting and instruction tuning — prefixes generalize to prompts.
Transformer encoder-decoder architecture — why a bidirectional encoder + autoregressive decoder makes text-to-text natural.
GPT and decoder-only models — contrast with the decoding in Ex 8.
BERT and masked language modeling — contrast with span corruption in Ex 5.