T5 and text-to-text framework: Question bank — AI-ML deep dive

Every item below is a one-line reveal. The answer side always carries the reasoning, because in T5 the traps live in the "why", not the "what".

Notation and vocabulary you need first

Before the traps, let us earn every symbol and word used below, from zero.

Formula From softmax to cross-entropy — the algebra spelled out

Each factor $p (y_{t} ∣ y_{< t}, x)$ is one softmax over the vocabulary: the decoder outputs scores (logits) $z_{v}$ for every vocabulary token $v$ , and $p (y_{t} = v ∣ y_{< t}, x) = \frac{e ^{z_{v}}}{\sum _{u} e ^{z_{u}}} .$ By the chain rule the whole-string probability is the product of these: $p (y ∣ x) = \prod_{t = 1}^{m} p (y_{t} ∣ y_{< t}, x) .$ Take $- lo g$ of both sides. The $lo g$ turns the product into a sum: $- lo g p (y ∣ x) = - \sum_{t = 1}^{m} lo g p (y_{t} ∣ y_{< t}, x) = L .$ Each term $- lo g p (y_{t} ∣ y_{< t}, x)$ is exactly the cross-entropy between the true one-hot target (all mass on the correct token) and the softmax distribution — the familiar $- \sum_{v} q_{v} lo g p_{v}$ with $q$ one-hot. So "minimize $- lo g p (y ∣ x)$ " and "sum of per-token cross-entropy" are literally the same objective.

True or false — justify

T5 uses a different output head for each task (classification, regression, translation).

False — the whole point is ONE output: a text string generated by the decoder. Every task shares the same vocabulary softmax and the same cross-entropy objective.

T5 is a decoder-only model like GPT.

False — T5 is an encoder–decoder Transformer. A bidirectional encoder reads the input, and the decoder cross-attends to it; GPT has no separate encoder.

For classification, T5 outputs a probability distribution over the fixed label set.

False — it generates the label as literal text (e.g. "positive"), token by token. There is no closed label set at the output; an off-vocabulary string just counts as wrong.

Span corruption and BERT masking are the same objective.

False — BERT predicts each masked token independently as classification at that position; T5 generates a variable-length target sequence of

⟨

sentinel

⟩

span pairs. One is classification, the other is generation.

T5's regression target "3.8" is produced by a linear neuron plus MSE.

False — the number is discretized to the nearest 0.2 and printed as digit tokens, then scored with the same cross-entropy. No regression head, no MSE anywhere.

Because targets are strings, T5 can never do numeric tasks well.

False — rounding to a finite grid (nearest 0.2) turns the number into a small finite vocabulary of short strings, which language models produce reliably.

The task prefix is a special learned architectural token separate from the text.

False — the prefix like "translate English to German:" is ordinary text prepended to the input. Nothing about the model changes; only the input string does.

The chain-rule factorization

p (y ∣ x) = \prod_{t} p (y_{t} ∣ y_{< t}, x)

is an approximation.

False — the chain rule of probability factorizes any joint distribution exactly. It is not an approximation; the approximation only enters when we model each factor with a softmax.

Teacher forcing means the model sees its own previous predictions during training.

False — teacher forcing feeds the ground-truth previous tokens

y_{< t}

, not the model's guesses. That makes each per-token loss well-defined and training stable.

Spot the error

"T5 minimizes

- lo g p (y ∣ x)

, which is a different objective from cross-entropy."

Error — they are the same thing.

- lo g p (y ∣ x) = - \sum_{t} lo g p (y_{t} ∣ y_{< t}, x)

is exactly token-level cross-entropy against the true target tokens.

"To do multi-task learning, T5 needs a separate copy of the model per task, glued together."

Error — multi-task learning is done with ONE model: just mix datasets and prepend a distinguishing prefix. No copies, no gluing.

"In span corruption the target is the full original sentence with the blanks filled back in."

Error — the target is only the sentinels followed by their missing spans (e.g.

⟨ X ⟩

for inviting

⟨ Y ⟩

last

⟨ Z ⟩

), not the whole reconstructed sentence. The decoder produces just the dropped pieces.

"BERT could be used unchanged as a text-to-text model because it also hides ~15% of tokens."

Error — BERT's output is per-position vocabulary classification, not a generated sequence. It has no autoregressive decoder to emit variable-length target text.

"The final

⟨ Z ⟩

sentinel in a target marks another corrupted span to fill."

Error — the trailing sentinel marks the end of the target sequence, telling the decoder there is nothing more to generate; it is not an additional blank.

"Since T5 generates greedily at inference, it can only ever be used greedily."

Error — greedy decoding is one choice; beam search (or sampling) is equally valid. Decoding runs token-by-token until the end-of-sequence token </s> (EOS), regardless of the search strategy.

"Cross-entropy over label strings can't distinguish unacceptable from acceptable because they share letters."

Error — the loss is over tokens, and the two strings tokenize to different token sequences. The model is scored on producing the correct token sequence, not on character overlap.

Why questions

Why does T5 use contiguous spans rather than single-token masks in pretraining?

Predicting a multi-token phrase is harder and richer than one token, yielding better representations, and it produces generated text — matching T5's text-to-text output shape.

Why are sentinel tokens

⟨ X ⟩, ⟨ Y ⟩

needed at all?

They are unique pointers so the decoder knows which blank it is currently filling. Without them, multiple dropped spans in the target would be ambiguous.

Why does the same single loss handle translation, classification, and regression?

Because every answer — a sentence, "positive", or "3.8" — is just a string. Cross-entropy over the target string's tokens optimizes all of them with one identical formula.

Why turn a product of probabilities into a sum of

- lo g

terms?

Maximizing the product

p (y ∣ x)

equals minimizing

- lo g

of it; the log converts the product into an additive per-token loss that is numerically stable and easy to backprop through.

Why is T5 strong on understanding tasks compared to decoder-only models?

Its encoder reads the input bidirectionally (each token sees both left and right context), giving richer input representations than a strictly left-to-right stream.

Why does the text-to-text framing make transfer learning clean?

Pretraining and every downstream task use the identical model, objective, and input/output shape, so fine-tuning is just "keep training the same model on new string pairs" — no architectural surgery.

Why can a task prefix be seen as a lightweight form of instruction?

The prefix is a natural-language instruction inside the input telling the model what to do, so switching tasks is just switching the instruction string — the seed of prompting.

Edge cases

What happens if T5 generates a classification string that is not any valid label?

It is simply scored as wrong — there is no output constraint to the label set, so an off-vocabulary string counts as an incorrect prediction.

What if two corrupted spans happen to be adjacent in the original text?

They still get distinct sentinels (

⟨ X ⟩, ⟨ Y ⟩

), so the target keeps them separate; the sentinel identity, not position, disambiguates them.

What if a regression target falls between two rounding grid points, e.g. true value 3.7?

It is snapped to the nearest 0.2 (here 3.8 or 3.6), because the target vocabulary is the finite discretized grid — the model can only emit grid-valued strings.

What if the decoder never emits </s> during greedy inference?

Generation is stopped by the max-length cap — a fixed hyperparameter set before decoding; without an EOS token the output is truncated at that cap. Training includes the terminating EOS token so the model learns to stop on its own.

What if you feed T5 an input with no task prefix?

The model has no instruction telling it which task to perform, so its behavior is undefined/ambiguous — the prefix is what selects the task among the many it was trained on.

At inference, does the encoder run once or once per generated token?

The encoder runs once to encode the fixed input; only the decoder steps repeatedly, cross-attending to the cached encoder output at every generated token.

Connections

BERT and masked language modeling — the masking contrast that most traps hinge on.
GPT and decoder-only models — the decoder-only foil for the architecture traps.
Cross-entropy loss — the single objective behind every "why one loss" item.