Before T5, every task had its own head: classification used a softmax over labels, span-QA used two pointer heads (start/end), regression used a scalar output. Each head needed custom code, custom loss, and custom fine-tuning tricks.
T5 = "Text-To-Text Transfer Transformer" (five T's → "T5").
Deriving why this ONE loss handles ALL tasks: Any task's answer is a string y. Classification label "positive" is a string. A number "3.8" is a string. Cross-entropy over the target string tokens therefore optimizes every task with the same formula. That's the whole trick.
Feed prefix + input, run the decoder greedily (or with beam search) until </s>. Read the produced string as the answer. For classification you check which label string was produced; if it's off-vocabulary, it simply counts as wrong.
Recall Feynman: explain to a 12-year-old
Imagine one super-smart student who only knows how to do ONE thing: you hand them a sentence, and they write back a sentence. To make them translate, you write "translate to German:" in front. To make them grade grammar, you write "is this okay:" and they write "yes" or "no." To learn, they play fill-in-the-blank: you erase a few words, and they guess what was there. Because everything is just "read a note, write a note," the same student can do a hundred different jobs.
Every NLP task is cast as "input string → output string", using one model, one architecture, and one cross-entropy loss.
What architecture does T5 use?
A standard encoder–decoder Transformer (bidirectional encoder + autoregressive decoder), not decoder-only.
How does T5 handle classification without a softmax label head?
It generates the label as text, e.g. "positive"/"unacceptable", and scores cross-entropy over those target tokens.
How does T5 handle regression (e.g. STS-B similarity)?
It emits the number as a string (rounded to nearest 0.2), predicting the digits as tokens with the same cross-entropy loss.
What is T5's pretraining objective?
Span corruption: drop ~15% of tokens in short contiguous spans, replace each span with a unique sentinel token, and have the decoder generate the sentinels followed by the missing text.
Why span corruption instead of BERT-style single-token masking?
T5 must OUTPUT text; predicting whole spans (as generated sequences) matches its text-to-text design and yields richer representations than independent per-token classification.
What are sentinel tokens (like , )?
Unique placeholder tokens marking each corrupted span, so the decoder knows which blank it is filling in the target.
Write T5's training loss for target y given input x.
L=−∑t=1mlogp(yt∣y<t,x), i.e. token-level cross-entropy under teacher forcing.
How does a task prefix work in T5?
A short instruction string (e.g. "translate English to German:") is prepended to the input so one model can be told which task to perform.
Why can one loss train all tasks in T5?
Every answer is a string, so cross-entropy over the target string optimizes classification, regression, translation, and summarization identically.
Dekho, T5 ka core idea bahut simple hai: har NLP task ko "string in, string out" bana do. Chahe translation ho, chahe sentiment classification, chahe summary, ya phir similarity ka number nikalna — sab kuch bas ek text lo aur ek text bahar do. Iske liye aap input ke aage ek chhota sa task prefix laga dete ho, jaise "translate English to German:" ya "summarize:", taaki model ko pata chale kaunsa kaam karna hai. Kamaal ki baat: classification ka label bhi text hai ("positive"), aur regression ka score bhi text hai ("3.8" — digits ko literally print kar deta hai!).
Architecture ke naam pe T5 ek encoder-decoder Transformer hai — GPT ki tarah decoder-only nahi. Encoder input ko bidirectionally padhta hai (dono directions se context), aur decoder ek-ek token karke output generate karta hai. Isi wajah se ek hi model, ek hi architecture, aur ek hi loss (token-level cross-entropy) se saare tasks train ho jaate hain. Yehi unification pura magic hai — na per-task head, na alag-alag code.
Pretraining ke liye T5 span corruption khelta hai — matlab MadLibs jaisa game. Sentence ke kuch chhote hisse (spans) hata do, unki jagah unique sentinel tokens (<X>, <Y>) daal do, aur decoder se bolo original missing text guess karo. BERT single token predict karta hai classification style mein, par T5 poora missing text generate karta hai — isliye pretraining bhi text-to-text shape mein rehta hai. Exam/interview ke liye yaad rakho: 5 T's, 1 shape (string → string), MadLibs-with-sentinels pretraining. Bas yahi 20% padho toh 80% samajh aa jaata hai.
