4.3.1
Intuition Ek sentence ki kahaniHar GPT essentially wahi decoder-only Transformer block hai jo zyada deep aur wide stack kiya gaya hai , aur zyada tokens par train kiya gaya hai — architecture mein khaas badlaav nahi aaya ; jo badla woh hai scale (parameters, data, context) aur kuch stability/efficiency tweaks . GPT ne yeh bet lagayi ki "scale + next-token prediction" kisi bhi clever architecture ko beat kar deta hai, aur isi se success mili.
Intuition Decoder-only kyun?2017 wale original Transformer mein ek encoder tha (jo poora input padhta tha) aur ek decoder tha (jo output generate karta tha). Pure language modeling ke liye — yaani "agli word predict karo" — tum future ko kabhi attend nahi karte, aur encode karne ke liye koi alag input bhi nahi hota. Isliye GPT ne encoder ko hataa diya aur sirf causal (masked) decoder rakha. Ek hi tower, ek hi objective, scale karna bahut simple.
OBJECTIVE kya hai? Har next token ki probability maximize karo, given sab pehle wale tokens:
Ek decoder layer do kaam karti hai: positions ke across information mix karo (attention), phir har position ko independently process karo (MLP). Dono ko residual connection aur normalization ke saath wrap kiya jaata hai.
Definition Model-by-model
GPT-1 (2018): 12 layers, 117M params, 512 context. Generative pretraining + supervised fine-tuning introduce kiya. Post-LN, learned absolute position embeddings, GELU activation.GPT-2 (2019): 1.5B params tak, 48 layers, 1024 context. Key change: LayerNorm ko pre-activation (pre-LN) par move kiya aur ek final LayerNorm add kiya. Zero-shot task transfer dikhaya — koi fine-tuning zaroorat nahi.GPT-3 (2020): 175B params, 96 layers, d = 12288 alternating dense aur locally-banded sparse attention . Star result: in-context / few-shot learning purely scale se emerge hoti hai.GPT-3.5 / InstructGPT: same backbone + RLHF (instruction tuning + reinforcement learning from human feedback). Architecture change nahi — ek alignment change hai.GPT-4 (2023): undisclosed, widely believed to be a Mixture-of-Experts (MoE) model, multimodal (text+image), kaafi lamba context. Architecture details closed hain, lekin scaling philosophy unchanged hai.
"GPT har saal alag hai" ka 80% sirf scale hai. Baaki 20% jo tumhe zaroor yaad rakhna chahiye: (1) stability ke liye pre-LN, (2) causal masking core hai, (3) in-context learning GPT-3-scale ki ek emergent property hai, (4) RLHF (GPT-3.5) aur MoE/multimodal (GPT-4) usi decoder skeleton par additions hain.
Common mistake "GPT encoder–decoder Transformer use karta hai."Kyun sahi lagta hai: 2017 ka famous paper hai encoder–decoder, aur GPT usisi se aaya. Fix: GPT decoder-only hai. Yeh sirf masked self-attention stack rakhta hai; koi cross-attention nahi hai aur koi encoder nahi hai. (BERT encoder-only hai; T5 encoder–decoder hai.)
Common mistake "GPT-3 ko har task par fine-tune kiya gaya tha."Kyun sahi lagta hai: GPT-1 ne per task fine-tune kiya tha, toh lagta hai poora family karta hai. Fix: GPT-3 ka headline result yeh hai ki woh few-shot in-context learning karta hai — examples prompt mein daalo, weights frozen rehte hain. Fine-tuning optional hai, required nahi.
d k Kyun sahi lagta hai: yeh ek arbitrary scaling jaisa dikhta hai. Fix: yeh specifically dot product ki variance growth (Var = d k d k
Common mistake "Pre-LN vs Post-LN se koi farak nahi padta."Kyun sahi lagta hai: dono sirf LayerNorm placements hain. Fix: Post-LN deep networks mein exploding/vanishing residual signals hoti hain; pre-LN ek clean residual highway rakhta hai , jis se GPT-2/3 ki 48–96 layers stably train hoti hain.
Worked example 1 — Attention block ke parameters count karnaModel dim d W Q , W K , W V , W O d × d = 4 d 2 d = 12288 4 × 1228 8 2 ≈ 6.0 × 1 0 8 per layer .
Yeh step kyun? Yeh dikhata hai ki params d 2 wajah hai ki model ko widen karna itna parameter-hungry hai, aur kul params (∼ 175
Worked example 2 — Causal mask strictly upper-triangular kyun honi chahiyePosition 3 par token, token 4 predict karte hue, positions { 1 , 2 , 3 } { 4 , 5 , … } M ij = 0 j ≤ i − ∞ Yeh step kyun? Agar token 3 token 4 dekh sakta, toh model "cheat" karta — training ke dauran answer pehle se visible hota, aur loss bina kisi real learning ke near-zero collapse ho jaata.
Worked example 3 — Scale jump forecast karnaGPT-2 (1.5B) → GPT-3 (175B) roughly 117 × α N = 0.076 Δ ( log L ) ≈ − 0.076 × ln ( 117 ) ≈ − 0.076 × 4.76 ≈ − 0.36 Yeh step kyun? Scaling law ka predictive use demonstrate karta hai: ek bada lekin bounded loss drop — capabilities improve hoti hain, lekin infinitely nahi, jo data/compute-optimal scaling (Chinchilla) ki search ko motivate karta hai.
Recall Feynman: ek 12 saal ke bachche ko explain karoSocho tum ek sentence padh rahe ho aur jo words aage hain unhe chhupa ke rakh rahe ho, aur agla word guess karne ki koshish kar rahe ho. GPT ek aisi machine hai jo yeh "agla word guess karo" wala game billions baar khel khel kar isme kamaal ban jaati hai. GPT ka har version wahi game machine hai , bas badi banayi gayi — zyada brain cells (parameters) aur padhne ke liye zyada books (data). "Masking" naam ka clever trick ek rule ki tarah hai: tumhe un words ko dekhne ki ijazat nahi jo tumne abhi tak nahi padhe, toh tum sach mein predict karna seekhte ho, sirf answer copy nahi karte. Jab machine kaafi badi ho gayi, toh kuch magical hua: tum apne question mein sirf do examples dikhaa sakte the aur woh khud pattern samajh jaati — kisi ne use dobara train nahi kiya tha.
Mnemonic Family yaad rakho"1 Fine-tunes, 2 Zero-shots, 3 Few-shots, 3.5 Aligns, 4 Multimodal-Experts."
(GPT-1 → fine-tuning, GPT-2 → zero-shot, GPT-3 → few-shot, GPT-3.5 → RLHF alignment, GPT-4 → multimodal + MoE.)
Original Transformer ka kaunsa sub-part GPT rakhta hai? Sirf decoder (masked/causal self-attention stack); koi encoder nahi, koi cross-attention nahi.
Autoregressive factorization ko kaunsa probability rule justify karta hai? Chain rule of probability, baar baar apply karke (telescoping se) — yeh exact hai, koi approximation nahi.
Attention scores ko d k Dot product ki variance d k O ( 1 )
Causal mask M Future positions ke scores ko − ∞
GPT-1 se GPT-2 mein key architectural change kya tha? LayerNorm ko pre-activation (pre-LN) par residual branch ke andar move karna + ek final LayerNorm add karna → stable deep training.
GPT-3 ki emergent capability kya thi? In-context / few-shot learning — frozen weights ke saath prompt examples se tasks solve karna, koi fine-tuning nahi.
GPT-3 ka parameter count aur depth? ~175B parameters, 96 layers, d=12288, 96 heads, 2048 context.
GPT-3.5 (InstructGPT) ko kya distinguish karta hai — architecture ya training? Training: instruction tuning + RLHF (alignment). Backbone architecture essentially unchanged hai.
GPT-4 ke widely-believed additions kya hain? Multimodality (text + image) aur Mixture-of-Experts; details undisclosed hain.
Test loss parameters N Power law ki tarah L ( N ) ≈ ( N c / N ) α N α N ≈ 0.076
Raw likelihood ki jagah log-likelihood kyun use karte hain? Bahut saari chhoti probabilities ka product underflow karta hai; log products ko stable gradients wale sums mein badal deta hai.
Per-layer attention parameter count d 4 d 2 W Q , W K , W V , W O
2017 Transformer encoder-decoder
Causal mask -inf on future
