GPT family architecture evolution

WHY does GPT exist at all?

WHAT is the objective? Maximize the probability of each next token given all previous tokens:

Formula Autoregressive language modeling

For a sequence $x_1,\dots,x_T$ , the model factorizes the joint probability using the chain rule of probability: $P(x_1,\dots,x_T)=\prod_{t=1}^{T}P(x_t\mid x_{<t})$ Derivation (first principles): any joint distribution can be written as $P(x_1,x_2)=P(x_1)P(x_2\mid x_1),$ and applying this repeatedly (telescoping) gives the product above. No assumption is made — it is exact. GPT just parameterizes each factor $P(x_t\mid x_{<t})$ with a neural net $f_\theta$ . The training loss is the negative log-likelihood (cross-entropy): $\mathcal{L}(\theta)=-\frac{1}{T}\sum_{t=1}^{T}\log P_\theta(x_t\mid x_{<t})$ Why the log? Products of tiny probabilities underflow; $\log$ turns the product into a sum and makes gradients well-behaved.

HOW is one GPT block built (from scratch)

A single decoder layer does two things: mix information across positions (attention) then process each position independently (MLP). Each is wrapped with a residual connection + normalization.

Formula Causal self-attention

Given token embeddings $X\in\mathbb{R}^{T\times d}$ , project into queries, keys, values: $Q=XW_Q,\quad K=XW_K,\quad V=XW_V$ Attention score between token $i$ and $j$ is the dot product $q_i\cdot k_j$ . Why divide by $\sqrt{d_k}$ ? For random $q,k$ with unit-variance entries, $q\cdot k$ has variance $d_k$ ; dividing by $\sqrt{d_k}$ keeps scores $O(1)$ so softmax doesn't saturate. $\text{Attn}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V$ The causal mask $M$ : set $M_{ij}=-\infty$ for $j>i$ so a token can never see the future. After softmax those weights become $0$ . This mask is the thing that makes GPT autoregressive.

Figure — GPT family architecture evolution

The evolution timeline (WHAT actually changed)

Definition Model-by-model

GPT-1 (2018): 12 layers, 117M params, 512 context. Introduced generative pretraining + supervised fine-tuning. Post-LN, learned absolute position embeddings, GELU activation.
GPT-2 (2019): up to 1.5B params, 48 layers, 1024 context. Key change: moved LayerNorm to pre-activation (pre-LN) and added a final LayerNorm. Showed zero-shot task transfer — no fine-tuning needed.
GPT-3 (2020): 175B params, 96 layers, $d=12288$ , 96 heads, 2048 context. Architecturally almost identical to GPT-2 + alternating dense and locally-banded sparse attention. Star result: in-context / few-shot learning emerges purely from scale.
GPT-3.5 / InstructGPT: same backbone + RLHF (instruction tuning + reinforcement learning from human feedback). Not an architecture change — an alignment change.
GPT-4 (2023): undisclosed, widely believed to be a Mixture-of-Experts (MoE) model, multimodal (text+image), far longer context. Architecture details closed, but the scaling philosophy is unchanged.

Why scale works: the scaling laws (Forecast-then-Verify)

Common mistakes (Steel-man + fix)

Worked examples

Recall Feynman: explain to a 12-year-old

Imagine you're reading a sentence and covering the words after where you are, trying to guess the next word. GPT is a machine that plays this "guess the next word" game billions of times until it's amazing at it. Each version of GPT is the same game machine, just built bigger — more brain cells (parameters) and more books to read (data). The clever trick called "masking" is like a rule: you're not allowed to peek at words you haven't reached yet, so you truly learn to predict instead of copying the answer. When the machine got big enough, something magical happened: you could just show it a couple of examples in your question and it would figure out the pattern by itself — nobody had to retrain it.

Connections

Transformer architecture — GPT is its decoder half.
Self-attention and multi-head attention — the core mixing operation.
Scaling laws (Kaplan, Chinchilla) — why bigger works and the compute-optimal correction.
BERT vs GPT (encoder vs decoder) — contrasting pretraining objectives.
RLHF and InstructGPT — how GPT-3.5 was aligned.
Mixture-of-Experts — the leading hypothesis for GPT-4.
Positional encodings — learned absolute (GPT-1/2) vs later rotary variants.
Cross-entropy loss — the training objective's foundation.

Flashcards

What sub-part of the original Transformer does GPT keep?

Only the decoder (masked/causal self-attention stack); no encoder, no cross-attention.

What probability rule justifies the autoregressive factorization?

The chain rule of probability, applied repeatedly (telescoping) — it's exact, no approximation.

Why divide attention scores by

\sqrt{d_k}

The dot product's variance grows as

d_k

; dividing keeps scores

O(1)

so softmax doesn't saturate and gradients don't vanish.

What is the role of the causal mask

M

Sets scores for future positions to

-\infty

so they become 0 after softmax — prevents a token from seeing the future, enforcing autoregression.

Key architectural change from GPT-1 to GPT-2?

Move LayerNorm to pre-activation (pre-LN) inside the residual branch + add a final LayerNorm → stable deep training.

What emergent capability defined GPT-3?

In-context / few-shot learning — solving tasks from prompt examples with frozen weights, no fine-tuning.

GPT-3 parameter count and depth?

~175B parameters, 96 layers, d=12288, 96 heads, 2048 context.

What distinguishes GPT-3.5 (InstructGPT) — architecture or training?

Training: instruction tuning + RLHF (alignment). The backbone architecture is essentially unchanged.

What are GPT-4's widely-believed additions?

Multimodality (text + image) and Mixture-of-Experts; details undisclosed.

How does test loss scale with parameters

N

As a power law

L(N)\approx(N_c/N)^{\alpha_N}

with

\alpha_N\approx0.076

; a straight line on a log-log plot.

Why use log-likelihood instead of raw likelihood?

Products of many small probabilities underflow; log turns products into sums with stable gradients.

Per-layer attention parameter count in terms of

d

4d^2

(from

W_Q,W_K,W_V,W_O

), so params grow quadratically with width.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, GPT ki sabse badi baat yeh hai ki har naya version koi bilkul naya jaadu nahi hai — woh same decoder-only Transformer block hai, bas usko deeper aur wider bana diya aur zyada data pe train kiya. Original 2017 Transformer mein encoder aur decoder dono the, lekin GPT ne encoder phenk diya kyunki language modeling mein sirf "agla word predict karo" karna hai — future dekhne ki zaroorat hi nahi. Yeh "future mat dekho" wala rule causal mask se lagta hai, aur yahi GPT ka dil hai.

Training ka objective simple hai: chain rule of probability se poore sentence ka probability tod do next-token predictions mein, aur cross-entropy loss minimize karo. Attention mein jo $\sqrt{d_k}$ se divide karte hain woh koi random cheez nahi — dot product ki variance $d_k$ ke saath badhti hai, toh usko control karke softmax ko saturate hone se bachate hain. Yeh chhoti-chhoti baatein exam aur interview dono mein poochi jaati hain.

Evolution yaad rakhne ka short trick: GPT-1 ne fine-tuning introduce kiya, GPT-2 ne zero-shot dikhaya aur pre-LN ka stability trick laaya, GPT-3 (175B) ne few-shot in-context learning ka magic dikhaya — sirf prompt mein examples daal do, weights freeze rehte hain. GPT-3.5 ne RLHF se alignment kiya (architecture same), aur GPT-4 multimodal + Mixture-of-Experts maana jaata hai. 80/20 funda: 80% badlav sirf scale hai, baaki 20% yeh key tweaks — inhi 20% pe focus karo.

Aur ek important galti se bacho: log mat bhoolo ki GPT-3 ko har task ke liye fine-tune nahi kiya jaata — usko prompt hi mein examples dikha dete hain. Yeh in-context learning scale se emerge hoti hai, planned feature nahi thi.

WHY does GPT exist at all?

HOW is one GPT block built (from scratch)

The evolution timeline (WHAT actually changed)

Why scale works: the scaling laws (Forecast-then-Verify)

Common mistakes (Steel-man + fix)

Worked examples

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Pretraining & Fine-Tuning LLMs

Connections