Compute-data-parameter tradeoffs

6.1.2 · AI-ML › Scaling & Efficient Architectures

WHY — hume kyun parwah karni chahiye?

WHY $C \approx 6 N D$ ? Har token ke liye roughly chahiye:

2 FLOPs per parameter forward pass ke liye (ek multiply + ek add per weight = 2 FLOPs).
~4 FLOPs per parameter backward pass ke liye (weights aur activations ke w.r.t. gradients, ≈ forward cost se do guna).

Toh total per token ≈ $6 N$ FLOPs. $D$ tokens ke upar:

Optimization problem (Chinchilla)

Empirical scaling law (Hoffmann et al., "Chinchilla", 2022) fit karta hai:

Formula Loss as a function of size aur data

$L (N, D) = irreducible E + too small \frac{A}{N ^{α}} + too little data \frac{B}{D ^{β}}$

$E$ : language ki entropy jise tum kabhi beat nahi kar sakte (irreducible loss).
$A / N^{α}$ : limited parameters ki penalty — $N \to \infty$ jaane par vanish ho jaati hai.
$B / D^{β}$ : limited data ki penalty — $D \to \infty$ jaane par vanish ho jaati hai.
Fitted (Hoffmann et al.): $α \approx 0.34$ , $β \approx 0.28$ . Caution: ye $α, β$ upar wale loss law mein $N$ aur $D$ ke exponents hain. Paper directly compute-optimal allocation exponents bhi report karta hai $a = 0.46$ , $b = 0.54$ (toh $N_{opt} \propto C^{0.46}$ , $D_{opt} \propto C^{0.54}$ ). Inhe raw fit values $α \approx 0.076$ , $β \approx 0.095$ se mat confuse karo jo tab aate hain jab loss bits mein differently-parameterized fit ke saath likhate hain — wo slope of loss vs. $lo g N$ dete hain aur ek alag quantity hain. Neeche hum $α \approx 0.34, β \approx 0.28$ use karte hain taaki derived $a, b$ paper ke headline $\approx 0.5$ se match karein.

Optimum KAISE dhundhein — derive karo

Hum $L (N, D)$ ko minimize karte hain $6 N D = C$ (fixed budget) ke subject to.

Step 1 — constraint substitute karo. Kyun? 2-variable constrained problem ko 1 variable mein convert karne ke liye. $6 N D = C$ se: $D = \frac{C}{6 N}$ .

$L (N) = E + \frac{A}{N ^{α}} + B (\frac{6 N}{C})^{β}$

Step 2 — differentiate karo aur zero pe set karo. Kyun? Ek smooth curve ke minimum ka slope zero hota hai.

$\frac{d L}{d N} = - α A N^{- α - 1} + β B (\frac{6}{C})^{β} N^{β - 1} = 0$

Step 3 — $N$ ke liye solve karo. Kyun? Hume chahiye ki $N_{opt}$ $C$ ke saath kaise scale karta hai. Terms move karo:

$α A N^{- α - 1} = β B (\frac{6}{C})^{β} N^{β - 1}$ $N^{- α - 1 - β + 1} = \frac{β B ( 6/ C ) ^{β}}{α A} \Rightarrow N^{- (α + β)} \propto C^{- β}$ $N_{opt} \propto C^{\frac{β}{α + β}}$

Step 4 — $D$ nikalo. Kyunki $D = C / (6 N)$ hai, $D_{opt} \propto C / C^{β / (α + β)} = C^{α / (α + β)}$ .

Figure — Compute-data-parameter tradeoffs

Flashcards

C, N, D ko link karne wali compute identity kya hai?

C \approx 6 N D

(2 FLOPs/param forward + 4 FLOPs/param backward, per token, D tokens ke upar).

C = 6 N D

mein 6 ka factor kyun?

2 FLOPs/param forward + 4 FLOPs/param backward (weights aur activations ke w.r.t. grads) = 6 FLOPs per parameter per token. (FMAs mein 3 hai, kyunki 1 FMA = 2 FLOPs.)

Chinchilla loss law likhiye.

L (N, D) = E + A / N^{α} + B / D^{β}

: irreducible + parameter-limited + data-limited terms.

N_opt compute C ke saath kaise scale karta hai?

N_{o pt} \propto C^{β / (α + β)} \approx C^{0.46}

(Hoffmann et al. ka reported a≈0.46).

D_opt compute C ke saath kaise scale karta hai?

D_{o pt} \propto C^{α / (α + β)} \approx C^{0.54}

(b≈0.54).

Exponents a aur b ka sum 1 kyun hona chahiye?

Kyunki

C \propto N D

hai, toh

N_{o pt} D_{o pt} \propto C^{a + b} = C^{1}

ke liye a+b=1 chahiye.

Tokens per parameter ke liye Chinchilla rule of thumb?

~20 tokens per parameter.

GPT-3 suboptimal kyun tha?

Apne data ke liye bahut bada: ~1.7 tokens/param vs optimal ~20; data-starved, over-parameterized.

Chinchilla se kab deviate karo aur ek chhote model ko over-train karo?

Jab inference cost dominate kare — chhota model + zyada data serve karna sasta hota hai.

N_opt derive karte waqt kaun sa constraint substitute karte ho?

D = C / (6 N)

compute identity se, phir L(N) minimize karo.

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tumhare paas ek fixed amount paisa hai ek robot banane aur sikhane ke liye. Paisa kharch kar sakte ho robot ka brain bada karne mein (zyada parameters) ya usse padhne ke liye zyada kitaabein dene mein (zyada data). Agar bahut bada brain banao lekin sirf ek kitaab do, toh woh smart hai lekin kuch jaanta nahi. Agar ek chhote brain ko ek million kitaabein do, woh sab yaad nahi rakh sakta. Trick hai balance: thoda bada brain, thodi zyada kitaabein — brain ke har bit ke liye ~20 pages. Yahi balance tumhare paison ke liye sabse smart robot deta hai.

Connections

Neural Scaling Laws — power-law backbone jisme yeh note fit hota hai.
Chinchilla vs GPT-3 — empirical head-to-head.
FLOP accounting in Transformers — precisely "6" kahaan se aata hai.
Inference cost vs training cost — kyun deployment tradeoff change karta hai.
Overfitting and capacity — data-starving ke peechhe intuition.
Learning rate schedules — token budget $D$ se match karna chahiye.

WHY — hume kyun parwah karni chahiye?

Optimization problem (Chinchilla)

Optimum KAISE dhundhein — derive karo

Worked examples

Common mistakes

Flashcards

Connections

Concept Map