Neural scaling laws (Chinchilla, compute-optimal)

6.1.1 · AI-ML › Scaling & Efficient Architectures

Scaling laws exist kyun karte hain?

Empirical finding: loss ek power law plus ek irreducible floor follow karta hai.

L (N, D) = irreducible E + \frac{A}{N ^{α}} + \frac{B}{D ^{β}}

$E$ = data ki entropy / Bayes error jo tum kabhi nahi beat kar sakte.
$A / N^{α}$ = error bahut kam parameters se (model function ko represent nahi kar sakta).
$B / D^{β}$ = error bahut kam data se (model jo represent kar sakta hai usse estimate nahi kar sakta).
Chinchilla fit: $α \approx 0.34$ , $β \approx 0.28$ (close hain, lekin equal nahi → optimum pe data ka role thoda zyada strong hai).

Compute-optimal split derive kaise karein (scratch se)

Goal: $L (N, D)$ minimize karo budget $C = 6 N D$ fix rakhte hue.

Step 1 — Irreducible floor drop karo. $E$ constant hai, toh yeh affect nahi karta ki minimum kahan hai. $f (N, D) = A N^{- α} + B D^{- β}$ minimize karo. Kyun yeh step? Constants optima move nahi karte; hum sirf trade-off terms ki parwah karte hain.

Step 2 — Constraint substitute karo. $C = 6 N D$ se, $D = \frac{C}{6 N}$ likho:

f (N) = A N^{- α} + B (\frac{C}{6 N})^{- β} = A N^{- α} + B (6/ C)^{β} N^{β} .

Kyun? Ab ek hi free variable hai — pure single-variable calculus.

Step 3 — Differentiate karo, zero pe set karo.

\frac{df}{d N} = - α A N^{- α - 1} + β B (6/ C)^{β} N^{β - 1} = 0.

Kyun? $f (N)$ $N > 0$ pe unimodal hai — pehla term $A N^{- α}$ strictly decrease karta hai aur doosra $\propto N^{β}$ strictly increase karta hai, isliye dono ka sum exactly ek interior minimum pe hoga jahan slope zero cross karta hai. (Globally convex hona zaroori nahi, lekin unimodality kaafi hai stationary point ko minimum guarantee karne ke liye.)

Step 4 — $N$ vs $C$ ke liye solve karo. Rearrange karne pe (VERIFY dekho):

N^{α + β} \propto C^{β} \Rightarrow N_{opt} \propto C^{β / (α + β)}, D_{opt} \propto C^{α / (α + β)} .

Kyun? Kyunki $D = C / (6 N)$ hai, $N$ aur $D$ ke exponents ka sum 1 hona chahiye (inhe multiply karo toh $C$ milta hai).

Figure — Neural scaling laws (Chinchilla, compute-optimal)

WORKED EXAMPLES

COMMON MISTAKES (Steel-manned)

Flashcards

Chinchilla loss functional form kya hai?

L (N, D) = E + A N^{- α} + B D^{- β}

: irreducible floor plus do power-law terms finite params aur finite data ke liye.

Dense transformer train karne ke approximate FLOPs?

C \approx 6 N D

(2 multiply-add ke liye × 3 fwd+bwd ke liye).

Compute

C

ke saath

N

ka compute-optimal scaling?

N_{opt} \propto C^{β / (α + β)} = C^{0.28/0.62} \approx C^{0.45}

Compute

C

ke saath

D

ka compute-optimal scaling?

D_{opt} \propto C^{α / (α + β)} = C^{0.34/0.62} \approx C^{0.55}

(data thoda tez badhta hai).

Parameter ke per tokens ke liye Chinchilla rule of thumb?

Lagbhag 20 tokens per parameter (

D \approx 20 N

Agar compute

4 \times

badhta hai, toh usse kaise split karte ho?

Roughly

N

double karo aur

D

double karo (

4^{0.45} \approx 1.9

4^{0.55} \approx 2.1

Gopher Chinchilla ke mukable suboptimal kyun tha?

Gopher bahut bada aur data-starved tha (

D / N \approx 1

); comparable compute pe Chinchilla ne ~20 tokens/param pe rebalance kiya 4× chhote model ke saath aur jeeta.

Term

E

kya represent karta hai?

Irreducible loss — data entropy / Bayes error jo tum scaling se kabhi nahi beat kar sakte.

Loss power law kyun follow karta hai (exponential kyun nahi)?

Diminishing returns:

N

D

ki har doubling kam help karti hai; yeh constant multiplicative decay exactly ek power law hai.

C

ke against

N

aur

D

ke exponents 1 kyun sum karte hain?

Kyunki

C = 6 N D

hai;

N \propto C^{a}

aur

D \propto C^{b}

force karte hain

a + b = 1

Kya compute-optimal exponents exactly 0.5 each hain?

Nahi — sirf tab agar

α = β

ho. Fitted values dete hain

N

ke liye ≈0.45 aur

D

ke liye ≈0.55; 0.5 ek rounded shortcut hai.

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum exam ke liye padh rahe ho. Tum ya toh bada brain grow kar sakte ho (zyada parameters) ya zyada books padh sakte ho (zyada data). Tumhare paas sirf itne hi ghante hain (compute). Agar bahut bada brain ho aur sirf ek book padho, tum ek genius ho jiske paas sochne ke liye kuch nahi — wasteful. Agar ek library padho lekin brain bahut chhota ho, tum zyaatar bhool jaate ho. Chinchilla ne sweet spot dhundha: brain aur reading dono ko almost same speed pe badhao (reading thodi si tez), lagbhag brain ke har bit ke liye 20 pages reading. Jab bhi $4 \times$ zyada ghante milein, brain ko roughly double karo aur roughly double padhna bhi karo.

Connections

Transformer architecture — $6 N D$ cost model dense attention/MLP FLOPs assume karta hai.
Compute budgets & FLOPs — $C \approx 6 N D$ kahan se aata hai.
Kaplan scaling laws — purane laws jo $N$ ko over-weight karte the; Chinchilla ne data term correct ki.
Overfitting vs underfitting — $A / N^{α}$ vs $B / D^{β}$ trade-off capacity vs estimation error hai.
Mixture-of-Experts — simple $6 N D$ assumption break karta hai (sparse activation).
Learning rate schedules — token budget $D$ ke saath match karna chahiye taaki law hold kare.