Foundations — Neural scaling laws (Chinchilla, compute-optimal)

Before you can trust a single line of the parent note, you must own every letter it throws at you. Below, each symbol appears in the order it becomes needed — nothing uses anything above it that wasn't already defined.

1. What is a "model" and what is $N$?

Picture a mixing desk in a music studio: hundreds of knobs, each nudging the sound. A neural net is the same idea but with billions of knobs. $N$ counts them.

Look at Figure 1: each little circle is one knob, and the pink arrow reminds you that counting every knob on the board gives you $N$.

• $N$ small → few knobs → the machine is too simple to capture complicated patterns.
• $N$ large → many knobs → it can represent complicated patterns (whether it learns to is a separate question — that's where data comes in).

Why the topic needs $N$: "make the model bigger" literally means "increase $N$". You cannot talk about the size–vs–data trade-off without a number for size.

Related reading once you're comfortable: Transformer architecture (the specific machine whose knobs we're counting).

2. What is $D$ — how much experience?

Picture a stack of flashcards. Each card is one token the model sees. $D$ is the height of the stack.

• $D$ small → little practice → even a big-brained model hasn't seen enough to learn.
• $D$ large → lots of practice → the model can pin down what its many dials should be set to.

Why the topic needs $D$: "feed it more data" means "increase $D$". Size ($N$) and experience ($D$) are the two things we trade off.

3. What is "loss" and the symbol $L$?

Think of $L$ as a golf score: lower is better, and zero (perfect) is basically impossible to reach.

Because $L$ depends on both how big the model is and how much it studied, we write it as a function of both: $L(N,D)$ — read aloud as "$L$ of $N$ and $D$", meaning the loss you get for that particular choice of size and data.

Look at Figure 2: the blue curve is the loss dropping as you pour in more size and data. Notice it flattens toward a dashed pink line — a mistake-floor it can never cross. That floor is real and important, and we give it the symbol $E$ in Section 6 (for now just notice the curve stops falling).

Why can loss never reach zero? Because language itself is partly unpredictable — no one can guess the exact next word every time — so there is always some leftover, unavoidable mistake. Section 6 names it.

4. Functions and $f(\cdot )$ notation

Picture a vending machine: press a button (input), get a snack (output). $f(N)$ later means "a machine that now only needs one input, $N$."

Why the topic needs this: the whole derivation is about finding which input to a function gives the smallest output. You must be at ease reading $L(N,D)$ and $f(N)$ before Step 2 of the parent.

5. Powers, and negative powers like $N^{-\alpha }$

The little raised number ($2$, or the Greek letter $\alpha$) is called the exponent — it controls how fast the growing or shrinking happens.

Look at Figure 3: three curves of $N^{-\alpha }$ for different exponents. Every one sags toward zero as $N$ grows, and the bigger the exponent, the faster it sags — that's the whole meaning of "the size of $\alpha$ controls the speed."

• $N^{-\alpha }$ with $\alpha >0$: bigger model → smaller mistake-term.
• The size of $\alpha$ says how quickly that term shrinks. (Note: this shrinking only works because $\alpha$ is positive — a negative $\alpha$ would make the term grow, which is the wrong physics.)

Why the topic needs this: the loss formula is built entirely from terms like $A/N^{\alpha }$ and $B/D^{\beta }$. If you can't read a negative power, the central equation is gibberish.

6. The Greek letters: $\alpha ,\beta$ (exponents) and $A,B,E$ (constants)

Picture a race: $A$ and $B$ are where two runners start, $\alpha$ and $\beta$ are their speeds, and $E$ is the finish line neither can cross.

These are not chosen by us — they are fitted (measured) by training many models and seeing what numbers make the formula match reality.

Putting sections 1–6 together, the parent's central equation now reads with zero mystery:

7. "Power law" vs "exponential" — and diminishing returns

Why the topic needs this: the phrase "diminishing returns = power law" is the reason scaling curves are smooth and predictable — the foundation of the whole field.

8. The symbol $\propto$ ("proportional to")

Picture two gears meshed together: turn one, the other turns in fixed ratio. $\propto$ is that fixed-ratio link, with the ratio left unnamed.

Why the topic needs this: the headline results $N_{\text{opt}}\propto C^{0.45}$ and $D_{\text{opt}}\propto C^{0.55}$ are stated with $\propto$ because only the exponent (how growth scales) is the interesting part.

9. FLOPs, compute $C$, and the number $6$

Picture a factory: $N$ workstations, $D$ items on the belt; each item visits each station and each visit costs a fixed 6 units of work.

Why the topic needs this: $C=6ND$ is the budget constraint. It locks $N$ and $D$ together — spend more on one, you have less for the other. That single equation is what turns "scaling" into an optimization problem. See Compute budgets & FLOPs.

10. "Constraint" and "optimum" — the shape of the whole problem

Picture a fence enclosing a field (the constraint) and you rolling a ball to find the lowest dip inside the fence (the optimum).

Why the topic needs this: the entire "HOW to derive" section is just "roll the ball to the lowest allowed point." Recognising it as constrained optimization tells you why calculus (setting a slope to zero) shows up next.

11. Slope, derivative, and "set it to zero"

Look at Figure 4: the blue curve is a valley (U-shape). The pink arrows show why it's a valley — the left branch falls (too few dials) and the right branch rises (the fixed budget forces $D$ down). The yellow dot marks the flat bottom, where the tangent line is horizontal, i.e. slope $=0$ — that is $N_{\text{opt}}$.

Here the loss-vs-$N$ curve is unimodal: one term ($A/N^{\alpha }$) always falls as $N$ grows, the other ($\propto N^{\beta }$ after substituting the budget) always rises — their sum makes a single U-shaped dip with exactly one flat bottom. (This "one always down, one always up" picture only holds because both $\alpha >0$ and $\beta >0$.)

Why the topic needs this: Step 3 of the parent differentiates and sets to zero. Without knowing "slope $=0$ at the bottom," that step looks like magic.

Prerequisite map

Equipment checklist

Test yourself — you're ready for the parent note only when each line's answer comes instantly.

Once every line above is second nature, jump back to the parent topic and the derivation will read like plain English. For neighbouring ideas you'll meet soon, see Kaplan scaling laws, Overfitting vs underfitting, Mixture-of-Experts, and Learning rate schedules.