This page builds every symbol the parent note Emergent abilities leans on, starting from absolute zero. Nothing below is assumed — if the parent used it, we earn it here first.
Before any symbol, picture the machine. A language model reads some text and produces, for the next slot, a probability for every possible word. It is a guessing machine that never says one word — it says "I'm 71% sure it's cat, 12% dog, …".
Figure s01 — what the model emits. A bar chart over five candidate next tokens (cat, dog, car, sky, run). The tallest blue bar (cat, 0.55) is flagged with a red arrow reading "model is 55% sure of 'cat'"; the other four bars are gray and shorter. The visual makes the point that the model never commits to one word — it hands back a whole probability distribution that sums to 1.
We need tokens because every symbol that follows counts them or scores them.
Why: "scale" is not one thing. The parent lists three axes — N, D, C — any of which can trigger a jump. You must know all three to read its plots. These feed directly into Scaling laws (Kaplan, Chinchilla) and Compute-optimal training.
This is the heart of the parent note, so we build it with pictures, not faith. First we must name the quantity we are computing.
Figure s02 — the leak in the chain. Exact-match accuracy pL on the vertical axis versus chain length L (1 to 20) on the horizontal axis, drawn for three per-token skills: green p=0.90, orange p=0.80, red p=0.70. All three start high at L=1 and sag toward zero as L grows; the lower p curves collapse fastest. A gray arrow labelled "every extra link shrinks the chance" points at the descending green curve, showing that even a strong p=0.90 decays badly once many tokens must all be right.
Why this exponent is the tool (not addition, not average): the task metric here is all-or-nothing — exact match. All-or-nothing = "AND across L steps" = multiplication, and multiplying L copies is exactly a power. That is why pL, and nothing else, is the right description.
Same rising p, two ways to score it — and they look completely different.
Figure s03 — one rising p, two rulers. The horizontal axis is per-token correctness p (rising smoothly from 0.5 to 1.0 as the model scales up). The orange solid curve is the sharp exact-match metric p20: it hugs the floor near zero and then leaps upward only near p≈0.9–1.0; a red arrow labelled "sharp metric leaps here" marks that late surge, with a gray dotted vertical guide at p=0.92. The green dashed line is the smooth per-token metric (∝p): a straight, gentle rise. Both are driven by the same smoothly increasing p — the visual proof that "emergence" can live entirely in the choice of ruler.
Why the topic needs both: the "mirage" explanation (Cross-entropy loss and perplexity measures things smoothly) says emergence lives in the ruler, not the model. You cannot follow that debate without seeing both lenses side by side.
Why: the whole surprise is a smooth loss curve (§7 power law) sitting under a sudden task curve (§7 phase transition). Knowing both shapes lets you hold the parent's central paradox: "Smooth Loss, Sharp Look."
Cover the right side and answer aloud; if you can, you're ready for the parent note.
What does one token represent, and what does the model output per token?
One text chunk; the model outputs a probability distribution over the next token.
What is N, in a picture?
Number of parameters — a wall of adjustable dials; more dials = larger model.
State the compute rule of thumb linking C, N, D, and where the 6 comes from.
C≈6ND; the 6 is 2 FLOPs for the forward pass plus 4 for the backward pass, per parameter per token.
What does p mean and what range does it live in?
Per-token probability of being correct, between 0 and 1.
Define Acc (exact-match accuracy).
Fraction of times the whole answer is produced with every token correct; one wrong token = whole answer wrong.
Why is exact-match accuracy pL and not Lp?
The chain rule multiplies L conditional probabilities (approximated as equal p), so probabilities multiply into a power; Lp counts expected correct tokens (a smooth average).
For 0<p<1, does raising to a power grow or shrink the number?
It shrinks — every extra factor of p leaks value.
What does taking log of pL achieve, and which base do we use?
Turns the product into Llogp (natural log, base e), exposing why small p gives Acc≈0 and p→1 makes it leap.
What is the difference between a power law and a phase transition curve?
Power law = smooth steady decline like A/Nα (loss); phase transition = flat then sudden leap past N∗ (task metric).
In L(N,D)=E+A/Nα+B/Dβ, what do α, β and E mean?
α,β are the decay exponents for parameters and data; E is the irreducible entropy floor a perfect model cannot beat.
Distinguish the two symbols written "L" in this topic.
Plain L = number of tokens a task needs; italic L(N,D) = the loss function.
