Emergent abilities in large models — Hinglish notes

6.1.3 · AI-ML › Scaling & Efficient Architectures

Emergent ability KYA hoti hai?

"Scale" ke teen axes jo emergence trigger kar sakte hain:

$N$ = parameters ki sankhya
$C$ = training compute (FLOPs) — aksar $C \approx 6 N D$ (params × tokens)
$D$ = training tokens ki sankhya

Curve KAISI dikhti hai? (Dual coding)

Figure — Emergent abilities in large models

Picture ko do overlaid kahaniyon ki tarah padho:

Cross-entropy loss (smooth blue): power law ki tarah decline karti hai, koi kink nahi.
Task accuracy (orange, S-shaped/hockey-stick): chance par pinned, phir $N^{*}$ ke paas phase transition.

Abilities emerge KYU karte hain? (Dono sides ko steel-man karo)

Iske do competing explanations hain. Ek achhha student dono ko saath rakhta hai.

Formula Metric-choice math (exact-match emergent kyun lagta hai)

YAHAN SE KYU SHURU KAREN: hum dikhana chahte hain ki ek smoothly-improving per-token quality ek abruptly-improving task metric produce kar sakta hai.

Maano ek task ko $L$ tokens ki zaroorat hai sab sahi. Maano per-token sahi hone ki probability $p (N)$ hai, jo scale $N$ ke saath smoothly badhti hai.

Exact-match accuracy (saare $L$ tokens sahi): $Acc (N) = p (N)^{L}$ YEH "jump" kaise create karta hai: logs lo. $lo g Acc = L lo g p (N)$ Jab $p$ chhota hota hai, $lo g p$ bahut negative hota hai, isliye $Acc \approx 0$ . Jaise $p \to 1$ , $lo g p \to 0$ aur $Acc$ 1 ki taraf shoot karta hai. $L$ jitna bada hoga, transition utni hi sharp hogi — ek number <1 ko badi power tak raise karne ki pure geometry.

Ab ek smooth metric use karo, jaise expected sahi tokens ki sankhya: $TokenScore (N) = L p (N)$ Yeh $p$ mein linear hai → smoothly improve karta hai, koi jump nahi. Same model, alag lens.

Derivation: scaling laws se connect karna

Worked examples (real reported emergent abilities)

Forecast-then-Verify

Common mistakes (Steel-manned)

Feynman

Recall Ek 12-saal ke bacche ko explain karo

Ek video game imagine karo jahan tum sirf tabhi jeetate ho jab 20 buttons ek row mein sahi dabao. Jab tum bure ho, shayad 10 mein se 8 buttons sahi lagate ho — lekin saare 20 sahi karna almost kabhi nahi hota, isliye tumhari win rate basically zero hai. Jaise thodi practice karo aur har button 99% time sahi ho, achanak zyaadatar games jeetne lagte ho! Tumne magically koi nayi power nahi seekhi — har button thoda better hua, lekin kyunki tumhe saare chahiye, chhote improvements "kabhi nahi jeeto" ko "almost hamesha jeeto" mein badal dete hain. Bade AI models aise hain: woh har word par thoda better hote hain, aur hard tasks jinhe har word sahi chahiye suddenly "click" kar jaate hain.

Recall — Active flashcards

Emergent ability kya define karta hai (Wei et al.)?

Woh capability jo chhote models mein absent hai, bade models mein present hai, jiski appearance small-model performance ko extrapolate karke predict nahi ki ja sakti.

Teen scale axes kaun se hain jo emergence trigger kar sakte hain?

Parameters

N

, training compute

C \approx 6 N D

, aur training tokens

D

Exact-match accuracy emergent kyun lagti hai jabki loss smooth lagti hai?

Accuracy

= p^{L}

ek smoothly-improving per-token correctness

p

ka non-linear (steep) transform hai; loss thresholded nahi hai.

Emergence ki "mirage" hypothesis kya hai?

Emergence discontinuous metrics ka artifact ho sakta hai; smooth metrics (edit distance, per-token log-likelihood) smooth, predictable improvement reveal karte hain (Schaeffer et al., 2023).

L

-token task ke liye task accuracy ko per-token correctness se link karne wala formula batao.

Acc = p^{L}

, isliye

lo g Acc = L lo g p

— bada

L

⇒ sharper transition.

Chain-of-thought large models ki help kyun karta hai lekin small models ko hurt karta hai?

Chhote models mein per-step reliability

p

low hoti hai;

k

steps chain karne par

p^{k}

milta hai jo shrink hota hai, isliye extra steps errors add karte hain. Bade models mein

p

itna high hota hai ki decomposition help karta hai.

Kya emergence scaling laws violate karta hai?

Nahi. Loss ek smooth law follow karta hai; downstream metric us smooth loss ka ek non-linear transform hai.

Chinchilla loss form?

L (N, D) = E + A / N^{α} + B / D^{β}

Emergence ek safety concern kyun hai?

Dangerous capabilities standard metrics par abruptly appear ho sakti hain; chhote models par smooth metrics use karna unhe early forecast karne mein help karta hai.

20-token task ke liye,

p = 0.80

p = 0.99

par roughly kya accuracy hai?

~0.012 vs ~0.818 — ek smooth per-token gain achanak jump jaisi lagti hai.

Connections

Scaling laws (Kaplan, Chinchilla)
Cross-entropy loss and perplexity
Chain-of-thought prompting
In-context learning
Inverse scaling
Grokking (delayed generalization)
Compute-optimal training
Phase transitions in physics