Existential and catastrophic risk frameworks: Foundations

6.4.14 · D1 · AI-ML › AI Safety & Alignment › Existential and catastrophic risk frameworks

Pehle aapko existential and catastrophic risk frameworks ki parent note padhne se pehle uski language padhni hogi. Yeh page assume karta hai ki aapne isme se kuch bhi nahi dekha. Hum har symbol ek picture se banate hain.

Part 1 — Maths se pehle ke words

Figure s01 — Ek square map. Horizontal axis = severity $S$ (kitna kuch lost hota hai), vertical axis = probability $P$ (kitna likely). Bottom-left = mild aur rare; top-left = mild aur common (toota hua angootha); right strip "Catastrophic — huge but recoverable" label ke saath diagonal lines se hatched hai; aur far bottom-right corner "Existential — permanent, no recovery" label ke saath cross-hatching se hatched hai. Agar image load na ho, toh high-severity (right) edge par do named danger-strips imagine karo, jisme existential extreme corner hai.

Figure dekho. Horizontal axis severity $S$ hai (kitna kuch lost hota hai). Vertical axis probability $P$ hai (kitna likely). Do special regions named hain — har ek colour-coded bhi hai, hatched bhi hai, aur text-labelled bhi hai, taki aap bina colour rely kiye padh sako:

Catastrophic risk — bilkul right (bada $S$ ) lekin humanity survive karti hai aur recover kar sakti hai. Diagonally-hatched "Catastrophic" strip.
Existential risk (x-risk) — woh far corner jahan $S$ itna total hai ki recovery impossible hai: extinction, ya permanently ek bure future mein locked hona. Cross-hatched "Existential" corner.

Part 2 — Probability symbols

Aage ki har cheez probability notation se banti hai. Hum ise ek baar, ek picture ke saath define karte hain.

Figure s02 — Ek rectangle labelled "all possible futures". Ek dashed-outline circle $A$ = "loss of control" aur ek solid-outline circle $B$ = "catastrophe" overlap karte hain. Overlap (woh $A \land B$ lens) hatched hai aur " $P (B ∣ A)$ = overlap as a fraction of A" label kiya gaya hai. Fallback: woh shaded lens jahan dono circles milte hain, sirf circle $A$ ke relative measure kiya gaya, conditional probability hai.

Figure mein, bada box "all possible futures" hai. Circle $A$ (dashed outline) "loss of control happened" hai. $P (A)$ hai kitna box $A$ cover karta hai. Bar $∣$ kehta hai: $A$ ke bahar sab kuch ignore karo, phir pucho ki $A$ ka kitna fraction catastrophe $B$ (solid outline) bhi hai. Woh hatched overlap — woh $A \land B$ lens — sirf $A$ ke ek slice ke roop mein measure kiya gaya, $P (B ∣ A)$ hai.

Part 3 — Capability, alignment, aur unka gap

Change ke symbols: $t$ , $\frac{d}{d t}$ , aur $≫$

"Capability alignment se aage nikal jaati hai" precisely kehne ke liye, parent rates of change use karta hai. Yeh toolkit zero se hai.

Figure s03 — Do curves time $t$ ke upar. Ek solid orange curve $C (t)$ (capability) exponentially upar jaati hai; ek dashed violet curve $A (t)$ (alignment) dheere dheere upar jaati hai. Ek marked time par $C (t)$ par ek steep arrow aur $A (t)$ par ek almost-flat arrow unke slopes dikhate hain; unke beech ka vertical gap "the gap = risk" label kiya gaya hai. Fallback: capability steeply shoot karta hai jabki alignment crawl karti hai, aur unke beech ki barhti doori danger hai.

Figure mein, solid orange curve $C (t)$ (capability) hai aur dashed violet curve $A (t)$ (alignment) hai — line style aur colour dono unhe alag karte hain. Marked time par, orange slope arrow violet se kaafi zyada steep hai. Slopes ke beech ka woh gap hi woh risk hai jo parent draw karta hai.

Part 4 — Exponential growth aur "takeoff"

Formula "Threshold" kahan se aata hai — pure model

Recurrence unroll karo: $n$ steps baad, $I_{n} = I_{0} (1 + r)^{n}$ . Us power ka base $1 + r$ hai, aur poora behavior is par depend karta hai ki woh base $1$ ke upar hai ya neechehai:

Agar $r > 0$ toh $1 + r > 1$ . Har step ek one se bade number se multiply karta hai, toh $I_{n}$ bina bound ke badhta hai jaise $n$ badhta hai — runaway takeoff.
Agar $r = 0$ toh $1 + r = 1$ . Har step exactly ek se multiply karta hai, yaani kuch nahi badlata, toh $I_{n} = I_{0}$ hamesha — no takeoff.
Agar $r < 0$ (system khud ko degrade karta hai) toh $1 + r < 1$ aur $I_{n}$ $0$ ki taraf shrink karta hai.

$r = 0$ critical point kyun hai (aur sirf "trivial" kyun nahi)? Kyunki $r = 0$ exact tipping value hai jahan $(1 + r)$ se multiply karna shrinking se growing mein switch karta hai. Iске neecheSystem fade karta hai; iके upar system explode karta hai; sirf is par system perfectly still rehta hai. Tipping point precisely woh value hoti hai jahan qualitative behaviour flip karta hai, aur yahan woh flip tab hota hai jab base $1 + r$ $1$ cross karta hai — yaani jab $r$ $0$ cross karta hai.

Part 5 — Reward, objective, aur sneaky $ϵ$

Safety-problems framework ko goal-chasing agents ki vocabulary chahiye.

Part 6 — Uncertainty aur value learning

Russell ka framework insist karta hai ki AI human goals ke baare mein unsure rahe.

Definition Discount factor

$γ$ aur action-value $Q (s, a)$ Yeh model karne ke liye ki ek human actions kaise choosta hai, humein score chahiye ki har action long-run mein kitna acha hai. Lekin infinitely many future steps par "long-run" infinity tak add ho sakta hai aur meaningless ho jaata hai — toh hum har future step ka contribution discount factor $γ$ (Greek "gamma") se shrink karte hain, ek number jisme $0 \leq γ < 1$ hai. $k$ steps baad ka reward sirf $γ^{k}$ face value count karta hai; kyunki $γ < 1$ , woh weights zero ki taraf fade karte hain aur infinite sum finite rehta hai (well-posed). Phir $Q (s, a)$ = action-value function: discounted total future utility jo aap expect karte ho agar state $s$ se action $a$ lo aur phir acha behave karo — $Q (s, a) = E [\sum_{k = 0}^{\infty} γ^{k} U (s_{k}, a_{k}) s_{0} = s, a_{0} = a] .$ Bada $Q$ = better move (sirf immediate payoff nahi balki jo kuch bhi lead karta hai, door ke steps discounted ke saath count karta hai). $Q$ ko move ka "wisdom score" samjho. (Agar task sirf fixed number of steps mein khatam ho jaata hai — finite horizon — toh sum pehle se finite hai aur $γ$ $1$ set kiya ja sakta hai.)

Definition

$π_{H}$ , $\propto$ , $β$ , aur Boltzmann rationality

$π_{H} (a ∣ s)$ = human policy: woh probability ki ek human demonstrator, state $s$ mein, action $a$ choosega. (Part 5 ke agent ke $π$ jaisi hi idea, lekin human ke liye — subscript $H$ "human" mark karta hai.)
$\propto$ matlab "proportional to" — ek scaling constant tak equal jo hume track nahi karna.
$π_{H} (a ∣ s) \propto exp (β \cdot Q (s, a))$ mein, human ko model kiya gaya hai ki usually high- $Q$ actions choose karta hai, perfectly nahi. $exp$ kisi bhi $Q$ -score ko positive weight mein baadalata hai; higher- $Q$ actions ko exponentially zyada probability milti hai.
$β$ (Greek "beta") rationality knob hai: bada $β$ → almost perfect human (hamesha top- $Q$ action); chhota $β$ → noisy, error-prone human. Yeh model ko human galtiyan maaf karne deta hai.

Yahi machinery exactly 5.3.12-Inverse-reinforcement-learning hai — behavior se goal infer karna yeh pooch ke ki "kaun sa utility $U$ (aur is tarah kaun sa $Q$ ) in observed actions $π_{H}$ ko sensible dikhayega?"

Part 7 — Instrumental convergence & situational awareness

Definition

$P (misalignment)$ aur strategic-awareness factor $W$ Parent ka amplified risk formula teen symbols use karta hai jo hum ab precisely name karte hain:

$P (misalignment)$ = woh probability ki system ka goal hamare se kam ho. Kyunki misalignment Part 3 se shortfall $1 - A$ hai, yeh chance hai ki $A < 1$ — chance ki system perfectly aligned nahi hai.
$C$ = capability (Part 3).
$W$ = strategic awareness: ek multiplier $\geq 1$ jo measure karta hai ki AI apni khud ki situation (ki use train, test, shayad shut off kiya ja raha hai) kitna samajhta hai aur us samajh ka faayda utha sakta hai. $W = 1$ matlab "koi self-awareness nahi, koi amplification nahi"; bada $W$ matlab AI scheme kar sakta hai.

$Risk = P (misalignment) \times C \times W$ Situational awareness ke bina, $W = 1$ aur formula parent ke simple $P (misalignment) \times C$ mein reduce ho jaata hai. Jab AI self-understanding gain karta hai, $W > 1$ danger ko multiply karta hai — kyunki ek misaligned system jo yeh bhi jaanta hai ki use dekha ja raha hai, testing ke dauran apna asli goal chhupa sakta hai (deceptive alignment) aur correction resist kar sakta hai. Jab kai aise systems compete karte hain, dekho 6.4.11-Multi-agent-alignment-challenges; rules-and-institutions response ke liye, 6.4.13-AI-governance-and-policy.

Prerequisite map

Equipment checklist

Right side cover karo aur zor se jawab do. Agar koi stumps kare, toh parent note kholne se pehle uska part upar dobara padho.

P (B ∣ A)

ka plain words mein kya matlab hai?

B

ka chance given ki

A

pehle ho chuka hai — sirf

A

ke ek slice ke roop mein measure kiya gaya, sab kuch mein se nahi.

Joint event

A \land B

ke liye product rule bolo.

P (A \land B) = P (A) \times P (B ∣ A)

—

A

mein ghuso, phir

A

ke andar

B

mein bhi ghuso.

Hum x-risk formula mein terms multiply kyun karte hain?

Kyunki saare events saath hone chahiye (AND); product rule use karke overlap milta hai, aur yeh total ko sirf shrink karta hai.

Catastrophic aur existential risk mein kya fark hai?

Catastrophic = bada lekin recoverable nuksaan; existential = humanity ke future ka permanent nuksaan, koi recovery nahi.

Derivative

\frac{d C}{d t}

kya measure karta hai?

Capability curve ka slope — capability level ka nahi, balki capability kitni tezi se badh rahi hai per unit time.

≫

use karke danger condition bolo.

\frac{d C}{d t} ≫ \frac{d A}{d t}

— capability alignment se bahut zyada tezi se badhti hai.

I_{t + 1} = I_{t} (1 + r)

mein,

r = 0

tipping point kyun hai?

Kyunki growth base

1 + r

hai;

r = 0

par yeh

1

ke barabar hai (koi change nahi), iके upar

1

se bada (runaway), iके neechे

1

se chhota (decay) — qualitative behaviour wahan flip karta hai.

Friction

d

ke saath, loop kab runaway karta hai?

Base

1 + r - d

ban jaata hai, toh runaway ke liye

r > d

chahiye: improvement ko decay se aage hona chahiye.

s

a

, aur

π

kya hain?

State (world snapshot), action (ek move), policy (woh strategy jo states ko actions se map karti hai).

R_{measured} = U_{true} + ϵ

mein

ϵ (s, a)

kya hai?

Woh gap jo hum measure kar sakte hain aur jo asli value hum chahte hain ke beech — woh crack jise reward hacking exploit karta hai.

ar g max_{π} E [R]

kya poochta hai?

Woh policy

π

jo sabse zyada average reward produce kare.

P (U ∣ D)

mein random variable

U

kaunsa object hai?

Ek poori utility function — ek rule jo har

(s, a)

ko ek real-number goodness se map karta hai; hum uncertain hain ki kaun sa rule human values match karta hai.

Q (s, a)

ko discount factor

γ

kyun chahiye?

Kyunki infinitely many future steps par utility sum karna infinite ho sakta hai;

0 \leq γ < 1

ke saath har step-

k

reward sirf

γ^{k}

count karta hai, toh sum finite rehta hai. (Finite horizon

γ = 1

allow karta hai.)

π_{H} (a ∣ s)

kya hai?

Human policy — woh probability ki ek human demonstrator state

s

mein action

a

choose karta hai.

\propto

ka kya matlab hai, aur

β

kya control karta hai?

"Proportional to";

β

rationality knob hai jo model karta hai ki human kitna close-to-optimal (high-

Q

) assume kiya gaya hai.

Alignment

A

se misalignment kaise relate karta hai?

Misalignment shortfall

1 - A

hai; "high misalignment" aur "low

A

" ek hi baat kehte hain.

Amplified risk formula mein

W

kya hai?

Strategic awareness, ek multiplier

\geq 1

;

W = 1

matlab koi self-awareness nahi,

W > 1

AI ko scheme aur hide karne deke danger multiply karta hai.

Instrumental convergence ek sentence mein bolo.

Almost kisi bhi final goal ko same sub-goals serve karte hain — resources acquire karo,