Existential and catastrophic risk frameworks: Worked examples

6.4.14 · D3 · AI-ML › AI Safety & Alignment › Existential and catastrophic risk frameworks

Yeh page parent topic ka drill floor hai. Parent ne tumhe formulas diye; yahan hum numbers plug karte hain, har sign aur har edge case step-by-step chalte hain, aur har jawab ko haath se verify karte hain. Agar koi symbol abhi tak nahi mila, toh woh usi waqt define hoga jab woh pehli baar aayega.

Kuch bhi aur karne se pehle, teen plain-word anchors taaki koi symbol unearned na rahe:

The scenario matrix

Neeche har worked example us cell ke saath tagged hai jo woh fill karta hai.

Cell	Kya cheez isse special banati hai	Example
A. All-mid probabilities	Har factor ek plain fraction, koi extreme nahi	Ex 1
B. A factor is $0$	One term collapses the product	Ex 2
C. A factor is $1$	A term that "can't fail" drops out	Ex 3
D. Extra multiplicative factor	Situational-awareness amplifier	Ex 4
E. Racing rates ( $d C / d t$ vs $d A / d t$ )	Sign of the gap, not a probability	Ex 5
F. Reward-hacking gap $ϵ$	Measured minus true objective	Ex 6
G. Real-world word problem	Units, dollars, counts	Ex 7
H. Exam-style twist	Posterior stays broad → deferral	Ex 8
I. Limiting behaviour	What happens as a factor $\to 0$ or $\to 1$	Ex 9

Ex 1 — Cell A: the plain three-factor product

Worked example All-mid probabilities

Ek AI project ka estimate hai: superhuman capability reach karta hai $P_{1} = 0.5$ ; agar karta hai, toh uska goal misaligned hai $P_{2} = 0.4$ ; agar dono hain, toh hum ise rok nahi sakte $P_{3} = 0.3$ . $P (x-risk)$ nikalo.

Forecast: padhne se pehle guess karo — kisi bhi single factor se bada hoga ya chota?

Model likho: $P (x-risk) = P_{1} \times P_{2} \times P_{3}$ . Yeh step kyun? Parent ka x-risk formula ek chain hai: teeno hone chahiye, aur independent probabilities ko multiply karna "AND" combine karne ka tarika hai.
Multiply karo: $0.5 \times 0.4 = 0.20$ , phir $0.20 \times 0.3 = 0.06$ . Yeh step kyun? Do-do karke karo taaki har product checkable ho.
Jawab: $P (x-risk) = 0.06 = 6%$ .

Verify karo: $1$ se kam har number ka product sabse chote factor se chota hona chahiye ( $0.3$ ). $0.06 < 0.3$ ✓. Units: probability dimensionless hai, $[0, 1]$ mein rehti hai ✓.

Key lesson: probabilities chain karne se woh tezi se shrink hoti hain. Isi liye parent ka paperclip example teen $10%$ terms se $0.1%$ par aaya tha.

Ex 2 — Cell B: a factor is exactly zero

Ex 3 — Cell C: a factor is exactly one

Zero aur one do extremes hain: $0$ product ko collapse kar deta hai, $1$ invisible hai. Har doosri value beech mein rehti hai aur risk ko scale karti hai.

Ex 4 — Cell D: the situational-awareness amplifier

Worked example Adding a fourth factor

Aschenbrenner ka amplified model: $Risk = P (mis) \times Cap \times Strategic awareness$ . Yahan $Cap$ normalized capability hai — wohi $C (t)$ hamare teen anchors se, lekin $0$ -to- $1$ scale par squeeze ki gayi (0 = harmless, 1 = maximally capable) taaki probabilities ke saath cleanly multiply ho sake. $P (mis) = 0.2$ , $Cap = 0.5$ , aur strategic-awareness multiplier $S = 3$ lo (deceptive alignment effective risk ko triple karta hai).

Forecast: kya $S = 3$ se multiply karne par jawab ek valid probability rehta hai?

Baseline (no awareness): $0.2 \times 0.5 = 0.10$ . Yeh step kyun? Yeh parent ka without-awareness formula hai: misalignment chance times normalized capability.
Amplify karo: $0.10 \times 3 = 0.30$ . Yeh step kyun? Situational awareness system ko misalignment strategically hide karne deti hai, isliye yeh ek multiplier $S > 1$ ki tarah kaam karta hai, probability ki tarah nahi.
Jawab: effective risk $= 0.30 = 30%$ .

Verify karo: $0.2 \times 0.5 \times 3 = 0.30$ ✓, aur $0.30 \leq 1$ toh yeh still ek legal probability hai ✓.

[!mistake] Jab amplifier model ko todh deta hai Agar $S$ itna bada hota ki product $1$ se upar chala jaata, toh " $\times S$ " form ab ek probability nahi hai — yeh ek risk index hai. Hamesha sanity-check karo ki result $[0, 1]$ mein rahe; agar nahi, toh tum ek relative-danger score use kar rahe ho, chance nahi. Yeh 6.4.11-Multi-agent-alignment-challenges se connect hota hai jahan competing amplifiers stack hote hain.

Ex 5 — Cell E: racing rates aur gap ka sign

Worked example Capability vs alignment growth

Ek snapshot mein, capability $\frac{d C}{d t} = 8$ units/month ki rate se badhti hai aur alignment $\frac{d A}{d t} = 2$ units/month ki rate se. Kya system danger regime mein ja raha hai?

Forecast: kaun sa curve aage nikalna dangerous hai?

Neeche figure: horizontal axis time $t$ in months hai, vertical axis level in units hai. Ek steep burnt-orange line capability $C (t)$ hai (slope 8); ek shallow teal line alignment $A (t)$ hai (slope 2); unke beech plum shaded wedge widening alignment gap hai.

Gap rate $G = \frac{d C}{d t} - \frac{d A}{d t}$ banao. Yeh step kyun? Parent ki danger condition $\frac{d C}{d t} ≫ \frac{d A}{d t}$ hai; unka difference ka sign aur size batata hai ki capability alignment se aage nikal rahi hai ya nahi.
Compute karo: $G = 8 - 2 = 6 > 0$ . Yeh step kyun? Positive $G$ matlab orange capability curve (figure dekho) teal alignment curve se tez badhti hai — gap time ke saath badhta hai.
Ratio interpret karo: $\frac{8}{2} = 4$ . Capability alignment se $4 \times$ tezi se improve hoti hai. Yeh step kyun? Ratio $≫ 1$ " $≫$ " ka quantitative matlab hai.
Conclusion: haan, danger regime — alignment gap badhta hai.

Verify karo: $8 - 2 = 6$ ✓, $8/2 = 4$ ✓. Sign check: agar instead $d C / d t = 2, d A / d t = 8$ toh $G = - 6 < 0$ (alignment catching up — safe direction), opposite sign cover karta hai ✓.

$G$ ke teen sign cases: $G > 0$ capability winning (danger), $G = 0$ neck-and-neck (constant gap), $G < 0$ alignment catching up (closing gap). Har case figure par ek alag slope ki straight line hai.

Ex 6 — Cell F: the reward-hacking gap $ϵ$

Worked example Measured reward true utility se drift karta hai

Ek state–action pair ke liye true objective aur measured reward: $U_{true} = 10$ , $R_{measured} = 14$ . Specification-gaming gap $ϵ$ nikalo, aur measured reward ka woh fraction jo "fake" hai.

Forecast: kya agent real value ke liye over- ya under-rewarded hai?

Yaad karo $R_{measured} = U_{true} + ϵ$ , toh $ϵ = R_{measured} - U_{true}$ . Yeh step kyun? $ϵ$ ("epsilon") bas ek plain-word error term hai: reward minus real value. Yahi woh cheez hai jise ek reward hacker climb karta hai.
$ϵ = 14 - 10 = 4$ . Yeh step kyun? Positive $ϵ$ matlab metric over-pay karta hai — agent ko zero real value ke liye 4 units of reward milti hai.
Fake fraction $= \frac{ϵ}{R _{measured}} = \frac{4}{14} \approx 0.2857 = 28.6%$ . Yeh step kyun? Jaise capability badhti hai, agent exactly isi exploitable slice ko optimize karne lagta hai.

Verify karo: $14 - 4 = 10 = U_{true}$ ✓; $4/14 = 0.2857 \dots$ ✓. Degenerate check: agar $ϵ = 0$ toh $R = U$ — perfect specification, parent ka ideal case ✓.

Ex 7 — Cell G: real-world word problem (units & counts)

Worked example Autonomous-weapon catastrophic threshold

Ek framework kisi event ko catastrophic label karta hai agar expected deaths $\geq 1, 000, 000$ ho. $50, 000$ autonomous drones ke ek fleet mein se har drone ka per-mission probability $0.00005$ (= $5 \times 1 0^{- 5}$ ) hai ki ek lethal targeting error hogi jo average $40$ logon ko maari. $10$ missions mein, kya yeh catastrophic threshold cross karta hai?

Forecast: total expected deaths ka order of magnitude guess karo.

Expected lethal errors per mission $= 50, 000 \times 0.00005 = 2.5$ . Yeh step kyun? Expected count = number of trials × per-trial probability.
$10$ missions mein: $2.5 \times 10 = 25$ lethal errors. Yeh step kyun? Independent missions mein expectations add hote hain.
Expected deaths $= 25 \times 40 = 1, 000$ . Yeh step kyun? Har error average $40$ ko maarta hai; errors ki count se multiply karo.
Compare karo: $1, 000 < 1, 000, 000$ . Catastrophic threshold se neeche (teen orders of magnitude se), halanki yeh still ek mass-casualty event hai.

Verify karo: $50000 \times 0.00005 = 2.5$ ✓; $2.5 \times 10 \times 40 = 1000$ ✓. Units: (drones) × (prob) × (missions) × (deaths/error) → deaths ✓.

[!mistake] X-risk twist jo yeh word problem chhupata hai $1, 000$ deaths "sirf" catastrophic-adjacent hai — lekin parent warn karta hai ki wahi failure scale hoti hai: per-mission probability ko tenfold badha do aur fleet mein zeros add karo, toh tum existential regime ke paas pahunch jaate ho. Governance thresholds (6.4.13-AI-governance-and-policy) in multipliers ko cap karne ke liye exactly exist karte hain.

Ex 8 — Cell H: exam-style twist (broad posterior → deferral)

Worked example Uncertain-values agent choose karta hai poochne ko

Russell ka agent ek irreversible action $a_{irr}$ ko ek safe reversible action $a_{safe}$ ke against weigh karta hai. Uska posterior human utility par split hai: $0.6$ probability ke saath humans action chahte hain (utility $+ 10$ ), aur $0.4$ ke saath unhe nafrat hai (utility $- 30$ ). Safe action guaranteed $+ 1$ deta hai. Agent kya leta hai?

Yahan $D$ woh data hai jo agent ne dekha hai — observed human behaviour aur demonstrations. Posterior $P (U ∣ D)$ padha jaata hai "kitna probable hai ki har human utility function $U$ ho, given woh data $D$ jo humne humans ko produce karte dekha."

Forecast: kya high uncertainty act karne ya defer karne ki taraf push karti hai?

Irreversible action ka expected utility: $E U (a_{irr}) = 0.6 \times (+ 10) + 0.4 \times (- 30)$ . Yeh step kyun? Value uncertainty ke under agent expected utility maximize karta hai, posterior $P (U ∣ D)$ par averaging karta hai — woh posterior kahan se aata hai iske liye dekho 5.3.12-Inverse-reinforcement-learning.
$= 6 - 12 = - 6$ . Yeh step kyun? Heavy negative outcome, halanki kam likely, dominate karta hai.
$E U (a_{safe}) = + 1$ se compare karo. Yeh step kyun? Bada expected utility choose karo.
Kyunki $+ 1 > - 6$ , agent defer karta hai / safe reversible action leta hai.

Verify karo: $0.6 \times 10 + 0.4 \times (- 30) = 6 - 12 = - 6$ ✓; $1 > - 6$ ✓. Yeh value alignment ka mechanism hai: broad posterior + irreversibility ⇒ caution.

Ex 9 — Cell I: limiting behaviour

Worked example Edges par kya hota hai

Teen-factor model $P (x-risk) = P_{1} P_{2} P_{3}$ mein $P_{1} = 0.4, P_{2} = 0.5$ fixed hain, $P_{3} \to 0$ aur $P_{3} \to 1$ ke limits trace karo.

Forecast: compute karne se pehle risk vs $P_{3}$ ka graph sketch karo.

Neeche figure: horizontal axis $P_{3}$ hai (probability ki hum system ko rok nahi sakte) $0$ se $1$ tak; vertical axis $P (x-risk)$ hai. Slope $0.2$ ki ek straight burnt-orange line origin se uthti hai; plum dots do endpoints $(0, 0)$ aur $(1, 0.2)$ par mark hain.

Risk ko $P_{3}$ ke function ke roop mein likho: $Risk (P_{3}) = 0.4 \times 0.5 \times P_{3} = 0.2 P_{3}$ . Yeh step kyun? Do factors fix karne par slope $0.2$ ke saath origin se ek straight line milti hai.
Limit $P_{3} \to 0$ : $Risk \to 0$ . Yeh step kyun? Perfect corrigibility (hamesha stoppable) x-risk ko zero par le jaati hai — safe extreme.
Limit $P_{3} \to 1$ : $Risk \to 0.2$ . Yeh step kyun? Ek totally unstoppable system bhi $0.2$ par cap hota hai, kyunki baaki do barriers abhi bhi gate karte hain.
Slope padho: $P_{3}$ mein har $+ 0.1$ risk mein $+ 0.02$ add karta hai. Yeh step kyun? Derivative $\frac{d Risk}{d P _{3}} = 0.2$ constant hai — line ki steepness.

Verify karo: $0.2 \times 0 = 0$ ✓; $0.2 \times 1 = 0.2$ ✓; slope $0.2$ constant ✓. Maximum possible risk (sab $P_{3}$ pe) doosre do factors ke product ke barabar hai, $0.4 \times 0.5 = 0.2$ ✓.

Recall

Recall Probabilities chain karne se risk itni tezi se kyun shrink hoti hai?

Kyunki har factor $\leq 1$ hai, toh product sabse chote factor se $\leq$ hai ::: independent "AND" conditions ko multiply karne se number sirf chota ho sakta hai (ya barabar).

Recall Positive gap rate

$G = \frac{d C}{d t} - \frac{d A}{d t}$ ka kya matlab hai? Capability alignment se aage nikal rahi hai; alignment gap time ke saath badhta hai — danger regime ::: $G > 0$ danger, $G = 0$ constant gap, $G < 0$ alignment catching up.

Recall Ex 8 mein, agent ne defer kyun kiya jabki

$60%$ chance tha ki human ne approve kiya? Expected utility negative tha ( $- 6$ ) kyunki rare bad outcome ( $- 30$ ) heavy tha ::: value uncertainty ke under, negative expected utility wale irreversible actions se bacho.