Existential and catastrophic risk frameworks: Worked examples

This page is the drill floor for the parent topic. The parent gave you the formulas; here we plug in numbers, walk every sign and every edge case, and check each answer by hand. If you have not yet met a symbol here, it is defined the moment it appears.

Before anything, three plain-word anchors so no symbol is unearned:

The scenario matrix

Every worked example below is tagged with the cell it fills.

Cell	What makes it special	Example
A. All-mid probabilities	Each factor a plain fraction, none extreme	Ex 1
B. A factor is $0$	One term collapses the product	Ex 2
C. A factor is $1$	A term that "can't fail" drops out	Ex 3
D. Extra multiplicative factor	Situational-awareness amplifier	Ex 4
E. Racing rates ( $d C / d t$ vs $d A / d t$ )	Sign of the gap, not a probability	Ex 5
F. Reward-hacking gap $ϵ$	Measured minus true objective	Ex 6
G. Real-world word problem	Units, dollars, counts	Ex 7
H. Exam-style twist	Posterior stays broad → deferral	Ex 8
I. Limiting behaviour	What happens as a factor $\to 0$ or $\to 1$	Ex 9

Ex 1 — Cell A: the plain three-factor product

The key lesson: chaining probabilities shrinks them fast. This is why the parent's paperclip example landed at $0.1%$ from three $10%$ terms.

Ex 2 — Cell B: a factor is exactly zero

Ex 3 — Cell C: a factor is exactly one

Zero and one are the two extremes: $0$ collapses the product, $1$ is invisible. Every other value lives between and scales the risk.

Ex 4 — Cell D: the situational-awareness amplifier

Worked example Adding a fourth factor

Aschenbrenner's amplified model: $Risk = P (mis) \times Cap \times Strategic awareness$ . Here $Cap$ is normalized capability — the same $C (t)$ from our three anchors, but squeezed onto a $0$ -to- $1$ scale (0 = harmless, 1 = maximally capable) so it can multiply cleanly with probabilities. Take $P (mis) = 0.2$ , $Cap = 0.5$ , and strategic-awareness multiplier $S = 3$ (deceptive alignment triples effective risk).

Forecast: does multiplying by $S = 3$ keep the answer a valid probability?

Baseline (no awareness): $0.2 \times 0.5 = 0.10$ . Why this step? This is the parent's without-awareness formula: misalignment chance times normalized capability.
Amplify: $0.10 \times 3 = 0.30$ . Why this step? Situational awareness lets the system strategically hide misalignment, so it acts as a multiplier $S > 1$ , not a probability.
Answer: effective risk $= 0.30 = 30%$ .

Verify: $0.2 \times 0.5 \times 3 = 0.30$ ✓, and $0.30 \leq 1$ so it is still a legal probability ✓.

[!mistake] When the amplifier breaks the model If $S$ were large enough to push the product above $1$ , the " $\times S$ " form is no longer a probability — it is a risk index. Always sanity-check that the result stays in $[0, 1]$ ; if not, you are using a relative-danger score, not a chance. This connects to 6.4.11-Multi-agent-alignment-challenges where competing amplifiers stack.

Ex 5 — Cell E: racing rates and the sign of the gap

Worked example Capability vs alignment growth

At a snapshot, capability grows at $\frac{d C}{d t} = 8$ units/month and alignment at $\frac{d A}{d t} = 2$ units/month. Is the system entering the danger regime?

Forecast: which curve pulling ahead is the dangerous one?

Figure below: horizontal axis is time $t$ in months, vertical axis is level in units. A steep burnt-orange line is capability $C (t)$ (slope 8); a shallow teal line is alignment $A (t)$ (slope 2); the plum shaded wedge between them is the widening alignment gap.

Form the gap rate $G = \frac{d C}{d t} - \frac{d A}{d t}$ . Why this step? The parent's danger condition is $\frac{d C}{d t} ≫ \frac{d A}{d t}$ ; the sign and size of their difference tells us if capability is outrunning alignment.
Compute: $G = 8 - 2 = 6 > 0$ . Why this step? Positive $G$ means the orange capability curve (look at the figure) climbs faster than the teal alignment curve — the gap widens over time.
Interpret the ratio: $\frac{8}{2} = 4$ . Capability improves $4 \times$ faster than alignment. Why this step? A ratio $≫ 1$ is the quantitative meaning of " $≫$ ".
Conclusion: yes, danger regime — the alignment gap grows.

Verify: $8 - 2 = 6$ ✓, $8/2 = 4$ ✓. Sign check: if instead $d C / d t = 2, d A / d t = 8$ then $G = - 6 < 0$ (alignment catching up — safe direction), covering the opposite sign ✓.

The three sign cases of $G$ : $G > 0$ capability winning (danger), $G = 0$ neck-and-neck (constant gap), $G < 0$ alignment catching up (closing gap). Every case is a straight line of a different slope on the figure.

Ex 6 — Cell F: the reward-hacking gap $ϵ$

Worked example Measured reward drifts from true utility

True objective and measured reward for one state–action pair: $U_{true} = 10$ , $R_{measured} = 14$ . Find the specification-gaming gap $ϵ$ , and the fraction of measured reward that is "fake".

Forecast: is the agent over- or under-rewarded for real value?

Recall $R_{measured} = U_{true} + ϵ$ , so $ϵ = R_{measured} - U_{true}$ . Why this step? $ϵ$ ("epsilon") is just the plain-word error term: reward minus real value. It is what a reward hacker climbs.
$ϵ = 14 - 10 = 4$ . Why this step? Positive $ϵ$ means the metric over-pays — the agent gets 4 units of reward for zero real value.
Fake fraction $= \frac{ϵ}{R _{measured}} = \frac{4}{14} \approx 0.2857 = 28.6%$ . Why this step? As capability rises, the agent optimizes toward exactly this exploitable slice.

Verify: $14 - 4 = 10 = U_{true}$ ✓; $4/14 = 0.2857 \dots$ ✓. Degenerate check: if $ϵ = 0$ then $R = U$ — perfect specification, the parent's ideal case ✓.

Ex 7 — Cell G: real-world word problem (units & counts)

Worked example Autonomous-weapon catastrophic threshold

A framework labels an event catastrophic if expected deaths $\geq 1, 000, 000$ . A fleet of $50, 000$ autonomous drones each has a $0.00005$ (= $5 \times 1 0^{- 5}$ ) per-mission probability of a lethal targeting error killing $40$ people on average. Over $10$ missions, does this cross the catastrophic threshold?

Forecast: guess the order of magnitude of total expected deaths.

Expected lethal errors per mission $= 50, 000 \times 0.00005 = 2.5$ . Why this step? Expected count = number of trials × per-trial probability.
Over $10$ missions: $2.5 \times 10 = 25$ lethal errors. Why this step? Expectations add across independent missions.
Expected deaths $= 25 \times 40 = 1, 000$ . Why this step? Each error kills $40$ on average; multiply by count of errors.
Compare: $1, 000 < 1, 000, 000$ . Below the catastrophic threshold (by three orders of magnitude), though still a mass-casualty event.

Verify: $50000 \times 0.00005 = 2.5$ ✓; $2.5 \times 10 \times 40 = 1000$ ✓. Units: (drones) × (prob) × (missions) × (deaths/error) → deaths ✓.

[!mistake] The x-risk twist this word problem hides $1, 000$ deaths is "merely" catastrophic-adjacent — but the parent warns the same failure scales: raise per-mission probability tenfold and add zeros to the fleet, and you approach the existential regime. Governance thresholds (6.4.13-AI-governance-and-policy) exist precisely to cap these multipliers.

Ex 8 — Cell H: exam-style twist (broad posterior → deferral)

Worked example Uncertain-values agent chooses to ask

Russell's agent weighs an irreversible action $a_{irr}$ against a safe reversible one $a_{safe}$ . Its posterior over human utility splits: with probability $0.6$ humans want the action (utility $+ 10$ ), with $0.4$ they hate it (utility $- 30$ ). The safe action yields $+ 1$ guaranteed. Which does the agent take?

Here $D$ is the data the agent has seen — the observed human behaviour and demonstrations. The posterior $P (U ∣ D)$ reads "how probable each human utility function $U$ is, given the data $D$ we watched humans produce."

Forecast: does high uncertainty push toward acting or deferring?

Expected utility of the irreversible action: $E U (a_{irr}) = 0.6 \times (+ 10) + 0.4 \times (- 30)$ . Why this step? Under value uncertainty the agent maximizes expected utility, averaging over the posterior $P (U ∣ D)$ — see 5.3.12-Inverse-reinforcement-learning for where that posterior comes from.
$= 6 - 12 = - 6$ . Why this step? The heavy negative outcome, though less likely, dominates.
Compare with $E U (a_{safe}) = + 1$ . Why this step? Choose the larger expected utility.
Since $+ 1 > - 6$ , the agent defers / takes the safe reversible action.

Verify: $0.6 \times 10 + 0.4 \times (- 30) = 6 - 12 = - 6$ ✓; $1 > - 6$ ✓. This is the mechanism of value alignment: broad posterior + irreversibility ⇒ caution.

Ex 9 — Cell I: limiting behaviour

Worked example What happens at the edges

In the three-factor model $P (x-risk) = P_{1} P_{2} P_{3}$ with $P_{1} = 0.4, P_{2} = 0.5$ fixed, trace the limits as $P_{3} \to 0$ and as $P_{3} \to 1$ .

Forecast: sketch the graph of risk vs $P_{3}$ before computing.

Figure below: horizontal axis is $P_{3}$ (probability we can't stop the system) from $0$ to $1$ ; vertical axis is $P (x-risk)$ . A straight burnt-orange line of slope $0.2$ rises from the origin; plum dots mark the two endpoints at $(0, 0)$ and $(1, 0.2)$ .

Write risk as a function of $P_{3}$ : $Risk (P_{3}) = 0.4 \times 0.5 \times P_{3} = 0.2 P_{3}$ . Why this step? Fixing two factors leaves a straight line through the origin with slope $0.2$ .
Limit $P_{3} \to 0$ : $Risk \to 0$ . Why this step? Perfect corrigibility (always stoppable) drives x-risk to zero — the safe extreme.
Limit $P_{3} \to 1$ : $Risk \to 0.2$ . Why this step? Even a totally unstoppable system caps at $0.2$ , because the other two barriers still gate it.
Read the slope: each $+ 0.1$ in $P_{3}$ adds $+ 0.02$ to risk. Why this step? The derivative $\frac{d Risk}{d P _{3}} = 0.2$ is constant — the line's steepness.

Verify: $0.2 \times 0 = 0$ ✓; $0.2 \times 1 = 0.2$ ✓; slope $0.2$ constant ✓. The maximum possible risk (all $P_{3}$ ) equals the product of the other two factors, $0.4 \times 0.5 = 0.2$ ✓.

Recall

Recall Why does chaining probabilities shrink risk so fast?

Because each factor is $\leq 1$ , so the product is $\leq$ the smallest factor ::: multiplying independent "AND" conditions can only make the number smaller (or equal).

Recall What does a positive gap rate

$G = \frac{d C}{d t} - \frac{d A}{d t}$ mean? Capability is outrunning alignment; the alignment gap widens over time — the danger regime ::: $G > 0$ danger, $G = 0$ constant gap, $G < 0$ alignment catching up.

Recall In Ex 8, why did the agent defer despite a 60% chance the human approved?

The expected utility was negative ( $- 6$ ) because the rare bad outcome ( $- 30$ ) was heavy ::: under value uncertainty, avoid irreversible actions with negative expected utility.