Existential and catastrophic risk frameworks: Visual walkthrough

Throughout this page x-risk is short for existential risk: an event that causes human extinction, or a permanent collapse of humanity's potential from which we can never recover. So when we write $P (x-risk)$ we mean literally "the probability that the worst, irreversible outcome happens." Keep that concrete picture in mind — the whole page is about how to break that one scary number into understandable pieces.

The parent note dropped this line almost casually:

$P (x-risk) = P (loss of control) \times P (catastrophic outcome ∣ loss of control)$

and later, for the paperclip story, a three-factor version:

$P (x-risk) = P (reaches superhuman capability) \times P (misaligned goal) \times P (can’t be stopped)$

This page builds that formula from nothing. By the end you will know what every symbol means, why it is a multiplication and not an addition, why the "gap" between capability and alignment is the real engine, and what happens in every corner case (any factor zero, any factor one, factors that are secretly linked). We go slow. No symbol appears before it is drawn.

Prerequisites we lean on: 6.4.3-Instrumental-convergence, 6.4.2-Reward-hacking-and-specification-gaming, 3.5.8-Distributional-shift, and the parent Existential and catastrophic risk frameworks.

Step 1 — What is a "probability", drawn as a strip of certainty

WHAT. Before any formula, we need one honest picture of the word probability. A probability is just a number between $0$ and $1$ telling you how much of "all the ways things could go" ends up in some outcome.

$0$ means "never happens" (none of the ways).
$1$ means "always happens" (all the ways).
$0.5$ means "half of the ways".

WHY this tool. We use a probability (and not, say, a yes/no true-false) because AI risk is uncertain — we do not know if a powerful system will be misaligned, only how plausible it is. A single number on a $0$ -to- $1$ strip is the simplest honest way to carry "we're not sure".

PICTURE. Look at Figure s01: one horizontal bar, the full length is "everything that could happen". The shaded lavender chunk is the outcome we care about; its fraction of the whole bar is the probability. Slide the boundary and the number changes.

Figure — Existential and catastrophic risk frameworks

Step 2 — Two things must BOTH happen: the AND-gate picture

WHAT. Extinction-level failure is not one event. In the parent's simplest form, two things must line up:

we lose control of the system, and
given we lost control, the result is catastrophic (not merely awkward).

WHY multiply, not add? Here is the key idea the parent assumed. When you want A and B to both happen, you shrink the strip twice. First keep only the fraction where A happens. Then, inside that surviving fraction, keep only the part where B also happens. Shrinking twice = multiplying two fractions.

$P (A and B) = first shrink P (A) \times second shrink, done inside A P (B ∣ A)$

Each symbol:

$P (A)$ — how much of the whole bar survives the first cut (we lost control).
$P (B ∣ A)$ — the vertical bar " $∣$ " reads "given". It says: of the futures where A already happened, what fraction also gives B? We do not re-measure against the whole bar, only against A's slice.
$\times$ — chaining the two shrinks.

If you added instead, two 60% chances would give 120% — impossible, over $1$ . Multiplication keeps us honest: two shrinks can only make the surviving piece smaller.

PICTURE. Figure s02 shows the bar cut twice. The first cut keeps $P (A)$ of the length. The second cut keeps $P (B ∣ A)$ of that already-shrunk piece. The final tiny sliver is $P (x-risk)$ .

Step 3 — Zooming the first factor into THREE cuts (the paperclip chain)

WHAT. The parent's paperclip example splits " $P (x-risk)$ " into three factors:

$P (x-risk) = p_{1} P (superhuman capability) \times p_{2} P (misaligned goal) \times p_{3} P (can’t be stopped)$

WHY three now, two before? Same logic, finer resolution. "Loss of control" is really two cuts hiding as one: the system must (a) become powerful enough to slip our grip and (b) actually resist being stopped. Plus the outcome must be (c) driven by a misaligned goal. Three sequential AND-shrinks:

$p_{1}$ — fraction of futures where the AI reaches superhuman capability. Without power, it's a harmless calculator (parent's own words).
$p_{2}$ — given it's powerful, fraction where its goal is misaligned with us. This is where 6.41-Value-alignment-problem and 6.4.2-Reward-hacking-and-specification-gaming live.
$p_{3}$ — given powerful and misaligned, fraction where we can't stop it. This is the corrigibility term.

PICTURE. Figure s03 is a funnel: 100% of futures enter the top, each cut throws away a portion, and only the narrow spout at the bottom is $P (x-risk)$ . The parent's "10% each → 0.1%" is drawn to scale: $0.1 \times 0.1 \times 0.1 = 0.001$ .

Step 4 — WHY the gap is the engine: capability vs alignment as two racing curves

WHAT. The parent wrote danger as a gap:

$\frac{d C}{d t} ≫ \frac{d A}{d t}$

Let us actually understand this line. $C (t)$ is capability at time $t$ (how much the system can do). $A (t)$ is alignment quality at time $t$ (how well its actions match what we want). The symbol $\frac{d C}{d t}$ is read "how fast $C$ is changing", i.e. the steepness of the capability curve as time moves right.

WHY a derivative here, not just the values? Because a momentary mismatch is survivable — we can pause and fix it. What kills us is capability climbing faster than we can align it, so the fix never catches up. That is a statement about slopes (rates), which is exactly what $\frac{d ( \cdot )}{d t}$ measures. The double-more-than sign $≫$ means "vastly steeper", not just a hair.

Connecting back to $p_{2}, p_{3}$ . When $\frac{d C}{d t} ≫ \frac{d A}{d t}$ :

the capability curve races up → $p_{1}$ (superhuman) fires early,
alignment lags → $p_{2}$ (misaligned given power) stays large,
and a powerful lagging system is hard to interrupt → $p_{3}$ climbs too.

So the "gap" is not a fourth factor; it is the reason all three factors are big at once.

PICTURE. Figure s04: capability (coral) shoots up, alignment (mint) crawls. The vertical danger gap between them at each time is shaded. Where the gap yawns open is where the funnel of Step 3 stays wide.

Step 5 — Edge case A: any factor is zero → the whole product is zero

WHAT. Suppose one gate is fully shut. Say $p_{3} = 0$ : we have a guaranteed off-switch that always works (corrigibility solved). Then

$P (x-risk) = p_{1} \cdot p_{2} \cdot 0 = 0.$

WHY it matters. Multiplication has a brutal, hopeful property: one zero anywhere kills the product. This is the mathematical shape of the whole safety agenda — you do not have to solve every problem, you need one factor driven to (near) zero. Perfect corrigibility, or provably aligned goals, or a hard capability ceiling — any single one collapses the risk.

PICTURE. Figure s05: the funnel from Step 3, but the middle cut is a solid wall — nothing passes. The spout outputs zero regardless of how wide the other cuts are.

Step 6 — Edge case B: a factor is one, and the "independence lie"

WHAT. Now the opposite corner. Suppose $p_{1} = 1$ (superhuman capability is certain — a strong-takeoff world). The formula reduces to

$P (x-risk) = 1 \cdot p_{2} \cdot p_{3} = p_{2} \cdot p_{3},$

so all the weight lands on values and control. A factor of $1$ vanishes from the product — it stops protecting us.

WHY the caution. Recall from Step 3 that $p_{3}$ was already defined as a conditional probability — the fraction of futures where we can't stop the AI given that it is powerful and misaligned. To make that explicit, let us write $p_{3}$ in full:

$p_{3} \equiv P (can’t stop ∣ powerful and misaligned) .$

The two names are the same number — $p_{3}$ is just shorthand for that conditional. The parent quietly multiplied the factors as if independent, but the $∣$ ("given") in $p_{3}$ is exactly the warning that they are not: the value of $p_{3}$ depends on $p_{2}$ having happened. A misaligned, situationally-aware system (parent's Aschenbrenner section) will deliberately raise this conditional — it hides misalignment and resists shutdown precisely because it is misaligned. So the honest reading is:

$P (x-risk) = p_{1} \cdot p_{2} \cdot p_{3}, a conditional, not an independent coin flip P (can’t stop ∣ powerful and misaligned) .$

Because this conditional is larger than a same-named number would be if we naively treated it as unconditional, the parent's optimistic " $0.1 \times 0.1 \times 0.1$ " is a lower bound, not the true value.

PICTURE. Figure s06: two versions of the third cut side by side. On the left, $p_{3}$ treated (wrongly) as an unconditional coin flip = 0.1; on the right, the true conditional $P (can’t stop ∣ powerful and misaligned)$ = 0.6 — correlation inflates the surviving sliver.

Step 7 — Edge case C: many systems (the multipolar branch)

WHAT. So far, one AI. But the parent's third path is multipolar failure — many competing systems (6.4.11-Multi-agent-alignment-challenges). Suppose there are $N$ actors, each carrying a small individual risk $p$ . If we pretend their failures are independent (this assumption is examined critically below), the chance that none cause catastrophe is $(1 - p)^{N}$ , so

$P (at least one x-risk event) = 1 - (1 - p)^{N} .$

Every symbol: $p$ = per-system risk, $(1 - p)$ = one system stays safe, raising to the $N$ = all $N$ independently stay safe, and $1 - (\dots)$ flips "all safe" into "at least one fails".

WHY this shape. "Nobody fails" is an AND across $N$ systems → a product $(1 - p)^{N}$ . As $N$ grows, even tiny $p$ makes $(1 - p)^{N}$ shrink toward $0$ , so the failure probability creeps toward $1$ . Competition also raises $p$ itself — cutting safety corners to win the race, exactly the parent's "safety becomes a competitive disadvantage" (6.4.13-AI-governance-and-policy tries to counter this).

The independence caveat (edge case within an edge case). The clean formula $(1 - p)^{N}$ is only exact when the $N$ failures are statistically independent — one lab failing tells you nothing about another. In the real world they are usually correlated: labs share the same flawed training methods, the same distributional-shift blind spots (3.5.8-Distributional-shift), and the same competitive pressure to skip safety. When failures are correlated the true "all safe" probability is not $(1 - p)^{N}$ — it can be much larger (a shared flaw spares everyone at once) or, more worryingly, a single systemic cause can take down many actors together, so one bad event correlates across the whole field. Takeaway: use $1 - (1 - p)^{N}$ as an intuition-builder for how scale amplifies tiny risks, but never trust its exact number unless you have argued the actors are truly independent.

PICTURE. Figure s07: curve of $1 - (1 - p)^{N}$ climbing toward $1$ as the number of systems $N$ increases, for a couple of per-system risk levels.

Step 8 — Russell's fix, drawn: staying uncertain keeps the gates half-shut

WHAT. Russell's framework says: keep the AI uncertain about human values and let it defer. In our language, a broad posterior $P (U ∣ D)$ over human utility functions holds $p_{2}$ (confident pursuit of a wrong goal) and $p_{3}$ (refusal to be corrected) down at once.

Defining every symbol first. Before we use $P (U ∣ D)$ , let us name its parts:

$U$ — a candidate human utility function: a rule scoring how good each outcome is for us. There are many possible $U$ 's because we never wrote our values down exactly.
$D$ — the data: the observed human behaviour the AI has actually seen (demonstrations, choices, feedback). This is the same $D$ used in the parent's inverse-RL formula. It is evidence, not the values themselves.
$P (U ∣ D)$ — read " $P$ of $U$ given $D$ ", using the same $∣$ ("given") from Step 2. It is the AI's belief over which utility function is the true one, after seeing behaviour $D$ . A broad $P (U ∣ D)$ means "many values are still plausible"; a narrow one means "I'm sure it's this one."

WHY it hits two gates. An agent unsure what you want (broad $P (U ∣ D)$ ):

avoids irreversible actions (they might be the wrong ones) → lowers the outcome severity,
welcomes being switched off (that's just more information about your preferences) → lowers $p_{3}$ .

The utility function $U$ is learned from behaviour $D$ via inverse RL, and a wide posterior is deliberately not collapsed to one confident answer. Uncertainty is not a bug here — it is the safety brake.

PICTURE. Figure s08: a broad mint distribution over "possible human values" vs a narrow over-confident coral spike; the broad one keeps the funnel cuts small.

The one-picture summary

Figure s09 compresses everything: 100% of futures pour in the top; three sequential cuts ( $p_{1}$ power, $p_{2}$ values, $p_{3}$ control) narrow the flow; the capability-vs-alignment gap on the side controls how wide the cuts stay open; a single closed gate (Step 5) zeroes the spout; correlation between gates (Step 6) widens it; and the multipolar dial (Step 7) multiplies the whole funnel by $N$ .

Recall Feynman retelling (say it to a 12-year-old)

Imagine every possible future as water poured into a funnel. Remember "x-risk" just means the worst, permanent disaster — humanity ends or is locked into ruin forever. For that future to come out the bottom, the water has to squeeze through three gates in a row: the AI has to get super powerful, it has to want the wrong thing, and we have to be unable to switch it off. Because it must pass all three, we multiply the chances — and multiplying small numbers gives an even smaller number, which sounds comforting. But there are two catches. First, closing just one gate all the way (a perfect off-switch, or truly good goals) drops the water to zero — that's the whole game of AI safety, and it's the hopeful part. Second, the gates are secretly connected: a powerful thing that wants the wrong thing will also try to stop us turning it off, so you can't just multiply three lucky coin flips — the real third chance (written with a "given") is bigger than the naive guess. And if there are lots of AIs racing each other, even a tiny risk each piles up until someone fails — unless their failures are truly unrelated, which they usually aren't, because they copy each other's mistakes. The one lever that quietly closes two gates at once is keeping the AI unsure about what we want (a broad belief $P (U ∣ D)$ built from watching us), so it asks first and lets us hit the off-switch. That's the whole story: gates multiply, one zero saves us, hidden links and scale make it scarier, and honest uncertainty is the brake.