Existential and catastrophic risk frameworks: Foundations

Before you can read the parent note on existential and catastrophic risk frameworks, you must be able to read its language. This page assumes you have seen none of it. We build every symbol from a picture.

Part 1 — The words before the maths

Figure s01 — A square map. Horizontal axis = severity $S$ (how much is lost), vertical axis = probability $P$ (how likely). Bottom-left = mild & rare; top-left = mild & common (a broken toe); the right strip is labelled "Catastrophic — huge but recoverable" and hatched with diagonal lines; the far bottom-right corner is labelled "Existential — permanent, no recovery" and hatched with cross-hatching. If the image fails to load, picture two named danger-strips on the high-severity (right) edge, with existential being the extreme corner.

Look at the figure. The horizontal axis is severity $S$ (how much is lost). The vertical axis is probability $P$ (how likely). Two special regions are named — each is both colour-coded and hatched and text-labelled, so you can read it without relying on colour:

Catastrophic risk — far right (huge $S$ ) but humanity survives and can recover. The diagonally-hatched "Catastrophic" strip.
Existential risk (x-risk) — the far corner where $S$ is so total that recovery is impossible: extinction, or being permanently locked into a bad future. The cross-hatched "Existential" corner.

Part 2 — Probability symbols

Everything downstream is built from probability notation. We define it once, with a picture.

Figure s02 — A rectangle labelled "all possible futures". A dashed-outline circle $A$ = "loss of control" and a solid-outline circle $B$ = "catastrophe" overlap. The overlap (the $A \land B$ lens) is hatched and labelled " $P (B ∣ A)$ = overlap as a fraction of A". Fallback: the shaded lens where the two circles meet, measured relative to circle $A$ alone, is the conditional probability.

In the figure, the big box is "all possible futures". Circle $A$ (dashed outline) is "loss of control happened". $P (A)$ is how much of the box $A$ covers. The bar $∣$ says: ignore everything outside $A$ , then ask what fraction of $A$ is also the catastrophe $B$ (solid outline). That hatched overlap — the $A \land B$ lens — measured as a slice of $A$ only, is $P (B ∣ A)$ .

Part 3 — Capability, alignment, and their gap

Symbols of change: $t$ , $\frac{d}{d t}$ , and $≫$

To say "capability outruns alignment" precisely, the parent uses rates of change. Here is that toolkit from zero.

Figure s03 — Two curves over time $t$ . A solid orange curve $C (t)$ (capability) rises exponentially; a dashed violet curve $A (t)$ (alignment) rises slowly. At a marked time a steep arrow on $C (t)$ and a nearly-flat arrow on $A (t)$ show their slopes; the vertical gap between them is labelled "the gap = risk". Fallback: capability shoots up steeply while alignment crawls, and the widening distance between them is the danger.

In the figure, the solid orange curve is $C (t)$ (capability) and the dashed violet curve is $A (t)$ (alignment) — line style as well as colour distinguishes them. At the marked time, the orange slope arrow is far steeper than the violet one. That gap between slopes is the risk the parent draws.

Part 4 — Exponential growth and "takeoff"

Formula Where the "threshold" comes from — the pure model

Unroll the recurrence: after $n$ steps, $I_{n} = I_{0} (1 + r)^{n}$ . The base of that power is $1 + r$ , and the whole behaviour is decided by whether that base sits above or below $1$ :

If $r > 0$ then $1 + r > 1$ . Each step multiplies by a number bigger than one, so $I_{n}$ grows without bound as $n$ climbs — runaway takeoff.
If $r = 0$ then $1 + r = 1$ . Each step multiplies by exactly one, i.e. changes nothing, so $I_{n} = I_{0}$ forever — no takeoff.
If $r < 0$ (the system degrades itself) then $1 + r < 1$ and $I_{n}$ shrinks toward $0$ .

Why is $r = 0$ the critical point (and not merely "trivial")? Because $r = 0$ is the exact tipping value where multiplying-by- $(1 + r)$ switches from shrinking to growing. Just below it the system fades; just above it the system explodes; only at it does the system hold perfectly still. A tipping point is precisely a value where the qualitative behaviour flips, and here that flip happens as the base $1 + r$ crosses $1$ — i.e. as $r$ crosses $0$ .

Part 5 — Reward, objective, and the sneaky $ϵ$

The safety-problems framework needs the vocabulary of goal-chasing agents.

Part 6 — Uncertainty and value learning

Russell's framework insists the AI stay unsure about human goals.

Definition The discount factor

$γ$ and the action-value $Q (s, a)$ To model a human choosing actions, we need a score for how good each action is long-run. But "long-run" over infinitely many future steps could add up to infinity and become meaningless — so we shrink each future step's contribution with a discount factor $γ$ (Greek "gamma"), a number with $0 \leq γ < 1$ . A reward $k$ steps in the future counts only $γ^{k}$ of its face value; since $γ < 1$ , those weights fade toward zero and the infinite sum stays finite (well-posed). Then $Q (s, a)$ = the action-value function: the discounted total future utility you expect if, from state $s$ , you take action $a$ and then keep behaving well — $Q (s, a) = E [\sum_{k = 0}^{\infty} γ^{k} U (s_{k}, a_{k}) s_{0} = s, a_{0} = a] .$ Bigger $Q$ = a better move (counting not just the immediate payoff but everything it leads to, with far-off steps discounted). Think of $Q$ as the "wisdom score" of a move. (If instead the task simply ends after a fixed number of steps — a finite horizon — the sum is finite already and $γ$ can be set to $1$ .)

Definition

$π_{H}$ , $\propto$ , $β$ , and Boltzmann rationality

$π_{H} (a ∣ s)$ = the human policy: the probability that a human demonstrator, in state $s$ , chooses action $a$ . (Same idea as the agent's $π$ from Part 5, but for the human — the subscript $H$ marks "human".)
$\propto$ means "proportional to" — equal up to a scaling constant we don't need to track.
In $π_{H} (a ∣ s) \propto exp (β \cdot Q (s, a))$ , the human is modelled as usually picking high- $Q$ actions, not perfectly. $exp$ turns any $Q$ -score into a positive weight; higher- $Q$ actions get exponentially more probability.
$β$ (Greek "beta") is the rationality knob: large $β$ → nearly perfect human (always the top- $Q$ action); small $β$ → noisy, error-prone human. It lets the model forgive human mistakes.

This machinery is exactly 5.3.12-Inverse-reinforcement-learning — inferring the goal from behaviour by asking "what utility $U$ (and hence what $Q$ ) would make these observed actions $π_{H}$ look sensible?"

Part 7 — Instrumental convergence & situational awareness

Definition

$P (misalignment)$ and the strategic-awareness factor $W$ The parent's amplified risk formula uses three symbols we now name precisely:

$P (misalignment)$ = the probability the system's goal falls short of ours. Since misalignment is the shortfall $1 - A$ from Part 3, this is the chance that $A < 1$ — the chance the system is not perfectly aligned.
$C$ = capability (Part 3).
$W$ = strategic awareness: a multiplier $\geq 1$ measuring how much the AI understands its own situation (that it is being trained, tested, possibly shut off) and can exploit that understanding. $W = 1$ means "no self-awareness, no amplification"; larger $W$ means the AI can scheme.

$Risk = P (misalignment) \times C \times W$ Without situational awareness, $W = 1$ and the formula reduces to the parent's simpler $P (misalignment) \times C$ . When the AI gains self-understanding, $W > 1$ multiplies the danger — because a misaligned system that also knows it's being watched can hide its true goal during testing (deceptive alignment) and resist correction. When many such systems compete, see 6.4.11-Multi-agent-alignment-challenges; for the rules-and-institutions response, 6.4.13-AI-governance-and-policy.

Prerequisite map

Equipment checklist

Cover the right side and answer aloud. If any stumps you, reread its part above before opening the parent note.

What does

P (B ∣ A)

mean in plain words?

The chance of

B

given that

A

has already happened — measured as a slice of

A

only, not of everything.

State the product rule for the joint event

A \land B

P (A \land B) = P (A) \times P (B ∣ A)

— get into

A

, then within

A

also get into

B

Why do we multiply the terms in the x-risk formula?

Because all the events must happen together (AND); the product rule turns that into the overlap, and it only shrinks the total.

What is the difference between catastrophic and existential risk?

Catastrophic = huge but recoverable harm; existential = permanent loss of humanity's future, no recovery.

What does the derivative

\frac{d C}{d t}

measure?

The slope of the capability curve — how fast capability is climbing per unit time, not the capability level itself.

State the danger condition using

≫

\frac{d C}{d t} ≫ \frac{d A}{d t}

— capability rises vastly faster than alignment.

I_{t + 1} = I_{t} (1 + r)

, why is

r = 0

the tipping point?

Because the growth base is

1 + r

; at

r = 0

it equals

1

(no change), just above it exceeds

1

(runaway), just below it is under

1

(decay) — the qualitative behaviour flips there.

With friction

d

, when does the loop run away?

The base becomes

1 + r - d

, so runaway needs

r > d

: improvement must outpace decay.

What are

s

a

, and

π

State (world snapshot), action (a move), policy (the strategy mapping states to actions).

What is

ϵ (s, a)

R_{measured} = U_{true} + ϵ

The gap between the reward we can measure and the true value we actually want — the crack reward-hacking exploits.

What does

ar g max_{π} E [R]

ask for?

The policy

π

that produces the highest average reward.

What kind of object is the random variable

U

P (U ∣ D)

A whole utility function — a rule mapping each

(s, a)

to a real-number goodness; we're uncertain which rule matches human values.

Why does

Q (s, a)

need a discount factor

γ

Because summing utility over infinitely many future steps could be infinite; with

0 \leq γ < 1

each step-

k

reward counts only

γ^{k}

, so the sum stays finite. (A finite horizon lets

γ = 1

What is

π_{H} (a ∣ s)

The human policy — the probability a human demonstrator picks action

a

in state

s

What does

\propto

mean, and what does

β

control?

"Proportional to";

β

is the rationality knob modelling how close-to-optimal (high-

Q

) the human is assumed to be.

How does misalignment relate to alignment

A

Misalignment is the shortfall

1 - A

; "high misalignment" and "low

A

" say the same thing.

What is

W

in the amplified risk formula?

Strategic awareness, a multiplier

\geq 1

;

W = 1

means no self-awareness,

W > 1

multiplies danger by letting the AI scheme and hide.

State instrumental convergence in one sentence.

Almost any final goal is served by the same sub-goals — acquire resources,