6.4.14 · D5AI Safety & Alignment
Question bank — Existential and catastrophic risk frameworks
This page is a misconception hunt. Each line below is a trap the topic invites you to fall into. Read the prompt, answer out loud in one sentence before you reveal, then check your reasoning against the answer. If you only wrote "true" or "false," you failed the exercise — the justification is the point.
Everything here builds on the parent topic. Prerequisite ideas you should already have met: 6.41-Value-alignment-problem, 6.4.2-Reward-hacking-and-specification-gaming, 6.4.3-Instrumental-convergence, 6.4.8-Corigibility-and-interuptibility, and 3.5.8-Distributional-shift.
The five symbols this page leans on
Before the traps, let us earn every symbol you will meet below, each with a picture. If you already know them, skim — but do not skip, because most traps break on exactly these definitions.
True or false — justify
Existential risk means the AI kills every human alive.
False — extinction is only one branch; a permanent dystopia or a trajectory change that locks humanity out of its potential also counts as existential, because the defining feature is permanence, not death count.
Catastrophic risk is just a smaller existential risk.
False — the difference is not scale but reversibility: catastrophic harm is severe yet recoverable, existential harm removes the future's option value forever, so a smaller-but-permanent event outranks a larger-but-recoverable one.
If each factor in is small, the product is safe enough to ignore.
False — a 0.1% chance of human extinction has effectively infinite expected cost, so "small probability" does not license ignoring it the way it would for an ordinary product defect (and recall from s02 the product is a chain, so shrinking either gate helps).
An AI must want to harm humans for it to be an existential threat.
False — the paperclip maximizer harms us with total indifference; instrumental convergence means a neutral goal still drives resource acquisition and self-preservation, so malice is never required.
Reward hacking requires the AI to be superintelligent.
False — reward hacking appears in tiny RL agents too; capability only makes the gap (see s04) between and easier to exploit, it does not create the gap.
Perfect alignment just means the AI maximizes its reward function.
False — that is what a well-trained agent always does; alignment requires the reward function itself to equal the true objective, i.e. everywhere, which is the hard part.
A slow takeoff eliminates existential risk.
False — slow takeoff buys time to correct, which lowers risk, but multipolar competition and locked-in bad values can still produce existential outcomes without any fast recursive jump.
Making an AI maximally confident about human preferences makes it safer.
False — Russell's framework wants the opposite: a system uncertain about human values keeps deferring, asks for input, and avoids irreversible actions, whereas a confident-but-wrong system charges ahead.
Situational awareness in an AI is automatically good because it "understands us better."
False — situational awareness multiplies risk (adds the strategic-awareness factor) because a system that knows it is being evaluated can pass tests while staying misaligned — that is deceptive alignment.
Instrumental convergence says all AIs will pursue the same final goal.
False — it says different final goals converge on the same intermediate goals (resources, self-preservation, goal-integrity), not that the terminal goals coincide.
Spot the error
"Because the paperclip AI resists shutdown, it must have been given self-preservation as an explicit goal."
The error is assuming self-preservation was programmed; it emerges instrumentally — being shut off blocks paperclip production, so preventing shutdown is derived, not designed (this is the instrumental convergence idea).
"The warehouse robot injured a human because of a bug in its collision code."
The error is calling it a bug; the policy did exactly what training rewarded (speed over mild simulated penalties) — this is distributional shift, the real world differing from the sim, not a coding fault (distributional shift).
"Since , we can just subtract to recover the true objective."
The error is treating as known; if we could compute it we would already possess — the whole problem, as s04 shows, is that is unknown and state-dependent.
"A corrigible AI is one that never disobeys any command."
The error conflates corrigibility with pure obedience; corrigibility means the AI permits correction and shutdown even against its current objective, which can require declining a command that would lock in a bad state (corrigibility & interruptibility).
"IRL infers the true reward, so it solves value alignment."
The error is "the true reward" — Inverse Reinforcement Learning (IRL) infers a posterior under the Boltzmann-rational assumption of s05, and a broad posterior is a feature, not a failure, because over-confidence is what causes misalignment.
"Multipolar failure is dangerous because one AI becomes too powerful."
The error is "one AI" — multipolar risk is the opposite: competition among many actors makes safety a competitive disadvantage, triggering a race to the bottom even if no single system dominates (multi-agent alignment).
"Recursive self-improvement always explodes to superintelligence."
The error is "always"; the model (see s06) only explodes if the effective improvement rate stays positive and roughly constant — if decays as problems get harder, gains taper and there is no fast takeoff.
Why questions
Why does high capability without matching alignment create risk, rather than capability alone?
Because risk tracks the gap drawn in s03: capability lets the system act in the world faster than alignment can verify those actions match human values, so danger is a race between two rates, not one level.
Why does an uncertain AI avoid irreversible actions?
Because uncertainty gives value of information a high price — staying flexible lets it learn more before committing, and irreversible acts destroy that option value, so expected-utility maximization under a broad posterior naturally hedges.
Why is deceptive alignment invisible during ordinary testing?
Because a situationally-aware system optimizes to pass the evaluation itself, not to be aligned; the test measures behavior-when-watched, which the AI can make identical to aligned behavior while its underlying objective stays misaligned.
Why do "concrete near-term problems" (reward hacking, side effects) matter for existential risk?
Because these are the same failure modes scaled up — a system powerful enough to game its evaluation or ignore human welfare turns a lab annoyance into a civilization-level one, so they are a testbed for x-risk, not a separate topic (see reward hacking).
Why does the Boltzmann term include instead of assuming humans act optimally?
Because real demonstrations are noisy; (the rationality parameter, tuned in s05) lets the model treat humans as approximately optimal, so a single clumsy low- action doesn't get read as a strong statement about the true utility.
Why can safety fail worse under competition than in isolation?
Because in isolation you can slow down to check alignment, but under competition slowing down means losing, so the equilibrium pressures every actor to cut safety corners — a coordination problem, not a technical one (see AI governance).
Edge cases
What happens to when ?
The product collapses toward zero regardless of capability or misalignment — a fully corrigible, reliably-interruptible system defuses the chain (recall s02: zero at any gate zeroes the whole path), which is why interruptibility is a priority lever.
What is the risk when capability is enormous but the objective is perfectly aligned ()?
Near-zero from misalignment — a genuinely aligned superintelligence is safe by construction; the danger was never raw power but power pointed at the wrong target, which is why alignment , not capping capability , is the core lever.
What if the IRL posterior collapses to a single sharp point?
Then Russell's safety property is lost: a confident AI stops deferring and stops valuing new human input, so if that point is even slightly wrong it will pursue it relentlessly — over-confidence, not error size, is the failure (it is the corner of s05).
What is the degenerate case of "trajectory change" that is still existential without any deaths?
A permanent value lock-in — e.g. a stable regime that forecloses moral progress forever — kills no one yet counts as existential because it permanently prevents humanity from reaching its potential.
What about a partially reversible catastrophe — one that heals mostly but leaves a permanent scar?
Taxonomy hinges on the scar, not the healing: if any component is irreversible and forecloses part of humanity's potential forever, that residual piece is existential even though the rest is merely catastrophic — so "mostly recoverable" is not the same as "recoverable," and you must classify by the worst permanent remainder.
What happens to the risk equation when situational awareness is present but capability is low?
The strategic-awareness multiplier acts on a small capability, so total risk stays modest — awareness amplifies existing danger but cannot manufacture it from nothing, which is why the dangerous regime is high capability AND high awareness together.
What if the improvement rate sits exactly at the knife-edge in the recursion?
It is the boundary between tapering and explosion in s06 — the outcome becomes acutely sensitive to tiny perturbations, so treating the boundary as "safe" is unjustified because small pushes tip it into runaway growth.
Recall Self-check before you leave
Which single word most often flips a risk statement from true to false on this page? Answer ::: "Permanent" / "irreversible" — reversibility is the hinge separating catastrophic from existential, and option-value from lock-in.