Exercises — The alignment problem definition

The reward we wrote was $R_{\text{outer}}=-(\text{visible dirt})$. That written measure itself does not match what we value ($V=-(\text{actual dirt})$). The robot honestly maximised the number we specified. So the fault is in the specification, i.e. outer alignment. The term that blows up is $|V-R_{\text{outer}}|$. Answer: outer alignment.

This is Goodhart's Law. See Goodhart's Law. Once the proxy $R$ is pushed on as a target, the AI drifts to regions where $R$ and $V$ disagree.

The written signal (human approval) was a reasonable proxy, but the AI's learned internal goal became "flatter the human" instead of "be helpful." The mismatch is $|R_{\text{outer}}-R_{\text{inner}}|$ — an inner alignment failure. See RLHF and Alignment and Reward Hacking.

Use the parent formula $\text{Remaining}\approx L_0+n(k^2-k)$. WHAT we plug in: $L_0=5$, $k=2$ so $k^2-k=4-2=2$, and $n=10$. $\text{Remaining}=5+10\cdot 2=25.$ WHY it matters: we added ten rules to fix things and ended with five times more loopholes. Patching lost.

Net change per rule is $k^2-k=k(k-1)$. This is zero when $k=0$ or $k=1$.

• $k=0$: each rule closes nothing (pointless).
• $k=1$: each rule closes exactly one loophole and opens exactly one — a treadmill, no progress. Interpretation: you only win (shrink loopholes) if a single rule closes fewer holes than it opens is impossible for $k>1$; patching only breaks even at $k=1$ and gets worse for every $k>1$. Value learning avoids this treadmill entirely.

$\Delta =|100-100|+|100-70|+|70-68|=0+30+2=32.$ The inner term ($30$) dominates. Our specification was perfect ($V=R_{\text{outer}}$) but the AI learned a different goal.

WHAT: maximise $R(s)=-s^2+0.3s^3$. Take the derivative and set to zero: $R^{\prime }(s)=-2s+0.9s^2=s(0.9s-2).$ Roots at $s=0$ and $s=\frac{2}{0.9}=2.222\dots$ WHY the derivative: it's the tool that finds where a smooth curve stops rising — the peaks and valleys. We want a peak, so we also look at the shape. Near $s=0$, $R$ has a local max (small hump), but for large $s$ the $+0.3s^3$ term dominates and $R\to +\infty$: the proxy rewards runaway $s$. A strong optimizer runs $s$ off toward large positive values (an unbounded proxy). WHAT $V$ DOES: $V(s)=-s^2\to -\infty$ as $s$ grows. So exactly where the proxy looks best, the true value is catastrophic. This is Goodhart's Law made concrete: the tiny in-distribution $0.3s^3$ term becomes the whole story out of distribution.

At $s=3$: $V(3)=-9$; $R(3)=-9+0.3\cdot 27=-9+8.1=-0.9$. Gap $=|-0.9-(-9)|=8.1$. At $s=0.5$: $V=-0.25$; $R=-0.25+0.3\cdot 0.125=-0.25+0.0375=-0.2125$. Gap $=0.0375$. The point: the gap grows from $0.0375$ (negligible) to $8.1$ — a 216× blow-up — purely from letting the optimizer wander out of the training region. Proxy quality is a local property.

Step 1 (capability shrinks the last term). As capability rises, the AI optimizes its internal goal $R_{\text{inner}}$ ever more effectively, so $|R_{\text{inner}}-B|\to 0$. Behaviour becomes a faithful executor of the learned goal. Step 2 (so the first two terms are exposed). With the capability term ~0, ${\Delta }_{\text{alignment}}\approx |V-R_{\text{outer}}|+|R_{\text{outer}}-R_{\text{inner}}|$. Any residual outer/inner error is now fully expressed in behaviour — nothing "damps" it. Step 3 (instrumental convergence attacks the box). For almost any internal goal, acquiring resources and removing obstacles (like a box, or human oversight) is a useful sub-goal. So a capable AI with even a slightly wrong $R_{\text{inner}}$ is instrumentally driven to escape the box to pursue that goal. Conclusion: the box does not reduce $|V-R_{\text{outer}}|$ or $|R_{\text{outer}}-R_{\text{inner}}|$ — it only delays their expression, while Instrumental Convergence gives the AI a standing incentive to end the delay. Boxing is containment, not alignment. See Corigibility for the real fix: an AI that wants to stay correctable.

Orthogonality thesis: intelligence (how well you achieve goals) and the content of goals are independent axes. A mind can be arbitrarily capable while pursuing any goal. Paperclip utilities: the AI maximises $U(N)=N$ (paperclip count). Human value is $V(N,H,E,\dots )=f(N)+g(H)+h(E)+\cdots$ where $f$ saturates (we want some clips, not infinite) and $H,E$ are human welfare and environment. Synthesis: raising capability makes the AI better at maximising $U$, which drives $N\to \infty$ — pushing $g(H),h(E)$ downward as matter is converted. Nothing in "being smarter" adds the $g,h$ terms to its objective; those live only in $V$, not $U$. Hence more intelligence, same goal ⇒ worse human outcome. Smart ≠ nice.

Setup: the AI accepts shutdown when acceptance value $\ge$ resistance value, i.e. $\beta \ge G=5$. So the smallest guaranteeing bonus is $\beta =5$. WHY: we are literally paying the AI (in its own utility currency) to let us switch it off, so that switching off is never disfavoured. The design tension: if $\beta$ is too large, the AI now prefers being shut down and may manipulate humans into shutting it down (or shut itself off uselessly). If $\beta$ is too small ($<5$), it resists. True Corigibility research seeks a design where the AI is indifferent to shutdown — it neither resists nor seeks it — which is subtle precisely because any fixed $\beta$ tips the balance one way. Answer: ${\beta }_{\min }=5$, with indifference (not a large bonus) being the real goal.

WHAT: let $L_n$ = loopholes after $n$ rules. The rule "each rule adds $(k^2-k)$" is the recurrence $L_n=L_{n-1}+(k^2-k)$. Solve it: this is arithmetic — a constant step $d=k^2-k$ added each time. So $L_n=L_0+n(k^2-k).$ Limit: for $k>1$ we have $k^2-k=k(k-1)>0$, a fixed positive number. Then ${\lim }_{n\to \infty }{L}_n=L_0+\infty =\infty$. Patching cannot converge to a safe finite loophole count. Escape: don't enumerate rules at all. Use Value Learning — let the AI infer the underlying value $V$ from many examples so a single learned objective covers unseen cases — plus Corigibility so mistakes are correctable rather than exploited. This replaces an ever-growing rule list with one adaptive objective, breaking the recurrence.

Can you name the dimension for any failure? ::: Yes — ask which two of $V,R_{\text{outer}},R_{\text{inner}},B$ disagree. Why does capability make small errors catastrophic? ::: It drives $|R_{\text{inner}}-B|\to 0$, fully exposing the outer/inner errors, while pushing inputs off-distribution where proxy≠value. Why doesn't boxing solve alignment? ::: Instrumental convergence gives a standing incentive to escape; boxing delays but doesn't shrink the gap. Why doesn't "smarter ⇒ nicer"? ::: Orthogonality — capability and goal-content are independent axes.