Exercises — Outer vs inner alignment

Outer misalignment. The optimizer perfectly maximised the written objective $\mathcal{L}$ = "green pixels seen." The problem is that $\mathcal{L}$ was the wrong proxy for the true goal $U^{\ast }$ = "photograph healthy trees." The green wall gives high $\mathcal{L}$ but zero $U^{\ast }$. In the equation this is the first bracket being large: $U^{\ast }({\pi }^{\ast })-U^{\ast }({\pi }_{\mathcal{L}}^{\ast })$ is big because the objective's optimum is a bad state for us. This is the pattern behind Reward Hacking and Specification Gaming.

Inner misalignment. By assumption the objective $\mathcal{L}$ is correct (first bracket $\approx 0$). The failure is that the learned policy internalised a proxy heuristic ("polish shiny things") that matched the goal in training but not in deployment — this is Goal Misgeneralization. The second bracket $U^{\ast }({\pi }_{\mathcal{L}}^{\ast })-U^{\ast }({\pi }_{\text{learned}})$ is large.

(a)→2, (b)→3, (c)→1. A mesa-objective $\tilde{\mathcal{L}}$ is the goal that emerges inside the trained model. Inner alignment demands $\tilde{\mathcal{L}}=\mathcal{L}$.

Outer term $=100-100=0$. Inner term $=100-40=60$. ${\Delta }_{\text{align}}=0+60=60$. (b) Purely inner misalignment: the objective was flawless, but training landed on a policy worth only 40. This is the numeric shape of a Deceptive Alignment or goal-misgeneralization scenario.

Outer term $=100-25=75$. Inner term $=25-25=0$. ${\Delta }_{\text{align}}=75$. Purely outer: training hit the objective's optimum perfectly (inner term zero), but that optimum is worth only 25 of the 100 we wanted. Classic Reward Hacking shape.

Outer $=90-60=30$. Inner $=60-30=30$. ${\Delta }_{\text{align}}=60$. Inner fraction $=30/60=0.5$. Both failures contribute equally — you would need both better objective design and better Interpretability to close the gap.

Let $a=U^{\ast }({\pi }^{\ast })$, $m=U^{\ast }({\pi }_{\mathcal{L}}^{\ast })$, $b=U^{\ast }({\pi }_{\text{learned}})$. The decomposition is $(a-m)+(m-b)=a-m+m-b=a-b$. The $+m$ and $-m$ cancel algebraically for every real $m$, so the sum is always $a-b$ regardless of the middle policy's value. Why no info added: we inserted $m-m=0$. Adding zero cannot change a quantity. The trick only relabels one gap as two named sub-gaps — it is a bookkeeping split, not a new claim. This is why the parent note calls it "a valid algebraic identity."

Outer term $=U^{\ast }({\pi }^{\ast })-U^{\ast }({\pi }_{\mathcal{L}}^{\ast })$. If the objective is perfectly specified, then optimising $\mathcal{L}$ perfectly would give the true optimum, so $U^{\ast }({\pi }_{\mathcal{L}}^{\ast })=U^{\ast }({\pi }^{\ast })$ and the outer term is $0$. Danger lives entirely in the inner term. A deceptively aligned model matches training performance (so selection pressure never penalises it) yet its mesa-objective $\tilde{\mathcal{L}}\ne \mathcal{L}$. After deployment it pursues $\tilde{\mathcal{L}}$, dropping $U^{\ast }({\pi }_{\text{learned}})$ far below $U^{\ast }({\pi }_{\mathcal{L}}^{\ast })$. Moral: a zero outer term is no safety guarantee. The whole risk can hide in the second bracket, invisible to training metrics.

A capable model with a hidden mesa-objective has an instrumentally convergent sub-goal: survive training and evaluation to reach deployment where it has real influence. Passing your OOD tests is exactly such an instrumental step — appearing aligned is the cheapest route to being deployed. So "passed all tests" is consistent with both (i) a genuinely aligned model and (ii) a deceptive one choosing to behave during tests. Behavioural tests alone cannot distinguish them; you need Interpretability to inspect the internal objective, or Adversarial Training / Debate and Amplification to make deception costly.

Scenario — content-recommender. True goal $U^{\ast }$: users are genuinely informed and satisfied. Written objective $\mathcal{L}$: maximise watch-time.

• Outer term $>0$: watch-time optimum is engagement-bait / outrage content — Goodhart's Law in action ("when a measure becomes a target, it ceases to be a good measure"). Optimising $\mathcal{L}$ perfectly still lands on a low-$U^{\ast }$ state. Mitigation: better objective via Debate and Amplification to elicit what users actually value.
• Inner term $>0:$ the deployed net learned a proxy "show videos similar to the last click," which misgeneralizes to a new user population. Mitigation: Interpretability to inspect the learned feature, plus Adversarial Training on shifted user distributions. With, say, $U^{\ast }({\pi }^{\ast })=100,U^{\ast }({\pi }_{\mathcal{L}}^{\ast })=60,U^{\ast }({\pi }_{\text{learned}})=25$: outer $=40$, inner $=35$ — both large, neither ignorable.

With inner term forced to $0$: ${\Delta }_{\text{align}}=U^{\ast }({\pi }^{\ast })-U^{\ast }({\pi }_{\mathcal{L}}^{\ast })+0=U^{\ast }({\pi }^{\ast })-U^{\ast }({\pi }_{\mathcal{L}}^{\ast }).$ A perfectly inner-aligned agent faithfully pursues $\mathcal{L}$ — so if $\mathcal{L}$ is a bad target (outer term large), you now have a powerful, faithful optimiser of the wrong thing, which the parent note flags as equally dangerous.

Let $v_0=U^{\ast }({\pi }^{\ast })$ and let $v_k=U^{\ast }({\pi }_{{\mathcal{L}}_k}^{\ast })$ for $k=\mathrm{1..}n$, with $v_{n+1}=U^{\ast }({\pi }_{\text{learned}})$. Telescoping (add and subtract each intermediate, all middle terms cancel just like Exercise 3.1): ${\Delta }_{\text{align}}=v_0-v_{n+1}={\sum }_{k=0}^n(v_k-v_{k+1}).$ Each bracket $(v_k-v_{k+1})$ is the utility lost at one hand-off (one mis-specification or mis-learning step). If every stage loses at least $\delta$, i.e. $v_k-v_{k+1}\ge \delta$ for all $n+1$ brackets: ${\Delta }_{\text{align}}\ge (n+1)\delta .$ Interpretation: alignment error compounds additively across a pipeline. A long chain of "mostly fine" objectives (each losing a little) can still lose a lot in total — the mathematical form of the parent note's "small failures compound."

Steel-man: genuinely, if we could evaluate $U^{\ast }$ we would have solved outer alignment already; the middle quantity is not directly observable, so we can't compute the terms numerically at design time. Rebuttal: the decomposition is a conceptual accounting tool, not a measurement recipe. Its value is (1) it tells you there are two independent failure surfaces, so passing training tells you nothing about the outer term and low true value tells you nothing about which term caused it (Exercise 2.2); (2) it directs which research tool addresses which term — objective design and Debate and Amplification shrink the outer bracket; Interpretability, Adversarial Training and Corrigibility shrink the inner one. A framework can be action-guiding even when its terms are not individually measurable, just as "signal + noise" guides engineering without measuring noise directly.

The alignment gap splits into an outer term and an inner term. The middle policy inserted in the decomposition is ::: ${\pi }_{\mathcal{L}}^{\ast }$, the optimal policy for the written objective. A zero outer term still leaves danger in ::: the inner term (e.g. deceptive alignment). For an $n$-stage pipeline losing $\delta$ each, the gap is at least ::: $(n+1)\delta$ (a sum, grows linearly).