Foundations — Outer vs inner alignment

Before you can read the parent note Outer vs inner alignment, every squiggle it uses must mean something concrete to you. This page builds each one from nothing. Read top to bottom — each symbol is earned before it is used.

1. A policy — the symbol $\pi$

The picture. Imagine a robot standing in a room. It sees the room (a situation), flips to the matching page in its rulebook, and reads off an action. The whole rulebook is one policy. A different rulebook is a different policy.

Why the topic needs it. Alignment is about behaviour, and behaviour is exactly what a policy encodes. When we later ask "did the machine do what we wanted?", we are comparing rulebooks. So every important object in the parent note is a specific policy.

2. Scoring behaviour — utility $U^{\ast }$ and objective $\mathcal{L}$

We need a way to say how good a policy is. That means attaching a number to it.

The picture. Two judges scoring the same figure-skating routine. $U^{\ast }$ is the perfect judge who sees everything we care about. $\mathcal{L}$ is the judge we could actually hire — usually pretty good, sometimes fooled.

3. Averaging over an uncertain world — ${\mathbb{E}}_{\pi }[\cdot ]$

A policy acting in the world doesn't always get the same result — dice get rolled, environments differ. So a single score isn't enough; we need an average score.

Why this tool and not just $\mathcal{L}$ directly? Because outcomes are random. Asking "what score does this policy get?" has no single answer — but "what score does it get on average?" does. Averaging is the tool that turns a cloud of possible results into one comparable number. That is the only reason $\mathbb{E}$ appears.

4. Choosing the best — $\arg \max$

Now we can score policies. The next question is: which policy scores highest?

The picture. Line every policy up along the ground and let each one's score be its height. $\max$ points at the ceiling; $\arg \max$ points at the policy standing tallest.

Why the topic needs it. The parent note defines its three star policies as "the best policy for goal X." "Best" = $\arg \max$. Without this symbol we could not name the ideal cleaner or the ideal training-scorer at all.

5. The learner's own hidden goal — mesa-objective $\tilde{\mathcal{L}}$

Here is the twist that makes inner alignment its own problem.

The picture. Two nested loops. The outer loop is us + the learning algorithm, turning the crank toward $\mathcal{L}$. Inside sits the model, which has quietly grown its own little compass $\tilde{\mathcal{L}}$. If that compass points the same way as $\mathcal{L}$, great. If not — inner misalignment.

Inner alignment is simply the demand that $\tilde{\mathcal{L}}=\mathcal{L}$. All of Deceptive Alignment and Goal Misgeneralization are stories about $\tilde{\mathcal{L}}\ne \mathcal{L}$.

6. Putting numbers to failure — the alignment gap ${\Delta }_{\text{align}}$

Notice we score both policies with the same judge $U^{\ast }$ — because the only fair way to say "we lost value" is to measure both by what we truly wanted. Then the parent note splits this gap into an outer piece and an inner piece by adding and subtracting $U^{\ast }({\pi }_{\mathcal{L}}^{\ast })$ — a trick that changes nothing (the extra terms cancel) but reveals the two failures separately.

Prerequisite map

How these connect onward

• The outer piece of the gap is the world of Specification Gaming, Reward Hacking, and Goodhart's Law — writing down the wrong $\mathcal{L}$.
• The inner piece is Goal Misgeneralization and Deceptive Alignment — the wrong $\tilde{\mathcal{L}}$.
• Tools to detect these gaps: Interpretability, Adversarial Training, Debate and Amplification, plus safety properties like Corrigibility and pressures like Instrumental Convergence.

Equipment checklist