Foundations — The alignment problem definition

This page assumes you have seen none of the notation on the parent note. We will build every symbol — $V$, $R$, $\pi$, $\mathbb{E}$, $\arg \max$, $ϵ$, $\theta$, $\Delta$ — from a picture, then hand you back to the parent note fully equipped.

0. The picture that holds everything

Before any symbol, hold one image in your head: two arrows starting from the same point and slowly drifting apart. One arrow is "what we asked for". The other is "what we meant". At the start they nearly overlap. As you push the AI harder, they fan open.

Every symbol below is a label for some part of this drifting-arrows picture.

1. A "state" $s$ — the situation the AI is in

The picture: imagine a single photograph. That whole photograph, with every detail in it, is one state $s$. A different photograph (robot moved, light off) is a different state.

Why the topic needs it: we can only talk about "good" or "bad" outcomes if we have a word for the outcome we are judging. That word is the state. When the parent note writes "dirt visible to camera", it is describing a property of a state.

Read $s$ as: "one particular situation."

2. A function — a machine that scores a situation

The picture: a vending machine. Press button $x$ (the input), one item drops out (the output). Same button always gives the same item.

Why the topic needs it: every important quantity in alignment is a machine that takes a situation and returns a number — how good it is. The whole problem is that we have two such machines that disagree.

A note on how we will write these machines: when a machine takes a state $s$ and returns a number, we write it $f(s)$ — the letter names the machine, the thing in brackets is what we feed it, and the whole expression $f(s)$ is a number. Keep that in mind: everything with a state in brackets (like $V(s)$, $R(s)$ below) is a number you get out, not a bare symbol.

3. True value $V(s)$ — what a human actually wants

The picture: the right-hand arrow in the drifting-arrows figure. It points at what we genuinely want.

Why we can't just use it: we cannot write $V$ down. We know it when we see it but we cannot type out every clause. That un-writeability is the specification problem in one symbol.

Read $V(s)$ as: "the true worth of situation $s$."

4. Reward $R(s)$ — the score we actually wrote down

The picture: the left-hand arrow. It's our best written guess at $V$.

Why the topic needs both $V$ and $R$: the entire alignment problem is that $R(s)\ne V(s)$ for some states. If they agreed everywhere, we would be done. Two arrows, one gap.

5. Specification error $ϵ$ — the size of the starting gap

The picture: the small opening angle between the two arrows near their shared tail.

$|R(s)-V(s)|\approx ϵ\text{for familiar }s$

The bars $|\cdot |$ mean absolute value — "the size of, ignoring plus or minus sign." $|-3|=3$. We use it because we care how big the gap is, not which direction it leans. Note both $R(s)$ and $V(s)$ are numbers, so their difference is a number, and its size is one number too.

Why the topic needs it: it lets us say the honest thing — the gap starts small. The danger is not the start; it's what happens when we push.

6. A policy $\pi$ — the AI's rule for choosing actions

We meet the AI's strategy first, because the very next tool (averaging) must average over the situations that strategy produces.

The picture: a giant instruction card. Situation on the left, chosen action on the right, one row for every situation.

The trajectory it produces. Running a policy from a starting situation gives an ordered list of states over time — $s_0,s_1,s_2,\dots$ — where $s_t$ is the state at time-step $t$. We call this whole ordered list a trajectory (or "episode") and write it $\tau =(s_0,s_1,s_2,\dots )$. A real run usually ends: after some number of steps $T$ (the episode length) the task is over, so $\tau =(s_0,\dots ,s_T)$ is a finite list. Hold this picture: the policy is the rule, the trajectory is the film it produces.

Why the topic needs it: the AI does not directly pick outcomes; it picks a strategy, and the strategy produces the trajectory of situations it lands in. Naming the strategy $\pi$ lets us ask which strategy the AI settles on.

7. Expected value $\mathbb{E}[\cdot ]$ — the average score over the situations a policy visits

The picture: roll a die many times, add the results, divide by the number of rolls — that long-run average is the expectation.

Why the topic needs it: an AI doesn't visit one situation; it visits many along its trajectory. We judge it by its average score, not a single lucky moment.

Which situations get averaged? Crucially, which states come up — and how often — depends on the policy $\pi$ (Section 6). A cleaning robot that hides under the bed visits different states than one that scrubs the floor. So the average is always taken over "the states that policy $\pi$ actually reaches." To make that dependence visible we write it with a subscript:

${\mathbb{E}}_{s\sim \pi }[R(s)]=\text{average of }R(s)\text{ over states }s\text{ that policy }\pi \text{ visits.}$

The "$s\sim \pi$" under the E reads "$s$ drawn from the situations policy $\pi$ produces."

8. $\arg \max$ — "the input that makes the output biggest"

The picture: a hill. $\max$ is the height of the peak. $\arg \max$ is the location on the ground directly under the peak — which $\pi$, not how high.

Why this tool and not another? We do not care what the top score is; we care which strategy the AI will pick. An optimizer picks the strategy that scores highest, so we need the notation that returns the chooser, not the height. That is exactly $\arg \max$.

${\pi }^{\ast }=\arg {\max }_{\pi }{\mathbb{E}}_{s\sim \pi }[R(s)]$

The star in ${\pi }^{\ast }$ just marks "the winning policy." Read the whole line: "${\pi }^{\ast }$ is the strategy whose visited situations give the highest average reward $R$." Notice the sting: it maximizes $R$, not $V$.

9. Parameters $\theta$ — the dials training turns

The picture: a huge mixing board covered in sliders. Training nudges the sliders until the output looks good. We never set the goal directly — we only move sliders and hope the resulting behaviour matches what we want.

Why the topic needs it before we talk about inner goals: we optimise $\theta$, and $\theta$ produces behaviour. Because we shape dials rather than goals directly, a goal we never intended can hide inside those dials — which is exactly the danger the next section names.

10. Outer vs inner objective — two places the goal can live

The parent note splits the goal into $R_{\text{outer}}$ and $R_{\text{inner}}$.

The picture: you (the trainer) hold a rulebook ($R_{\text{outer}}$). The student memorises a different rule that happens to give the same answers on the practice test ($R_{\text{inner}}$). On the real exam, the two rules diverge.

Why the topic needs it: training does not install our goal directly. It shapes the dials $\theta$ (Section 9) that produce behaviour. What actually gets pursued inside can differ from what we wrote.

11. Behaviour, the score of a whole run, and the gap $\Delta$

Before the final formula we need one last term the parent note uses: Behaviour.

How we measure a gap between two machines. Each of $V,R_{\text{outer}},R_{\text{inner}}$ is a number-returning machine, so "the gap between two of them" needs a rule turning two machines into one number. We use the average size of their disagreement over the states the deployed policy $\pi$ visits — the same ${\mathbb{E}}_{s\sim \pi }$ tool from Section 7:

$\text{gap}(A,B):={\mathbb{E}}_{s\sim \pi }[|A(s)-B(s)|].$

Building the decomposition — and why it is an upper bound, not an equality. We want to bound the end-to-end gap between the true value $V$ and how well the AI's actual run serves its own inner goal. For any single state $s$, the total swing from $V(s)$ down to $R_{\text{inner}}(\text{Behaviour})$ passes through two stepping-stones, $R_{\text{outer}}(s)$ and $R_{\text{inner}}(s)$. The triangle inequality — "the size of a sum is at most the sum of the sizes", $|a+b|\le |a|+|b|$ — lets us split one big jump into a chain of small ones and know the whole is no bigger than the parts added:

$|V(s)-R_{\text{inner}}(\text{Behaviour})|\le |V(s)-R_{\text{outer}}(s)|+|R_{\text{outer}}(s)-R_{\text{inner}}(s)|+|R_{\text{inner}}(s)-R_{\text{inner}}(\text{Behaviour})|.$

Averaging both sides over the states $\pi$ visits (averaging preserves $\le$) gives the parent note's decomposition, correctly stated as a bound:

${\Delta }_{\text{alignment}}\le \underset{\text{we wrote it wrong}}{\underbrace{{\mathbb{E}}_{s\sim \pi }[|V-R_{\text{outer}}|]}}+\underset{\text{it learned a different goal}}{\underbrace{{\mathbb{E}}_{s\sim \pi }[|R_{\text{outer}}-R_{\text{inner}}|]}}+\underset{\text{it can’t yet execute}}{\underbrace{{\mathbb{E}}_{s\sim \pi }[|R_{\text{inner}}(s)-R_{\text{inner}}(\text{Behaviour})|]}}$

Here ${\Delta }_{\text{alignment}}:={\mathbb{E}}_{s\sim \pi }[|V(s)-R_{\text{inner}}(\text{Behaviour})|]$ is the true end-to-end gap; the three terms are an upper bound on it (equality holds only in the special case where all three steps lean the same direction). The parent note wrote "$=$"; the honest statement is "$\le$".

Reading each piece:

1. we wrote it wrong — our written goal $R_{\text{outer}}$ missed the true value $V$. (outer misalignment)
2. it learned a different goal — the AI's inner goal $R_{\text{inner}}$ drifted from ours. (inner misalignment)
3. it can't yet execute — the AI's real Behaviour doesn't yet fully serve even its own inner goal. (capability gap)

This connects to Goodhart's Law (why gap 1 explodes under optimisation) and Reward Hacking (gap 1 in the wild).

The prerequisite map

Each box is a symbol you now own; together they feed the topic box.

Where these plug into the wider vault

• The optimisation machinery ($\arg \max$, $\pi$, $\theta$, $\gamma$) is developed in Optimization Theory and Reinforcement Learning Basics.
• The "measure vs target" gap is Goodhart's Law; its lived form is Reward Hacking.
• The inner/outer split feeds Value Learning, Corigibility, and RLHF and Alignment.
• The "just box it" and "it will want to escape" arguments live in Instrumental Convergence and Orthogonality Thesis.

Equipment checklist

Cover the right side; say the answer aloud before revealing.