Foundations — Goal misgeneralization

This page assumes nothing. Before you can read the parent note's formulas, you need to know exactly what each squiggle means and what picture lives behind it. We build them in order — each one uses only the ones before it.

1. The agent, states, and actions

Before any math, meet the three actors.

The picture: imagine a photograph of the game screen ($s$) and an arrow the agent draws on it saying what to do next ($a$).

Why the topic needs it: goal misgeneralization is about an agent choosing actions in states. Every reward below is scored on a state-action pair $(s,a)$ — "how good was doing $a$ in situation $s$?"

2. Reward: the score the agent chases

The picture: after the agent's arrow, a scoreboard flashes a number. That number is $R(s,a)$.

The parent note uses two reward functions, and keeping them apart is the whole game:

Why "proxy"? A proxy is a stand-in. "Move right" stands in for "get coin" — and works, until it doesn't.

3. Distributions: the bag of situations

The agent doesn't see one state — it sees a whole collection of them, some common, some rare.

The parent uses two bags:

• $D_{train}$ — the situations seen during training (all the practice levels).
• $D_{deploy}$ — the situations met after release, in the real world.

The picture: two bags of screenshots. The training bag might only contain "coin-on-right" screenshots. The deployment bag contains screenshots we never practised on — like "coin-on-left."

4. Expectation $\mathbb{E}$: the average over the bag

Now we need one tool to summarise a whole bag with a single number.

Why this tool and not just adding? A plain sum would over-count rare situations and drown in the size of the bag. An average asks the right question: "on a typical draw from this bag, what do I get?" That's exactly the question we want when comparing training to deployment.

The picture: pour every screenshot's score into one bucket, then read off the balance point — that balance point is $\mathbb{E}$.

The parent uses this to define the mean of each reward. Both rewards get their own mean, over the same training bag: ${\mu }_{true}={\mathbb{E}}_{D_{train}}[R_{true}],{\mu }_{proxy}={\mathbb{E}}_{D_{train}}[R_{proxy}]$ which read "the average true-score" and "the average proxy-score" over the training bag. The symbol $\mu$ (Greek "mu") is the standard shorthand for a mean; the subscript tells you which reward it belongs to.

5. Spread: variance and standard deviation $\sigma$

An average alone can't tell you if scores are all clustered or wildly scattered. We need a spread — and, just like the mean, each reward gets its own.

Whenever you see a bare ${\sigma }^2={\mathbb{E}}_{D_{train}}[(R-\mu {)}^2]$ in the parent note, read it as a template: substitute $R_{true},{\mu }_{true}$ to get ${\sigma }_{true}$, or $R_{proxy},{\mu }_{proxy}$ to get ${\sigma }_{proxy}$. The formula is the same shape; only the labels change.

Why square the distance? Distances above and below the mean would cancel if we just added them. Squaring makes every gap positive so they accumulate instead of cancelling; the square root at the end puts us back in the original units.

The picture: a cloud of scores around the mean. A fat cloud = big $\sigma$; a tight cloud = small $\sigma$.

Why the topic needs $\mu$ and $\sigma$: they are the two ingredients the correlation formula needs. You cannot understand $\rho$ (next) without them.

6. Correlation $\rho$: do two scores move together?

This is the star of the parent's derivation. We have $R_{true}$ and $R_{proxy}$, and now each has its own mean (${\mu }_{true},{\mu }_{proxy}$) and its own spread (${\sigma }_{true},{\sigma }_{proxy}$). We want one number saying "do these two rise and fall together?"

${\rho }_{train}=\frac{{\mathbb{E}}_{D_{train}}[(R_{true}-{\mu }_{true})(R_{proxy}-{\mu }_{proxy})]}{{\sigma }_{true}{\sigma }_{proxy}}$

Let's read this piece by piece, because you now know every symbol:

1. $(R_{true}-{\mu }_{true})$ = "how far above/below its own average is the true score right now?" This step is called centering — it slides the scores so their average sits at zero. The proxy factor $(R_{proxy}-{\mu }_{proxy})$ does the same for the proxy score.
2. Multiply the two centered scores. When both are above average (both positive) or both below (both negative), the product is positive → they moved together. When they disagree, the product is negative.
3. $\mathbb{E}[\dots ]$ averages this agreement over the whole bag.
4. Divide by ${\sigma }_{true}{\sigma }_{proxy}$. This step is scaling — it strips out the raw size of the scores so the answer always lands in $[-1,1]$.

The picture: plot every situation as a dot with true-score on one axis, proxy-score on the other. $\rho \approx 1$ = the dots fall on a rising line. $\rho \approx 0$ = a shapeless blob.

The deployment correlation ${\rho }_{deploy}$ — spelled out

The parent note also writes a correlation at deployment, using primed quantities. The prime symbol ${}^{\prime }$ just means "the same thing, but recomputed on the deployment bag $D_{deploy}$ instead of the training bag." Nothing new is invented — every mean and spread is the exact same formula from Sections 4–5, with $D_{train}$ swapped for $D_{deploy}$:

So ${\rho }_{deploy}$ asks the identical question as ${\rho }_{train}$ — "do the two rewards move together?" — but over the real-world bag. The primes are the only difference.

The symbol $\ll$ means "much less than."

7. Policy $\pi$ and the misalignment gap $\Delta$

The picture: deterministic = a signpost pointing one way at each state; stochastic = a spinner at each state whose slice sizes are the probabilities $\pi (a\mid s)$.

The parent uses two policies:

• ${\pi }_{proxy}$ — the rulebook that maximises the proxy score (what the agent actually became).
• ${\pi }_{true}$ — the ideal rulebook that would maximise the true score (what we wished for).

For a stochastic policy, "the true reward it earns in state $s$" means the expected reward over its own action-spinner, ${\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[R_{true}(s,a)]$. In the deterministic case this collapses to just $R_{true}(s,\pi (s))$, which is the form the parent writes.

The picture: two scores side by side in each reachable deployment state — what the misaligned agent gets vs. what a perfect agent would get. $\Delta$ is the height of the tallest gap you can find.

8. One more symbol: $d_{KL}$

The parent mentions $d_{KL}(D_{deploy}\Vert D_{train})$ as an informal size-of-shift gauge.

How the foundations feed the topic

Related failure modes that build on these same symbols: Outer Alignment (choosing $R_{true}$ well), Inner Alignment Problem (why $R_{proxy}$ can differ from $R_{true}$ inside the model), Reward Hacking, Robustness, and Interpretability (tools to inspect which goal the agent really learned). Deeper machinery lives in Mesa-Optimization and Deceptive Alignment.

Equipment checklist

This is a self-test. Each line below is written as Question ::: Answer — in the vault, the ::: marks a reveal line: you see the question, and the answer stays hidden until you click to reveal it. Cover the right side, try to answer, then check.