Foundations — Constitutional AI overview

This page introduces every symbol the parent note assumes, in the order that each one leans on the one before it. Nothing is used before it is drawn.

0. The cast of symbols (what you will meet)

Before we build them, here is the full alphabet the parent note throws at you. If any of these look like hieroglyphics right now, that is exactly why this page exists.

We now earn each row.

1. $x$ and $y$ — prompt and response

The picture: think of a vending machine. You press a button — that press is $x$. A snack drops out — that snack is $y$. Press the same button twice and you might get slightly different snacks ($y_1$, $y_2$) because the model has some randomness.

Why the topic needs it: the whole method is a factory of $(x,y)$ pairs — first good revised pairs (Stage 1), then pairs of competing answers to be judged (Stage 2). Everything downstream is a function of these two letters.

2. $y_w$, $y_l$ and the symbol $\succ$ — winner, loser, "is preferred"

Once a judge looks at $y_1$ and $y_2$ and picks a favourite, we rename them by their role, not their number.

The picture: a talent show with exactly two contestants. The judge raises a paddle toward one of them. The paddled one is $y_w$; the other is $y_l$. The arrow-like $\succ$ is the paddle pointing left-to-right: this beats that.

Why the topic needs it: the parent note's Bradley–Terry model is entirely about "how likely is the winner to beat the loser?" We can't write that until "winner" and "loser" have names.

3. $r_{\theta }(x,y)$ — the hidden quality score

The judge said "$y_w$ beats $y_l$" — a yes/no fact. To do maths we want a number measuring how good each answer is.

The picture: a judge's scorecard. Feed in (question, answer); a needle swings to a number. As we train, we tighten the internal screws $\theta$ so the needle agrees with the observed winners.

Why this tool and not just "good/bad"? A raw label ("winner") gives no magnitude. With a number we can say answer A is far better than B, or only slightly — and, crucially, we can take derivatives of a number to train. This is why we invent $r_{\theta }$. See Reward modeling.

4. $e^{(\cdot )}$, $\sigma$, and $P$ — turning a score difference into a probability

We have two numbers, $r_{\theta }(x,y_w)$ and $r_{\theta }(x,y_l)$. We want a probability — a number between 0 and 1 — saying "how likely is the winner to actually win?" We need a machine that takes any real number and squashes it into $[0,1]$.

4a. The exponential $e^z$

Why this tool? Scores can be negative. Probabilities can't. Exponentiating forces positivity, so a ratio of exponentials is guaranteed to sit safely in $[0,1]$.

4b. The sigmoid $\sigma$

Before the formula, let us give the two scores short names so the algebra stays readable.

The picture: a smooth "S" ramp. Far left it hugs $0$, far right it hugs $1$, and it crosses exactly $\frac{1}{2}$ at $z=0$.

Now watch the parent note's key identity fall out. Let $z=r_w-r_l$ be the score gap:

$\frac{e^{r_w}}{e^{r_w}+e^{r_l}}=\frac{1}{1+e^{r_l-r_w}}=\frac{1}{1+e^{-z}}=\sigma (z)$

What we did: divided top and bottom by $e^{r_w}$. Why: it reveals the whole thing depends only on the difference $z=r_w-r_l$, not the two raw scores. What it looks like: the S-curve of §4a, read at the point $z=r_w-r_l$.

This is exactly the Bradley-Terry preference model the parent uses.

5. $\log$ and $\mathcal{L}$ — from probability to a loss to minimise

We can now predict how likely each observed win was. Training means: tune $\theta$ so the wins we actually saw come out as likely as possible. Multiplying many probabilities together gives tiny unwieldy numbers, so we take logarithms to turn products into sums.

The picture: a curve that is $0$ at $p=1$ (certain, no surprise) and plunges to a deep pit as $p\to 0$ (a thing we said was likely but that failed — huge penalty).

6. $\mathbb{E}[\cdot ]$ — the averaging bracket

The loss above was for one comparison. A dataset has thousands. We need the average loss over all of them.

The picture: a giant bin of scorecards; reach in, read a card's loss, repeat many times, take the average. That average is what we actually minimise.

Why the topic needs it: it lets the parent note write one clean formula that stands for "do this over the whole dataset and average," instead of a messy sum with indices.

So the parent's reward-model loss reads, fully decoded:

${\mathcal{L}}_r=-\underset{\text{average over all judged pairs}}{\underbrace{{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}}}\left[\underset{\text{punish surprise (base }e)}{\underbrace{\log }}\underset{\text{gap}\to \text{prob}}{\underbrace{\sigma }}\right(\underset{\text{score gap }z}{\underbrace{r_w-r_l}}\left)\right]$

Every piece is now something you built by hand. This is the same machinery used in RLHF and DPO Direct Preference Optimization — only the source of the pairs differs.

7. ${\pi }_{\varphi }$, ${\pi }_{\text{ref}}$, and ${\max }_{\varphi }$ — the policy we steer

The reward model scores answers. But the thing we ultimately want to improve is the answer-generator itself.

The picture: a dartboard where each possible answer is a spot, and the policy is a heat-map of where the dart is likely to land. Training reshapes the heat-map to favour high-reward answers.

8. $\mathrm{KL}(\cdot \Vert \cdot )$ and $\beta$ — the leash

If we only maximise reward, the policy will find cheap tricks that fool the score machine (the parent's "As an AI, I strictly comply!!!" example). We need a term that punishes drifting away from sane, fluent language. That penalty — hinted at in §7 — is built now.

The picture: a dog (${\pi }_{\varphi }$) on a leash held at the anchor post (${\pi }_{\text{ref}}$). Reward is a tasty treat pulling the dog forward; the leash tension is $\beta \cdot \mathrm{KL}$ pulling it back. The dog settles where treat-pull balances leash-pull.

Now the two halves — the reward pull from §7 and the leash from §8 — combine into the parent's full objective, fully readable:

${\max }_{\varphi }\underset{\text{grab reward (treat forward)}}{\underbrace{{\mathbb{E}}_{x,y\sim {\pi }_{\varphi }}[r_{\theta }(x,y)]}}-\underset{\text{leash: don’t wander (pull back)}}{\underbrace{\beta \mathrm{KL}({\pi }_{\varphi }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x))}}$

This is the same PPO Proximal Policy Optimization objective used in RLHF; see also Red teaming and harmlessness for how the harmful prompts that feed this pipeline are sourced.

Prerequisite map

Each box is one section above; the two chains (reward-model chain and policy chain) meet at the RLAIF objective, which is what the parent Constitutional AI overview page assumes you already own.

Equipment checklist

Cover the right side and test yourself. If any answer is fuzzy, reread that section before the parent note.