Visual walkthrough — Constitutional AI overview

We are going to answer one question: after the AI judge has picked winners and losers, what exact number should training push up or down?

---

Step 1 — A judge that only says "this one"

WHAT. Start with the rawest thing we have. Give the model a prompt — call it $x$ (just a name for "the question we asked"). The model produces two answers. We name them $y_1$ and $y_2$ (again, just names — "answer one" and "answer two"). An AI judge reads both and points at one: "this response follows the principle better." The one it points at we rename $y_w$ (winner), the other $y_l$ (loser).

WHY. Everything downstream needs a number to optimise. But a pointing finger is not a number — it is a category, a yes/no. Step 1 just fixes vocabulary so the next steps have something to grip.

PICTURE. The judge is a gate that takes two arrows in and lets exactly one (red) out.

This is exactly the Bradley-Terry preference model setup, and it is identical to RLHF — only the judge is an AI instead of a human.

---

Step 2 — Give each answer a hidden "goodness" score

WHAT. Imagine every answer secretly carries a single number: how good it is according to the constitution. We write this number $r_{\theta }(x,y)$ and read it as "the reward of answer $y$ to prompt $x$." The little $\theta$ (Greek "theta") just means "this scorer has dials we can tune."

$$r_{\theta }(x,y)=\underset{\text{a score}}{\underbrace{r}}\underset{\text{tunable dials}}{\underbrace{\theta }}(\underset{\text{prompt}}{\underbrace{x}},\underset{\text{an answer}}{\underbrace{y}})$$

WHY. Why invent a hidden score at all? Because a raw ranking ("A beats B") does not compose — if A beats B and B beats C, how much better is A than C? A number on a line answers that for free: put every answer on a ruler, and distance on the ruler is strength of preference. That is the single most useful reason to convert a choice into a score.

PICTURE. Both answers sit on a number line. $y_w$ sits to the right of $y_l$; the gap between them (red) is what matters — not the absolute positions.

---

Step 3 — Turn the score-gap into a probability with $\sigma$

WHAT. The judge is not perfect; sometimes it would flip. So instead of "$y_w$ always wins," we ask: how likely is $y_w$ to be preferred? We need a machine that eats the score-gap (any real number, from $-\infty$ to $+\infty$) and outputs a probability (a number squeezed into $0$ to $1$). That machine is the logistic function $\sigma$:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

WHY this tool and not another? We specifically need something that:

• maps every real number into $(0,1)$ — a probability can't be negative or above 1;
• gives $\sigma (0)=0.5$ — a zero gap should mean a coin-flip, total uncertainty;
• is smooth and increasing — a bigger gap should mean higher confidence, always.

The logistic curve is the simplest S-shape with all three properties, which is why Bradley–Terry picks it. Plug the gap $z=r_{\theta }(x,y_w)-r_{\theta }(x,y_l)$ in:

$$P(y_w\succ y_l\mid x)=\sigma (\underset{\text{winner score}}{\underbrace{r_{\theta }(x,y_w)}}-\underset{\text{loser score}}{\underbrace{r_{\theta }(x,y_l)}})$$

The symbol $\succ$ reads "is preferred over."

PICTURE. The S-curve. Mark three spots: gap $=0\to 0.5$; large positive gap $\to$ near $1$; large negative gap $\to$ near $0$ (this is the degenerate case where the "winner" actually scores worse — the model is confidently wrong, and training will punish it hard next).

Recall What does

$\sigma (0)$ equal, and what does it mean here? $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$. A zero score-gap means the model is totally undecided — a 50/50 coin flip between the two answers.

---

Step 4 — Score the scorer: the negative-log-likelihood

WHAT. We now have a predicted probability that the winner wins. The judge already told us the winner did win — so we want our predicted probability to be high (close to 1). We measure "how badly did we predict?" with the negative logarithm:

$$\text{loss on one pair}=-\log \sigma (r_{\theta }(x,y_w)-r_{\theta }(x,y_l))$$

WHY $-\log$? Look at what $-\log (p)$ does as $p$ (our predicted probability of the correct answer) moves:

• $p\to 1$ (we were right and confident) $\Rightarrow -\log (1)=0$: no penalty.
• $p\to 0$ (we were confidently wrong) $\Rightarrow -\log (0)=+\infty$: enormous penalty.

So $-\log$ is a pain-meter that is gentle when we're right and vicious when we're confidently wrong. Averaging this pain over the whole AI-labelled dataset $\mathcal{D}$ gives the reward-model loss:

Minimising ${\mathcal{L}}_r$ = tuning the dials $\theta$ so the winner's score sits reliably above the loser's.

PICTURE. The pain-meter curve: penalty on the $y$-axis, our predicted probability on the $x$-axis — flat near $p=1$, shooting to the sky near $p=0$ (red).

---

Step 5 — Use the scorer to steer the actual model (and how PPO really climbs)

WHAT. The reward model $r_{\theta }$ only judges text. Now we tune the model that writes text — call it the policy ${\pi }_{\varphi }$ (Greek "pi" for policy, $\varphi$ = its own dials). The dream is:

$$\underset{\varphi }{\max }\underset{\text{over answers our model writes}}{\underbrace{{\mathbb{E}}_{x,y\sim {\pi }_{\varphi }}}}\left[\underset{\text{their reward}}{\underbrace{r_{\theta }(x,y)}}\right]$$

Read it: "adjust the dials $\varphi$ so that the answers our model samples tend to score high." But you cannot just do gradient ascent on this directly — and understanding why is the whole point of PPO.

WHY not vanilla policy gradient? Two problems, each with a fix drawn below.

1. Raw reward is a noisy signal. An answer with reward $8.3$ — is that good? Only relative to what this model usually scores on that prompt. So we subtract a baseline $b(x)$ (the average reward we'd expect) and keep only the surplus. That surplus is the advantage: $A(x,y)=\underset{\text{reward we got}}{\underbrace{r_{\theta }(x,y)}}-\underset{\text{what we expected (baseline / value)}}{\underbrace{b(x)}}$ Positive $A$ = "better than usual, do more of this"; negative $A$ = "worse than usual, do less." Subtracting the baseline does not change where the maximum is, but it hugely reduces the variance of the update — without it the gradient jitters so violently that training is useless. The baseline $b(x)$ is itself learned by a small value model that predicts expected reward.

2. Big steps blow up. Each update samples answers from the current policy, so a single wild step can push ${\pi }_{\varphi }$ into garbage from which the estimates are worthless. PPO fixes this by optimising a surrogate that only trusts small changes. Let $\rho (\varphi )=\frac{{\pi }_{\varphi }(y\mid x)}{{\pi }_{{\varphi }_{\text{old}}}(y\mid x)}$ be the probability ratio — how much more (or less) likely the new policy makes an answer versus the old one. PPO maximises the clipped objective: ${\mathcal{L}}^{\text{PPO}}=\mathbb{E}\left[\min \right(\underset{\text{normal push}}{\underbrace{\rho A}},\underset{\text{push, but capped}}{\underbrace{\text{clip}(\rho ,1-ϵ,1+ϵ)A}}\left)\right]$

The $\min$ + clip together say: reward genuine improvement, but the moment a step tries to move too far, freeze the incentive so the optimiser can't sprint off a cliff. That is PPO's trust region.

PICTURE (5a). The full pipeline PPO actually runs: sample trajectories (answers) → score each → subtract baseline to get the advantage → turn advantages into the clipped gradient that nudges the policy.

PICTURE (5b). The clip in action: the flat red shelves outside $[1-ϵ,1+ϵ]$ are where PPO refuses to reward further movement — the practical "small steps only" leash.

---

Step 6 — The KL leash: don't wander off into gibberish

WHAT. Even with PPO's per-step clip, over many steps the policy can slowly crawl far from the trusted starting model ${\pi }_{\text{ref}}$ (the SL-CAI model — fluent, on-topic). So we add a global penalty for how far ${\pi }_{\varphi }$'s whole distribution of word-choices has drifted. That distance is the Kullback–Leibler divergence. Here is its actual definition — no black box:

Plugging $p={\pi }_{\varphi }$ (our current policy) and $q={\pi }_{\text{ref}}$ (the anchor), $\mathrm{KL}$ becomes a single number: how weird has our model's output distribution become versus the trusted one? Now the full objective:

Term by term:

• $\mathbb{E}[r_{\theta }]$ ::: pulls the policy toward high reward (the pull);
• $\mathrm{KL}({\pi }_{\varphi }\Vert {\pi }_{\text{ref}})$ ::: grows (per its definition above) as the policy's word-choices diverge from the reference (the tether);
• $\beta$ (Greek "beta") ::: sets how strong the tether is — big $\beta$ = short leash (stay safe, gain less reward); small $\beta$ = long leash (chase reward, risk gibberish).

WHY. The KL term converts "don't cheat" from a wish into a cost. If the policy finds a reward hack, its KL shoots up, $-\beta \mathrm{KL}$ subtracts, and the total objective can actually drop — so the optimiser refuses the hack.

PICTURE. Same reward hill, but now the policy is on a red rubber band anchored at ${\pi }_{\text{ref}}$. It reaches toward the peak but the band tugs it back — it settles at a balanced spot, not the cheating peak.

---

Step 7 — Two degenerate cases you must be able to handle

WHAT. Real training hits corners. Two matter:

1. The judge ties ($r_{\theta }(x,y_w)=r_{\theta }(x,y_l)$). Gap $=0$, so $P=\sigma (0)=0.5$, and the loss is $-\log (0.5)\approx 0.693$ — the maximum uncertainty penalty. Training receives a strong "you can't tell these apart, learn to" signal. Nothing breaks.
2. $\beta =0$ (no leash). The KL term vanishes; the objective is pure reward-chasing. This is the reward-hacking regime — text degrades into repetitive canned phrases. As $\beta \to \infty$ the leash is infinitely stiff: the policy can't move at all and never improves. The useful world lives between these extremes.

WHY. A method you can't stress-test at its boundaries is a method you don't understand. These two cases are the boundaries.

PICTURE. A dial for $\beta$: far left ($\beta =0$) = "gibberish / reward hack"; far right ($\beta \to \infty$) = "frozen, no learning"; the red sweet-spot in the middle.

Recall What is the loss on a perfectly tied pair, and why does it make sense?

$-\log \sigma (0)=-\log (0.5)\approx 0.693$. It's the largest ordinary penalty because a tie means the scorer has learned nothing useful about which answer is safer — so it gets pushed hard.

---

The one-picture summary

Everything above, in one flow: prompt → two answers → AI judge picks a winner → Bradley–Terry turns the score-gap into a probability → $-\log$ loss trains the reward model → PPO (advantage + clip) climbs the reward → the KL leash keeps it honest.

Recall Feynman retelling — the whole walkthrough in plain words

A referee looks at two answers and points at the better one. We give every answer a secret score on a ruler; only the gap between the two scores matters. We bend that gap through an S-shaped curve so it becomes a probability — "how sure are we the winner really is better?" — where a zero gap means a coin-flip. Then we grade our own scorer with a pain-meter: no pain when we confidently agree with the referee, screaming pain when we're confidently wrong. Once the scorer is trained, we let it coach the real model to write higher-scoring answers — but carefully: we compare each answer to what the model usually scores (the advantage) so the signal isn't noisy, and we only allow small steps (PPO's clip) so it can't leap off a cliff. On top of that we tie the model to a rubber band anchored on its trustworthy old self — the KL distance — so it can't cheat by babbling one magic phrase. Loosen everything and it talks gibberish; tighten it fully and it never learns. The whole art is finding the middle.

---

Prerequisites & neighbours: Bradley-Terry preference model · Reward modeling · PPO Proximal Policy Optimization · KL divergence · RLHF · DPO Direct Preference Optimization · Prompt engineering · Red teaming and harmlessness · Hinglish version