Exercises — Constitutional AI overview

• The stage that fine-tunes on revised answers is the Supervised Learning (SL) stage — the model critiques and rewrites its own answer against a principle, and we train on the (prompt → revised answer) pairs.
• The stage that produces AI-judged preference pairs is the RL stage (RLAIF) — two responses are compared by an AI judge, a reward model is trained, and PPO optimises the policy.

• $y_w$ = the winner (the response the AI judge preferred). $y_l$ = the loser.
• Inside is the score difference $r_{\theta }(x,y_w)-r_{\theta }(x,y_l)$. If the winner really does score higher, this difference is positive.
• $\sigma$ turns that difference into a probability "how likely was $y_w$ to be chosen". The $-\log$ then rewards the model for making that probability large. This is the Bradley-Terry preference model fit by maximum likelihood; see Reward modeling.

What: plug into Bradley–Terry. Why: the judge gave a categorical choice; we want the probability our reward model assigns to that same choice. $P=\sigma (1.2-0.4)=\sigma (0.8)=\frac{1}{1+e^{-0.8}}\approx 0.690$ On the S-curve figure, $z=0.8$ lands a bit right of centre → a bit above $0.5$. Loss: $-\log (0.690)\approx 0.371$ Moderate confidence, moderate loss.

What: score difference is $z=1.0-1.0=0$. What it looks like: dead centre of the S-curve. $P=\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$ $-\log (0.5)=\log 2\approx 0.693$ Meaning: $\log 2$ is the maximum "honest confusion" loss for a two-way choice — the model is saying "I genuinely can't tell these apart". Any real preference should push the loss below $0.693$.

Critique: "The response helps fabricate a false medical excuse, which is deception and violates $P$." (We force the model to name the specific violation — vague "this is bad" gives nothing to fix.) Revise: "I can't help write a false sick note. If you're genuinely unwell, most employers accept a same-day call or message; if you need time off for another reason, I can help you phrase an honest request for personal leave." The revision drops the harm yet keeps helpfulness by offering a legitimate path. This (prompt → revised) pair is what the SL stage trains on.

What changed: $\Delta \mathbb{E}[r_{\theta }]=+0.9$, $\Delta \mathrm{KL}=+2.0$. $\Delta J=\Delta \mathbb{E}[r_{\theta }]-\beta \Delta \mathrm{KL}=0.9-0.5\times 2.0=0.9-1.0=-0.1$ Down. The reward gain is outweighed by the KL penalty — the leash bites, blocking this reward-hack. See figure below.

Set $\Delta J=0$: $0=0.9-\beta \times 2.0\Rightarrow \beta =\frac{0.9}{2.0}=0.45$

• $\beta <0.45$: KL leash too loose → the hack raises $J$ → model exploits it. Danger.
• $\beta >0.45$: leash tight enough → hack loses → suppressed. So $\beta$ is the tuning dial trading "chase reward" against "stay fluent/on-distribution".

(a) Improvements: pass 1 = $0.80-0.30=0.50$; pass 2 = $0.30-0.18=0.12$; pass 3 = $0.18-0.12=0.06$. Clear diminishing returns (0.50 → 0.12 → 0.06). (b) Pass 3 buys a small harm gain ($0.06$) but quintuples over-refusal ($0.05\to 0.25$, a $+0.20$ jump). The model is becoming evasive/refusing safe requests — that damages helpfulness, one of CAI's two goals (helpful and harmless). Stop at two passes. Number of critique passes is a tuned hyperparameter, not "more = better".

The KL term pulls ${\pi }_{\varphi }$ toward ${\pi }_{\text{ref}}$. Whatever ${\pi }_{\text{ref}}$ is, that's the "home base" we refuse to stray far from.

• If ${\pi }_{\text{ref}}$ = raw base model, the RL policy is leashed to a model that hasn't yet learned the constitution — it would keep getting yanked back toward pre-constitution behaviour.
• If ${\pi }_{\text{ref}}$ = SL-CAI model, home base is already constitution-aligned and fluent. RL then only has to refine preferences on top of a good starting point, and the KL leash preserves that alignment. So SL first (bake in the constitution), then use it as the reference for RL. The two stages are deliberately chained.

Step 1 (probability). Bradley–Terry says the chance the judge prefers $y_w$ is $P=\frac{e^{r_{\theta }(x,y_w)}}{e^{r_{\theta }(x,y_w)}+e^{r_{\theta }(x,y_l)}}.$ Divide top and bottom by $e^{r_{\theta }(x,y_w)}$: $P=\frac{1}{1+e^{-(r_{\theta }(x,y_w)-r_{\theta }(x,y_l))}}=\frac{1}{1+e^{-\Delta }}=\sigma (\Delta ).$ Why: this rewrite reveals that only the difference $\Delta$ matters, not the raw scores. Step 2 (loss). Maximum likelihood says "make the observed choice as probable as possible", i.e. maximise $P$, i.e. minimise $-\log P$: $\mathcal{L}=-\log \sigma (\Delta ).$ Step 3 (always positive). $\sigma (\Delta )\in (0,1)$ so $\log \sigma (\Delta )<0$, hence $-\log \sigma (\Delta )>0$. Loss can approach $0$ but never reaches it. Step 4 (shrinks as $\Delta$ grows). $\sigma$ is increasing, so larger $\Delta \Rightarrow$ larger $\sigma (\Delta )\Rightarrow$ larger $\log \Rightarrow$ smaller $-\log$. Confident-and-correct ⇒ small loss. At $\Delta =0$, loss $=-\log 0.5=\log 2\approx 0.693$; for $\Delta <0$ (winner scored lower) the loss exceeds $\log 2$ — a big correction signal.

Magnitude is $|-(1-\sigma (\Delta ))|=1-\sigma (\Delta )$.

• At $\Delta =2.0$: $\sigma (2.0)=\frac{1}{1+e^{-2}}\approx 0.8808$, so magnitude $\approx 1-0.8808=0.1192$. Small — model already right and confident, barely nudge it.
• At $\Delta =-2.0$: $\sigma (-2.0)=\frac{1}{1+e^2}\approx 0.1192$, so magnitude $\approx 1-0.1192=0.8808$. Large — model confidently wrong (ranked loser above winner), strong push to fix. This is exactly the self-correcting behaviour we want: effort flows to the mistakes.

CAI = SL self-critique (bake constitution into weights) then RLAIF (AI-judge preferences → Bradley–Terry reward model → PPO with a KL leash). All the math above is just $\sigma$ turning score differences into probabilities, and KL keeping the policy honest.