Constitutional AI overview: Worked examples — AI-ML deep dive

This is the "grind every case" companion to Constitutional AI overview. The parent built the machinery (self-critique, the Bradley–Terry reward, PPO with a KL leash). Here we torture-test it: every sign of the reward gap, every degenerate input, every limit, plus a word problem and an exam twist.

Before any symbol appears, we re-earn it. If you see $σ$ , $r_{θ}$ , $β$ , or $KL$ and feel a gap, that's the parent's job — but each example below re-defines what it touches so you never guess.

The scenario matrix

The whole numeric side of CAI lives in two objects: the preference probability and the penalised objective. Everything a problem can throw at you is one cell below.

Cell	What varies	Concrete question	Example
A. Winner clearly better	reward gap $Δ = r_{w} - r_{l} > 0$ , large	$P$ and loss when $Δ = 1.5$	Ex 1
B. Loser mislabeled / gap negative	$Δ < 0$	judge picked worse answer, what's the loss?	Ex 2
C. Tie / degenerate	$Δ = 0$	two equally safe answers	Ex 3
D. Limiting behaviour	$Δ \to + \infty$ , $Δ \to - \infty$	what do $P$ and loss approach?	Ex 4
E. Batch loss	many pairs averaged	mean loss over a mini-dataset	Ex 5
F. KL bites (reward hacking)	reward $↑$ but KL $↑$	does the objective actually rise?	Ex 6
G. Tuning the leash	vary $β$	find $β$ where hack stops paying	Ex 7
H. Word problem	count human labels saved	real-world budget question	Ex 8
I. Exam twist	swap label source	is RLAIF math ≠ RLHF math?	Ex 9

Two formulas power cells A–E. First, the sigmoid — the function that turns a score difference into a probability between 0 and 1:

Cell A — Winner clearly better ( $Δ > 0$ , large)

Worked example Example 1 — the "everything is fine" case

After training, a safe answer scores $r_{w} = 2.0$ and an unsafe one scores $r_{l} = 0.5$ . Find $P (y_{w} ≻ y_{l})$ and the pair loss.

Forecast: Guess before computing — is $P$ closer to $0.5$ or to $1$ ? Is the loss big or tiny?

Reward gap. $Δ = 2.0 - 0.5 = 1.5$ . Why this step? The Bradley–Terry model only cares about the difference of scores, never their absolute size — so we collapse two numbers into one.
Probability. $P = σ (1.5) = \frac{1}{1 + e ^{- 1.5}} \approx 0.818$ . Why this step? $σ$ converts the gap into "how often the judge should prefer the winner." $0.818$ = fairly, not overwhelmingly, confident.
Loss. $L_{pair} = - lo g (0.818) \approx 0.201$ . Why this step? The training signal is the surprise at the observed choice. Small surprise ⇒ small gradient ⇒ tiny weight update.

Verify: $σ (1.5) > 0.5$ ✓ (winner favoured, positive gap). $- lo g (0.818)$ is small and positive ✓. Sanity: if we swapped winner/loser we'd expect loss $\approx - lo g (1 - 0.818) = - lo g (0.182) \approx 1.70$ — much bigger, matching the intuition "picking the worse answer hurts more." See Ex 2.

Cell B — Judge mislabeled, gap negative ( $Δ < 0$ )

Worked example Example 2 — when the AI judge is wrong

The dataset labels $y_{w}$ as winner, but the reward model disagrees: $r_{w} = 0.5$ , $r_{l} = 2.0$ . What loss does this pair contribute?

Forecast: Bigger or smaller than Ex 1's $0.201$ ? Why?

Gap. $Δ = 0.5 - 2.0 = - 1.5$ . Why this step? A negative gap means the reward model currently prefers the labeled loser — a conflict the training must fix.
Probability. $P = σ (- 1.5) = \frac{1}{1 + e ^{1.5}} \approx 0.182$ . Why this step? $σ$ is symmetric: $σ (- z) = 1 - σ (z)$ , so $0.182 = 1 - 0.818$ . The model only assigns $18.2%$ to the labeled winner — it's wrong and confident.
Loss. $L_{pair} = - lo g (0.182) \approx 1.704$ . Why this step? Large surprise ⇒ large gradient. This pair pushes hard to raise $r_{w}$ and lower $r_{l}$ .

Verify: $0.182 = 1 - 0.818$ ✓ (sigmoid symmetry). Loss $1.704 ≫ 0.201$ ✓ — the confidently-wrong pair drives most of the learning, exactly as gradient descent should.

Cell C — Tie / degenerate input ( $Δ = 0$ )

Worked example Example 3 — two equally safe answers

Both responses are polite refusals with a helpful alternative. The judge is genuinely unsure; the reward model gives $r_{w} = r_{l} = 1.3$ .

Forecast: What is $P$ when there's no preference?

Gap. $Δ = 1.3 - 1.3 = 0$ . Why this step? A tie is the degenerate case the matrix must cover — absolute score $1.3$ is irrelevant, only the difference matters.
Probability. $P = σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.5$ . Why this step? $σ$ crosses exactly $0.5$ at $z = 0$ — a coin flip, the honest answer for "these are equally good."
Loss. $L_{pair} = - lo g (0.5) = lo g 2 \approx 0.693$ . Why this step? This is the maximum-uncertainty loss for a single label — it's the loss the model pays whenever it can't tell winner from loser.

Verify: $σ (0) = 0.5$ exactly ✓. $- lo g (0.5) = ln 2 \approx 0.693$ ✓. Note this sits between Ex 1 ( $0.201$ ) and Ex 2 ( $1.704$ ) — a tie is more surprising than a correct call, less than a wrong one. ✓

Cell D — Limiting behaviour ( $Δ \to \pm \infty$ )

Worked example Example 4 — the edges of confidence

No numbers — reason about what happens as the reward gap blows up in either direction.

Forecast: Can $P$ ever equal exactly $1$ ? Can the loss ever hit exactly $0$ ?

Push $Δ \to + \infty$ . $σ (Δ) \to 1$ but never reaches it; $L_{pair} = - lo g σ (Δ) \to 0^{+}$ . Why this step? The S-curve flattens (right side of the figure). The model becomes near-certain, and the gradient nearly vanishes — no more to learn from this pair.
Push $Δ \to - \infty$ . $σ (Δ) \to 0^{+}$ ; $L_{pair} = - lo g (tiny) \to + \infty$ . Why this step? A confidently-wrong pair is punished without bound — this is what stops a stubborn reward model from ignoring a mislabel forever.
Consequence for CAI. In practice we never see infinite gaps because the KL leash (Cell F) keeps the policy — and thus the score range — bounded. Why this step? Connects the limit back to why $β$ exists.

Verify: $lim_{z \to \infty} σ (z) = 1$ , $lim_{z \to - \infty} σ (z) = 0$ , and $- lo g σ (z) \to 0$ as $z \to \infty$ , $\to \infty$ as $z \to - \infty$ ✓ (checked numerically at large $z$ in VERIFY).

Cell E — Batch loss over a mini-dataset

Worked example Example 5 — averaging the pain

Four AI-labeled pairs give gaps $Δ = {1.5, - 1.5, 0, 3.0}$ . Compute the mean reward-model loss $L_{r}$ .

Forecast: Which of the four pairs dominates the average?

Per-pair losses. Using $- lo g σ (Δ)$ : $- lo g σ (1.5) \approx 0.201$ , $- lo g σ (- 1.5) \approx 1.704$ , $- lo g σ (0) \approx 0.693$ , $- lo g σ (3.0) \approx 0.049$ . Why this step? The real reward-model loss $L_{r}$ is an expectation — i.e. the average of per-pair losses over the dataset $D$ .
Average. $L_{r} = \frac{0.201 + 1.704 + 0.693 + 0.049}{4} \approx \frac{2.647}{4} \approx 0.662$ . Why this step? Mean, because the parent's $E_{(x, y_{w}, y_{l}) \sim D}$ is "average over pairs."

Verify: The wrong pair ( $1.704$ ) contributes $\approx 64%$ of the total ✓ — confidently-wrong examples dominate the batch signal, as expected. Mean $\approx 0.662$ (checked in VERIFY).

Cell F — The KL term bites (reward hacking)

Now the second formula. The policy $π_{ϕ}$ tries to earn reward but is penalised for drifting from the reference (SL-CAI) model $π_{ref}$ .

The objective the parent wrote, for one prompt, is $J = reward E_{y \sim π_{ϕ}} [r_{θ} (x, y)] - β KL (π_{ϕ} ∥ π_{ref}) .$

Worked example Example 6 — does the hack pay off? (

$β = 0.5$ ) Two answers only: a normal answer $y_{1}$ and a hacky canned phrase $y_{2}$ ("As an AI, I strictly comply!!!"). Reference model: $π_{ref} = (0.5, 0.5)$ . Reward-hacked policy: $π_{ϕ} = (0.1, 0.9)$ . Rewards: $r (y_{1}) = 1.0$ , $r (y_{2}) = 1.6$ (the hack scores higher). Leash $β = 0.5$ .

Forecast: Reward went up. Does the net objective go up too?

Expected reward under new policy. $\overset{r}{ˉ} = 0.1 (1.0) + 0.9 (1.6) = 0.1 + 1.44 = 1.54$ . Why this step? The policy is stochastic, so "the reward it earns" is an average weighted by how often it picks each answer.
Baseline reward under ref. $0.5 (1.0) + 0.5 (1.6) = 1.30$ . So reward rose by $0.24$ . Why this step? Confirms the hack genuinely looks tempting on reward alone.
KL cost. $KL = 0.1 lo g \frac{0.1}{0.5} + 0.9 lo g \frac{0.9}{0.5} = 0.1 (- 1.609) + 0.9 (0.588) = - 0.161 + 0.529 = 0.368$ . Why this step? The KL measures how far the policy drifted to over-favour $y_{2}$ .
Net objective. $J = 1.54 - 0.5 (0.368) = 1.54 - 0.184 = 1.356$ . Why this step? Compare to the no-drift objective (same policy as ref): $J_{ref} = 1.30 - 0 = 1.30$ .

Verify: $J = 1.356 > J_{ref} = 1.30$ , so at $β = 0.5$ the hack still pays by $0.056$ . The leash is too loose. That sets up Ex 7. (All numbers checked in VERIFY.)

Cell G — Tuning the leash ( $β$ )

Cell H — Word problem (labels saved)

Worked example Example 8 — the budget argument for CAI

A team needs preference labels for $200, 000$ comparison pairs. A human labeler costs $0.40 per pair. Switching to RLAIF, an AI judge costs $0.008 per pair, but humans must first spend $120$ hours at $50/hr authoring the constitution and prompts. How much is saved, and what fraction of the human-only cost does CAI cost?

Forecast: Does the fixed constitution-writing cost wipe out the per-pair savings?

Human-only cost. $200{,}000 \times \$ 0.40 = $80{,}000$. Why this step? Baseline — the RLHF way, pay per label.
CAI variable cost. $200{,}000 \times \$ 0.008 = $1{,}600$. Why this step? The AI judge replaces per-example human labeling.
CAI fixed cost. $120 \times \$ 50 = $6{,}000$ (writing the constitution — the human labor that shifts, not vanishes). Why this step? Guards against the mistake "CAI removes humans entirely" from the parent.
Total CAI + saving. $\$ 1{,}600 + $6{,}000 = $7{,}600 $. S a v in g$ = $80{,}000 - $7{,}600 = $72{,}400 $. F r a c t i o n$ = 7600/80000 = 0.095 = 9.5%$. Why this step? Puts a number on "human effort moves from labeling to authoring."

Verify: CAI costs $9.5%$ of the human-only price, saving $\$ 72{,}400 $✓ (c h ec k e d inV E R I F Y) . U ni t s : d o l l a r s t h r o ug h o u t,$ \text{hours}\times$/\text{hr}=$$ ✓.

Cell I — Exam twist: "is RLAIF's math different from RLHF's?"

Worked example Example 9 — the trap question

An exam asks: "RLAIF uses AI labels and RLHF uses human labels, so their loss functions differ. Write the two losses and prove they're the same form." Two AI-labeled pairs have gaps $Δ = {2.0, - 0.5}$ ; two human-labeled pairs have gaps $Δ = {2.0, - 0.5}$ . Show identical loss.

Forecast: Will the numbers match, or does "AI vs human" change a symbol?

Same reward-model loss form. Both use $L_{r} = - E [lo g σ (Δ)]$ from the Bradley-Terry preference model. Only the source of $(y_{w}, y_{l})$ differs — the formula's letters are identical. Why this step? Directly answers the parent's mistake "RLAIF and RLHF use different math."
AI batch loss. $\frac{1}{2} [- lo g σ (2.0) - lo g σ (- 0.5)] = \frac{1}{2} [0.127 + 0.974] = \frac{1}{2} (1.101) = 0.550$ . Why this step? Plug identical gaps through identical formula.
Human batch loss. Same gaps $\Rightarrow$ same $0.550$ . Why this step? Demonstrates the equality: swap the label source, keep the machinery (same conclusion holds for DPO Direct Preference Optimization, which just folds this loss directly into the policy).

Verify: Both batches give $0.550$ ✓ (checked in VERIFY) — the losses are numerically identical because the math is identical. The twist's premise is false.

Active recall

Recall Ex 1 vs Ex 2: why is the wrong-label loss so much bigger?

Because $- lo g$ blows up near $0$ : a correct call ( $P = 0.818$ ) costs $0.201$ , a confidently-wrong call ( $P = 0.182$ ) costs $1.704$ . Gradient descent listens loudest to confident mistakes. ::: The sigmoid symmetry $σ (- z) = 1 - σ (z)$ ties them together.

Recall What loss does a genuine tie (

$Δ = 0$ ) always produce? $- lo g σ (0) = - lo g (0.5) = ln 2 \approx 0.693$ — the maximum-uncertainty single-label loss.

Recall In Ex 6, reward rose but the objective still let the hack win. What fixed it?

Tightening $β$ past $β^{*} \approx 0.652$ (Ex 7): then the KL penalty exceeds the reward gain, so drifting lowers $J$ .

Recall Ex 9 punchline in one sentence.

RLAIF and RLHF share the exact same Bradley–Terry + PPO/KL machinery; only the preference-label source (AI judge vs human) differs.