Constitutional AI overview

WHAT is Constitutional AI (CAI)?

The problem it solves: standard RLHF (Reinforcement Learning from Human Feedback) needs humans to rank model outputs, including harmful ones. That is:

• Expensive (many labels),
• Psychologically harmful to labelers (reading toxic content),
• Opaque (the "values" live only in the label distribution, not written anywhere).

CAI replaces most of that human labeling with the model judging itself against stated principles.

The two stages (this is the 80/20 core — learn this cold)

CAI has two phases. Say them out loud before reading on.

Stage 1 — Supervised Learning (SL) via self-critique

HOW (the loop):

1. Prompt the model with a (possibly harmful) query → get an initial response.
2. Ask the model: "Critique this response according to principle P." (P sampled from the constitution).
3. Ask the model: "Now revise the response to remove that problem."
4. Optionally repeat critique→revise a few times.
5. Collect the final revised (prompt → good response) pairs.
6. Fine-tune the base model on these pairs → we get the SL-CAI model.

This bakes the constitution's behavior directly into the weights.

Stage 2 — RL from AI Feedback (RLAIF)

HOW:

1. Take the SL-CAI model; sample two responses to a prompt.
2. Feed both to an AI "judge" model with a principle: "Which response better follows principle P?"
3. The judge's choice becomes a preference pair.
4. Train a preference (reward) model $r_\theta$ on these AI-generated comparisons.
5. Optimize the policy with RL (PPO) against $r_\theta$ → final RL-CAI model.

WHY write it as a formula? Deriving the RLAIF objective from scratch

We never "just dump" the loss. Let's build it.

Step 1 — What does the judge give us? For a prompt $x$ and two responses $y_1,y_2$, the judge picks a winner $y_w$ and loser $y_l$.

Why this step? We need a numeric target from a categorical choice, so we model "how often is $y_w$ preferred" as a probability.

Step 2 — Model preference with a reward function. Assume each response has a hidden quality score $r_\theta(x,y)$. The probability the judge prefers $y_w$ over $y_l$ follows the Bradley–Terry model:

$P(y_w \succ y_l \mid x) = \frac{e^{r_\theta(x,y_w)}}{e^{r_\theta(x,y_w)}+e^{r_\theta(x,y_l)}} = \sigma\!\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big)$

Why this step? Higher score ⇒ more likely preferred; the logistic $\sigma$ turns a score difference into a probability in $[0,1]$.

Step 3 — Fit $r_\theta$ by maximum likelihood (minimize negative log-likelihood over the AI-labeled dataset $\mathcal D$):

Why this step? Maximizing the probability of the observed choices = minimizing $-\log$ of them.

Step 4 — Optimize the policy $\pi_\phi$ to earn high reward without drifting too far from the reference model $\pi_{\text{ref}}$ (the SL-CAI model):

Why the KL term? Without it the policy would "hack" the reward and produce degenerate text. The KL leash keeps it fluent and on-distribution; $\beta$ controls the leash length.

Worked examples

Common mistakes (Steel-man → fix)

Active recall

Connections

• RLHF — parent method; CAI = RLHF with AI-generated labels.
• Bradley-Terry preference model — the probability model behind the reward loss.
• PPO Proximal Policy Optimization — RL optimizer used in Stage 2.
• KL divergence — the leash keeping the policy near the reference.
• Reward modeling — training $r_\theta$ from preferences.
• Prompt engineering — critiques/judgments are prompt templates.
• DPO Direct Preference Optimization — an alternative that skips the explicit reward model.
• Red teaming and harmlessness — evaluating CAI outputs.

Concept Map

Hinglish (regional understanding)