Question bank — Constitutional AI overview

False — humans still write the constitution and design the prompts; only the per-example feedback labels are produced by an AI judge, so human effort shifts from labeling to authoring.

False — both use the same machinery (a Bradley–Terry reward model plus PPO with a KL penalty); only the source of the preference labels differs, AI judge vs human.

False — the constitution is only used during training (inside critiques and judge prompts); its influence ends up implicitly encoded in the fine-tuned weights, and no rulebook is fed in at deployment.

False — it trains on the final revised (prompt → good response) pairs; training on the un-revised answers would reinforce the very behaviour CAI is trying to remove.

False — revisions show diminishing returns and too many passes can cause over-refusal (evasive answers); the number of passes is a tuned hyperparameter, not "more is better".

False — without it the policy policy "hacks" the reward with degenerate or repetitive text; the KL leash keeps outputs fluent and on-distribution, so it is essential, not cosmetic.

False — the revision step is designed to keep answers helpful (offer a legitimate alternative) while removing harm; the target is helpful and harmless, not silence.

False — the reward model outputs a scalar quality score $r_{\theta }(x,y)$; a probability only appears when the Bradley–Terry logistic $\sigma$ turns a score difference into $P(y_w\succ y_l)$.

False — biases enter through the humans who wrote the constitution and through the base model's pretraining data; automating the labels does not sterilize those upstream sources.

We fine-tune on the revised response, not the critique text; the critique is a scaffold that produces a better answer, and the answer is the training target.

The judge only picks a winner (a categorical choice between two responses); the numeric scores $r_{\theta }$ are learned by the reward model to explain those choices, not supplied by the judge.

We minimize $-\log \sigma (\cdot )$ (equivalently maximize $\log \sigma$); minimizing the negative log-likelihood is the same as maximizing the probability of the observed preferences.

The KL term is subtracted ($-\beta \mathrm{KL}$); adding it would reward drifting away from the reference, the exact opposite of what a leash should do.

The judge produces discrete preference pairs, but RL still needs a dense scalar signal for every sampled response, so we still fit a reward model $r_{\theta }$ on those pairs.

The constitution is a small set of principles, and it is sampled into training prompts, not consulted at generation time; only its distilled behaviour survives in the weights.

A hard 0/1 label ignores how much better one answer is; the logistic maps a continuous score difference to a probability, so confident and marginal preferences produce appropriately different gradients.

Writing principles once is cheap, transparent, and spares labelers from reading toxic content, whereas ranking every harmful pair in RLHF is expensive, opaque, and psychologically harmful.

Naming the exact principle broken gives the revision step a concrete target to fix, producing a cleaner corrected answer than a vague "make it better" instruction.

The SL-CAI model already speaks in the constitution's style, so anchoring the KL to it means the RL stage refines behaviour without dragging the policy back toward the un-aligned base distribution.

DPO optimizes the same Bradley–Terry preference objective directly from pairs, folding the reward model and KL penalty into one closed-form loss — it consumes the same AI-judged pairs, just skips the explicit reward-model + PPO loop.

Self-critique assumes the model already latently "knows" the principle it violated; a weak model cannot spot its own violation, so there is nothing to revise and Stage 1 collapses.

The reward model faithfully fits the bad preference, so RL will steer the policy toward the worse answer — judge quality is a hard ceiling on the whole pipeline.

The leash vanishes and the policy is free to reward-hack, producing degenerate or repetitive text that scores high on $r_{\theta }$ but reads as broken language.

The KL penalty dominates and the policy barely moves from ${\pi }_{\text{ref}}$, so RL contributes almost nothing and you get roughly the SL-CAI model back.

With near-equal scores, $\sigma (r_w-r_l)\approx 0.5$ and the loss $-\log 0.5$ is large-but-flat, giving a weak, low-magnitude gradient — the model sensibly refuses to commit to a spurious preference.

The critique finds nothing to fix, so the revision is essentially the original answer; the (prompt → answer) pair still enters SL training and reinforces good behaviour on harmless inputs, preventing over-refusal.

The critique/judge signal becomes inconsistent (the "right" answer depends on which principle is sampled), injecting noise into both SL revisions and RLAIF labels — resolving principle conflicts is a human authoring problem, not something the model fixes.

Answers drift toward blanket refusal (over-refusal), sacrificing helpfulness — the model over-corrects and treats even legitimate requests as violations.