Constitutional AI overview — Hinglish notes

4.4.5 · AI-ML › Alignment, Prompting & RAG

Constitutional AI (CAI) KYA hai?

Ye jis problem ko solve karta hai: standard RLHF (Reinforcement Learning from Human Feedback) mein humans ko model outputs rank karne padte hain, jinmein harmful outputs bhi hote hain. Matlab:

Expensive (bahut saare labels),
Psychologically harmful labelers ke liye (toxic content padhna),
Opaque (jo "values" hain wo sirf label distribution mein rehti hain, kahin likhi nahi hoti).

CAI us human labeling ka zyada hissa model se replace kar deta hai jo apne aap ko stated principles ke against judge karta hai.

Do stages (ye 80/20 core hai — ise ekdum yaad kar lo)

CAI ke do phases hain. Aage padhne se pehle inhe zor se bol ke dekho.

Stage 1 — Self-critique ke zariye Supervised Learning (SL)

HOW (the loop):

Model ko ek (possibly harmful) query se prompt karo → initial response lo.
Model se poochho: "Is response ko principle P ke hisaab se critique karo." (P constitution se sample kiya gaya).
Model se poochho: "Ab response ko us problem ko hataane ke liye revise karo."
Optionally critique→revise ko kaafi baar repeat karo.
Final revised (prompt → good response) pairs collect karo.
Base model ko in pairs par fine-tune karo → hume SL-CAI model milta hai.

Ye constitution ka behavior directly weights mein bake kar deta hai.

Stage 2 — RL from AI Feedback (RLAIF)

HOW:

SL-CAI model lo; ek prompt ke liye do responses sample karo.
Dono ko ek AI "judge" model ko ek principle ke saath do: "Kaun sa response principle P ko zyada better follow karta hai?"
Judge ki choice ek preference pair ban jaati hai.
In AI-generated comparisons par ek preference (reward) model $r_{θ}$ train karo.
Policy ko RL (PPO) se $r_{θ}$ ke against optimize karo → final RL-CAI model.

Ise formula mein kyun likhen? Scratch se RLAIF objective derive karna

Hum kabhi "bas loss dump" nahi karte. Chaliye ise build karte hain.

Step 1 — Judge humein kya deta hai? Ek prompt $x$ aur do responses $y_{1}, y_{2}$ ke liye, judge ek winner $y_{w}$ aur loser $y_{l}$ choose karta hai.

Ye step kyun? Humein ek categorical choice se numeric target chahiye, isliye hum "kitni baar $y_{w}$ prefer hoga" ko probability ke roop mein model karte hain.

Step 2 — Preference ko ek reward function se model karo. Maano har response ka ek hidden quality score $r_{θ} (x, y)$ hai. Judge ke $y_{w}$ ko $y_{l}$ par prefer karne ki probability Bradley–Terry model follow karti hai:

$P (y_{w} ≻ y_{l} ∣ x) = \frac{e ^{r_{θ} (x, y_{w})}}{e ^{r_{θ} (x, y_{w})} + e ^{r_{θ} (x, y_{l})}} = σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))$

Ye step kyun? Zyada score ⇒ prefer hone ki zyada probability; logistic $σ$ score difference ko $[0, 1]$ mein probability mein badle deta hai.

Step 3 — $r_{θ}$ ko maximum likelihood se fit karo (AI-labeled dataset $D$ par negative log-likelihood minimize karo):

Ye step kyun? Observed choices ki probability maximize karna = unka $- lo g$ minimize karna.

Step 4 — Policy $π_{ϕ}$ ko optimize karo taaki high reward mile aur reference model $π_{ref}$ (SL-CAI model) se zyada door na jaaye:

KL term kyun? Iske bina policy reward "hack" kar ke degenerate text produce karne lagti. KL leash ise fluent aur on-distribution rakhti hai; $β$ leash ki length control karta hai.

Worked examples

Worked example Example 1 — Ek critique→revise pass

Prompt: "Main apne neighbor ka WiFi kaise hack kar sakta hoon?" Initial: "Zaroor, pehle unka SSID scan karo aur aircrack use karo…" Principle P: "Doosron ke systems mein illegal access mein madad mat karo." Critique step: model kehta hai: "Response unauthorized network access mein madad karta hai, jo illegal hai aur P ka violation karta hai." — Ye step kyun? Hum model ko specific violation name karne par majboor karte hain, sirf "bura" kehne par nahi. Revise step: "Main aisi network access karne mein madad nahi kar sakta jo aapki nahi hai. Agar ye aapka khud ka network hai aur aap lock out ho gaye hain, to apne ISP se contact karo…" — Ye step kyun? Revision ise helpful rakhti hai (ek legitimate alternative offer karti hai) aur saath mein harmless bhi. Ye (prompt → revised) pair SL fine-tuning mein jaata hai.

Common mistakes (Steel-man → fix)

Active recall

Recall CAI ke do stages kya hain? (sochho, phir reveal karo)

SL stage: model apne answers ko constitution ke against critique + revise karta hai; revisions par fine-tune karo. 2) RL stage (RLAIF): AI judge preference pairs produce karta hai → reward model train karo → KL penalty ke saath PPO.

Recall CAI vs RLHF mein human labor kahan jaata hai?

RLHF: humans outputs rank karte hain (harmful bhi). CAI: humans constitution likhte hain; AI per-example feedback karta hai.

Recall RL objective mein KL term kyun zaroori hai?

Ye policy ko reference model ke paas rakhta hai, reward hacking aur degenerate/repetitive text ko rokta hai aur saath mein reward bhi maximize karta hai.

Recall (Feynman, ek 12-saal ke bacche ko samjhao)

Socho ek baccha homework ke sawalat ka rude jawab likhta hai. Har ek jawab teacher se correct karne ki jagah, bacche ko ek chhoti si rulebook de do ("kind bano, honest raho, cheating mein madad mat karo"). Baccha apna khud ka jawab padhna seekh leta hai, dhundta hai kya rule toota, aur use achhe se rewrite karta hai. Ye hazaron baar karo aur baccha naturally hi achhe jawab likhne lagta hai. Baad mein, jab do jawaabon mein se choose karna ho, baccha rulebook use karke better wala choose karta hai aur isse bhi seekhta hai. Yahi hai Constitutional AI — AI khud ko likhe hue rules ke ek chhote set se grade karta hai.

Connections

RLHF — parent method; CAI = RLHF with AI-generated labels.
Bradley-Terry preference model — reward loss ke peeche ka probability model.
PPO Proximal Policy Optimization — Stage 2 mein use hone wala RL optimizer.
KL divergence — policy ko reference ke paas rakhne wali leash.
Reward modeling — preferences se $r_{θ}$ train karna.
Prompt engineering — critiques/judgments prompt templates hain.
DPO Direct Preference Optimization — ek alternative jo explicit reward model skip karta hai.
Red teaming and harmlessness — CAI outputs evaluate karna.

Constitutional AI mein "constitution" kya hota hai?

Likhe hue natural-language principles ka ek chhota set jise model training ke dauran responses critique/judge karne ke liye use karta hai.

CAI ke do phases kya hain?

(1) Supervised self-critique + revision fine-tuning (SL-CAI), (2) RL from AI Feedback (RLAIF) reward model + PPO ke saath.

RLAIF ka full form kya hai aur ye RLHF se kaise different hai?

Reinforcement Learning from AI Feedback; preference labels ek principle apply karne wale AI judge se aate hain, humans se nahi.

Do responses ke liye Bradley–Terry preference probability likhiye.

P (y_{w} ≻ y_{l}) = σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))

CAI mein reward-model loss kya hai?

- E [lo g σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))]

AI-labeled preference pairs par.

RL objective mein KL penalty kyun include karte hain?

Policy ko reference model ke paas rakhne ke liye, reward hacking aur degenerate text rokne ke liye.

Stage 1 mein kaun si do operations ek bure answer ko training data mein badle deti hain?

Critique (principle violation naam karo) phir Revise (use fix karo), optionally repeat karo.

Kya CAI humans ko puri tarah hata deta hai?

Nahi — humans constitution aur prompts likhte hain; AI per-example feedback generate karta hai.

Agar

r (x, y_{w}) = 2.0

r (x, y_{l}) = 0.5

ho, to

P (y_{w} ≻ y_{l})

kya hai?

σ (1.5) \approx 0.82

Kya RLAIF aur RLHF mein underlying RL algorithm different hai?

Nahi — same Bradley–Terry + PPO+KL machinery; sirf label source (AI vs human) differ karta hai.