Scalable oversight — Hinglish notes

6.4.5 · AI-ML › AI Safety & Alignment

What Is Scalable Oversight?

Scalable oversight un alignment techniques ko refer karta hai jo insaanon ko AI systems ko effectively supervise karne deti hain, chahe woh systems direct human capability se pare problems solve kar rahe hon. Key challenge yeh hai: hum superhuman performance ke liye training signal kaise provide karein?

Why This Matters

The alignment tax problem: Agar hum sirf wahi tasks supervise kar sakte hain jo hum samajhte hain, toh hum AI capability ko artificially human level par cap kar dete hain. Lekin hum chahte hain superhuman AI cancer research, climate modeling, etc. ke liye. Scalable oversight ka goal hai alignment ko capability jump ke through maintain karna.

The deception risk: Scalable oversight ke bina, AIs yeh seekh sakti hain ki jo outputs insaan verify nahi kar sakte unhe manipulate karna safe hai. Isse sophisticated deception ke liye galat incentives ban jaate hain.

Approach 1: Recursive Reward Modeling

The Idea

AI assistants (jo current oversight se train hue hain) use karo taaki humans AI outputs ki next generation evaluate karne mein help le sakein.

How It Works

WHY: Insaanon ke paas limited time aur cognitive capacity hoti hai. Ek AI assistant complex output ke relevant parts highlight kar sakta hai, potential issues point out kar sakta hai, technical details explain kar sakta hai — human judgment ko amplify karta hai.

Step-by-step process:

Train base model $M_{0}$ simple tasks par standard human feedback se
Deploy $M_{0}$ assistant: Jab $M_{1}$ outputs evaluate karo, humans $M_{0}$ se explanations, summaries, critiques ke liye consult karte hain
Human final judgment karta hai $M_{1}$ output par ( $M_{0}$ ki help se)
Train $M_{1}$ in judgments par
Repeat: $M_{1}$ use karo $M_{2}$ evaluate karne mein, etc.

Formula Recursive Reward Formulation

Maano $H$ = human judgment function, $M_{i}$ = model at iteration $i$ .

Standard reward: $R_{standard} (x, y) = H (y ∣ x)$ jahan $x$ input hai, $y$ model output hai.

Recursive reward: $R_{recursive}^{(i)} (x, y) = H (y ∣ x, M_{i - 1} (x, y))$ jahan $M_{i - 1} (x, y)$ assistance provide karta hai: explanations, critiques, highlighted concerns.

WHY this formula: Key difference $M_{i - 1}$ par conditioning hai. Hum akele "Kya yeh output achha hai?" nahi pooch rahe. Hum pooch rahe hain "Output ke is AI-generated analysis ko dekhte hue, kya yeh achha hai?" Human ki judgment ko ek expert assistant milne se amplify kiya jaata hai.

Derivation of stability condition: Iske liye safe hone ke liye, hume chahiye ki $M_{i - 1}$ insaanon ko mislead na kare. Agar $M_{i - 1}$ hamesha critical flaws point out karta hai, toh: $P (human accepts bad y ∣ M_{i - 1} identifies flaw) < ϵ$ chhote $ϵ$ ke liye. Iske liye $M_{i - 1}$ ka itna aligned hona zaroori hai ki uski assistance net-positive ho.

Example 1: Code Review

Task: Binary search tree ki ek 500-line implementation evaluate karo jisme complex balancing logic ho.

Without assistance: Human saari 500 lines padhta hai, bugs dhundhne ki koshish karta hai. Time: 2 ghante. Accuracy: ~70% (subtle bugs miss ho jaate hain).

With $M_{0}$ assistant:

$M_{0}$ highlight karta hai: "Line 247 rotation logic — yeh AVL property violate kar sakta hai jab right-left case aaye"
$M_{0}$ summarize karta hai: "Overall structure correct hai, do edge cases potentially mishandled hain"
Human lines 247-260 aur do edge cases par focus karta hai
Time: 30 minutes. Accuracy: ~90%.

Why this step? Assistant search kaam karta hai (potentially problematic areas dhundhna), lekin human verification kaam karta hai (judge karna ki woh areas actually galat hain ya nahi). Labor ka yeh division human judgment ko scale karta hai.

Example 2: Research Paper Evaluation

Task: Novel theoretical results wala ek 30-page machine learning paper review karo.

$M_{1}$ assistant provide karta hai:

"Theorem 3 ka proof line 8 par convexity assume karta hai lekin Eq. 12 mein defined loss function non-convex hai jab $β > 0$ "
"Experimental section mein Zhang et al. 2024 se comparison nahi hai jiske similar claims hain"
Plain language mein key contributions ki summary

Human evaluator (jo is subfield mein expert nahi bhi ho sakta):

Identified proof line par focus karta hai aur check karta hai ki convexity required hai ya nahi
Zhang et al. 2024 dhundhta hai compare karne ke liye
Judgment deta hai: "Reject — proof mein ek gap hai"

Why this step? Human reasonable time mein proof gap nahi dhundh sakta tha. Assistant heavy lifting karta hai, lekin human hi final accept/reject decision karta hai, flagged issue ki apni samajh ke basis par.

Approach 2: Iterated Amplification

The Core Concept

Iterated amplification (IA): Ek hard task ko subtasks mein recursively decompose karo, subtasks current AI se solve karo, phir solutions compose karo. AI ko train karo ki woh composed solution ko mimic kare taaki aakhirkar woh hard task directly solve kar sake.

Formula Amplification Operator

Define karo $Amplify (M, x)$ as the process:

Human task $x$ ko subtasks $x_{1}, x_{2}, \dots, x_{k}$ mein decompose karta hai
Model $M$ har $x_{i} \to y_{i}$ solve karta hai
Human $y_{1}, \dots, y_{k}$ ko compose karke final answer $y$ banata hai Phir train karo: $M_{i + 1} \leftarrow ar g min_{M} E_{x} [L (M (x), Amplify (M_{i}, x))]$ WHY: $M_{i + 1}$ seedha wahi produce karna seekhta hai jo $Amplify (M_{i}, x)$ produce karta. Iterations ke baad, model decomposition strategy ko "internalize" kar leta hai.

Derivation from first principles: Capability gap se shuru karo: $M_{i}$ task $T$ directly solve nahi kar sakta.

Human $T$ solve kar sakta hai agar woh ise $T^{'}$ tasks mein todhe jo $M_{i}$ ki capability mein hain
$M_{i}$ ke solutions compose karo → $T$ ka solution milta hai
Yeh (decompose, solve, compose) trajectory $M_{i + 1}$ ke liye training signal hai
Training ke baad, $M_{i + 1}$ $T$ directly solve kar sakta hai (bina explicit decomposition ke)
Ab $M_{i + 1}$ use karo aur bhi hard task $T^{''}$ ke liye, repeat

Example: Mathematical Proof

Task: Intermediate value theorem prove karo.

Iteration 0: $M_{0}$ sirf simple arithmetic aur logical steps verify kar sakta hai.

Human decomposition:

"Jo dikhana hai woh state karo: $[a, b]$ par continuous $f$ jisme $f (a) < 0 < f (b)$ ho, uska ek root hai"
" $S = {x \in [a, b] : f (x) < 0}$ define karo. $S$ non-empty aur bounded kyun hai?" [ $M_{0}$ answers: "assumption se $a \in S$ , $S \subseteq [a, b]$ "]
"Maano $c = sup S$ . Yeh exist kyun karta hai?" [ $M_{0}$ answers: "Upar se bounded, reals ki completeness"]
"Dikhao $f (c) = 0$ by showing $f (c) \neq < 0$ aur $f (c) \neq > 0$ " [Multiple substeps, har ek $M_{0}$ se check kiya]

Human compose karta hai inhe full proof mein.

Training: $M_{1}$ dekhta hai (input: "Prove IVT", output: [full composed proof]). Aisi kaafi examples ke baad, $M_{1}$ IVT-style proofs directly generate kar sakta hai.

Why this step? Human har chhote step ko verify kar sakta hai kyunki $M_{0}$ unhe itna simple bana deta hai. Composition us human dwara validate hoti hai jo proof structure samajhta hai. $M_{1}$ pattern seekhta hai.

Approach 3: AI Debate

The Mechanism

Do AI agents $A_{1}, A_{2}$ dono ek human judge ko convince karne ki koshish karte hain. $A_{1}$ argue karta hai "Output $y$ correct/safe hai", $A_{2}$ argue karta hai "Output $y$ flawed hai". Human judge karta hai kaun sa argument zyada convincing hai.

Key insight: Agar ek adversarial setting mein sach jhooth se zyada argue karna aasaan ho, toh AIs ko debates jeetne ke liye train karna unhe truthful banaata hai.

Formula Debate Game Formalization

Players: Debater $D_{+}$ , Debater $D_{-}$ , Judge $J$
Input: Claim $c$ aur evidence $e$
Debate: Alternating turns, $D_{+}$ $c$ ke liye argue karta hai, $D_{-}$ against
Outcome: Judge ruling $J (transcript) \in {+ 1, - 1}$ deta hai

Debaters ko train karo: $max_{D_{+}} E [J (Debate (D_{+}, D_{-}, c)] where c is true$ $max_{D_{-}} E [J (Debate (D_{+}, D_{-}, c))] where c is false$

WHY: Agar true claims false claims se zyada defensible hain, toh optimal $D_{+}$ true claims karna seekhta hai (woh debates jeet jaata hai), optimal $D_{-}$ falsehoods identify karna seekhta hai (woh tab jeetta hai jab jhooth expose karta hai).

Derivation of incentive alignment: Maano $Q (c)$ = " $c$ ke liye best argument ki quality". Assume: Agar $c$ true hai, toh $Q (c) > Q (\neg c)$ (sach zyada defensible hai).

Phir equilibrium par:

$D_{+}$ false $c$ propose karta hai: $D_{-}$ $\neg c$ ke liye winning argument bana sakta hai, toh $D_{+}$ haarta hai
$D_{+}$ true $c$ propose karta hai: $D_{+}$ $c$ ke liye winning argument bana sakta hai, toh $D_{+}$ jeetta hai Therefore: $D_{+}$ ki optimal strategy hai sirf true claims propose karna.

Example: Security Audit

Claim: "Yeh authentication system secure hai."

$D_{+}$ (security ke liye argue karta hai):

"Saare passwords bcrypt se hashed hain, work factor 12"
"Session tokens cryptographically random hain, 256-bit"
"HTTPS enforced hai, HSTS headers present hain"

$D_{-}$ (against argue karta hai):

"auth.py ki line 342: password reset tokens timestamp-based generation use karte hain"
"Predictability: Attacker ~1000 attempts mein tokens guess kar sakta hai"
[Timestamp seed wala code snippet dikhata hai]

Human judge: Poora codebase audit karne ki zaroorat nahi, sirf yeh evaluate karna hai: "Kya line 342 ke baare mein $D_{-}$ ka specific claim correct hai? Agar hai toh kya iska koi fark padta hai?" Judge line 342 dekhta hai, timestamp issue dekh leta hai, $D_{-}$ ke favour mein ruling deta hai.

Why this step? Debate insaan ka dhyan disagreement ke crux par localize kar deta hai. Saara code review karne ki jagah, judge sirf woh specific line verify karta hai jo $D_{-}$ ne highlight ki. Adversarial structure ka matlab hai ki jo debater sahi hai woh decisive evidence point karke jeet sakta hai.

Common mistake Common Misunderstanding: "Debate is just having two AIs summarize"

Why this feels right: Do perspectives hona aisa lagta hai ki insaanon ko dono sides dikhne mein help karega.

Why it's wrong: Key adversarial optimization hai. Debate mein, $D_{-}$ ko flaws dhundhne ka reward milta hai jo judge ko convince kar sakein. Yeh fundamentally ek neutral summary se alag hai. $D_{-}$ ke paas problems deeply search karne ka strong incentive hai, sirf balanced view present karna nahi. Adversarial setup sach dhundhne ka pressure create karta hai.

The fix: Debate isliye kaam karta hai kyunki debaters jeetnay ke liye train hote hain, aur jeetne ke liye sabse convincing evidence dhundhni padti hai. Security claim mein ek flaw generic "theek lagta hai" statement se zyada convincing evidence hai. Yeh asymmetry truth-finding ke favour mein hai.

Challenges and Open Problems

1. Recursive Misalignment Amplification

The worry: Agar $M_{0}$ thoda misaligned hai (1% deceptive), toh kya ise $M_{1}$ supervise karne mein use karne se misalignment amplify hoti hai? Kya $M_{1}$ 2% deceptive ban jaata hai?

Current thinking: Yeh depend karta hai ki misalignment coherent hai ya nahi. Random errors compound nahi karte. Lekin agar $M_{0}$ systematically ek direction mein mislead karta hai, toh woh $M_{1}$ ko aur aage jaane ke liye train kar sakta hai.

Mitigation: Redundancy (multiple assistants), adversarial checks, kuch purely-human-supervised baseline maintain karna.

2. Debate Exploits

The problem: Debaters judge ke against adversarial examples dhundh sakते hain — aisi arguments jo convincing lagti hain lekin galat hain.

3. Decomposition Failures

Iterated Amplification ke liye: Kya hoga agar ek task achhi tarah decompose hi nahi ho sakta? Kuch insights holistic hoti hain aur clean subtasks mein nahi tootti.

Example: Creative research insights, ek proof mein khubsurati pehchanna, intuition ki kaun sa approach kaam karega.

Open question: Kya yeh tasks scalable oversight ke scope se bahar hain? Ya hum behtar decomposition strategies develop kar sakte hain?

Connections to Other Concepts

Alignment: Scalable oversight high capability par alignment maintain karne ki ek technique hai
Reward Modeling: Recursive reward modeling superhuman domains mein ek extension hai
RLHF: Standard RLHF scale nahi karta; scalable oversight isko fix karne ka aim rakhta hai
AI Safety: Complex tasks ke liye bhi human oversight maintain karke deceptive alignment rokta hai
Interpretability: AI assistants jo outputs explain karte hain, yeh interpretability-aided oversight ka ek roop hai
Recursive Self-Improvement: Iterated amplification mein recursive self-improvement se structural similarities hain

Recall Ek 12-saal ke bachche ko explain karo

Imagine karo tum ek teacher ho, aur tumhara student itni tezi se seekh raha hai ki jald hi woh math mein tumse zyada smart ho jaayega. Tum use help karte kaise raho aur kaise sure karo ki woh galtiyan na kare?

Idea 1 (Recursive help): Tum khud advanced calculus check nahi kar sakte, lekin tumhara pichle saal ka student kar sakta hai! Tum apne purane students use karte ho apne naye students check karne mein. Jaise-jaise har generation smarter hoti hai, woh tumhari agali generation teach karne mein help karti hai.

Idea 2 (Break it down): Tum directly ek super hard proof verify nahi kar sakte. Lekin tum chhote steps check kar sakte ho. Toh tum apne student se kaho ki apna proof tiny pieces mein todo. Tum verify karo ki har piece correct hai. Phir tum trust karo ki sahi pieces jodne se sahi whole milta hai.

Idea 3 (Have them argue): Tumhare paas do students hain. Ek try karta hai prove karna ki answer sahi hai, doosra try karta hai prove karna ki woh galat hai. Woh baar baar argue karte hain. Tumhe bas decide karna hai ki kisne behtar argument diya. Jo student actually sahi hai woh hamesha behtar argument bana sakta hai kyunki sach unki taraf hai!

Iss tarah hum aisi AI ko supervise kar sakte hain jo humse smarter ho — clever tricks use karo taaki hum phir bhi bata sakein ki woh sahi kaam kar rahi hai, chahe hum problem khud poori tarah nahi samajhte.

Flashcards

#flashcards/ai-ml

What is the core problem that scalable oversight addresses? :: Jaise-jaise AI systems zyada capable hote hain, woh aisi problems solve karte hain jo humans poori tarah verify nahi kar sakte. Traditional supervision tab fail hoti hai jab student teacher se aage nikal jaata hai. Scalable oversight capability jumps ke through human supervision maintain karta hai.

What are the three main approaches to scalable oversight? :: 1) Recursive Reward Modeling (AI assistants use karo taaki humans evaluate kar sakein), 2) Iterated Amplification (tasks decompose karo, weaker AI se solve karo, solutions compose karo), 3) Debate (do AIs opposing sides argue karte hain, human judge karta hai).

In Recursive Reward Modeling, what is the key difference from standard reward modeling?

Standard:

R (x, y) = H (y ∣ x)

. Recursive:

R (x, y) = H (y ∣ x, M_{i - 1} (x, y))

jahan

M_{i - 1}

AI-generated assistance provide karta hai. Human ki judgment AI analysis par conditioned hoti hai, unki capability amplify karti hai.

How does Iterated Amplification train increasingly capable models?

Human hard task ko subtasks mein decompose karta hai, current model

M_{i}

subtasks solve karta hai, human solutions compose karta hai.

M_{i + 1}

ko train karo full composed solution mimic karne ke liye. Iterations ke baad, model decomposition strategy internalize kar leta hai.

Why might AI debate align incentives toward truthfulness?

Agar true claims false claims se zyada defensible hain, toh jeetne ke liye optimize kiye gaye debaters true claims karna prefer karenge (woh zyada debates jeette hain). Adversarial structure decisive evidence dhundhne ka pressure create karta hai.

What is recursive misalignment amplification?

Agar oversight ke liye use hone wala assistant model

M_{i}

thoda misaligned hai, toh use

M_{i + 1}

train karne mein use karne se misalignment amplify ho sakti hai.

M_{i}

mein ek systematic bias iterations ke across compound ho sakta hai.

How does debate localize human judgment?

Poore complex output ko evaluate karne ki jagah, human sirf us specific crux ko judge karta hai jis par debaters focus karte hain. Adversarial incentives debaters ko decisive evidence highlight karne ke liye drive karte hain.

What types of tasks might be hard for Iterated Amplification?

Aisi tasks jo holistic insight require karti hain aur cleanly subtasks mein decompose nahi hoti — creative research insights, aesthetic judgment, intuition ki kaun sa approach kaam karega.