Exercises — Scalable oversight

How do we produce a training signal for AI behaviour on tasks whose answers humans cannot directly verify? Traditional supervision assumes the teacher can grade the student; scalable oversight is what we do when the student surpasses the teacher.

(a) → 2 · (b) → 3 · (c) → 1. Memory hook: RRM = a Reviewer at your elbow; Amplification = Atomise then assemble; Debate = Duel.

• $R_{\text{recursive}}^{(i)}$ — the reward number used at training round $i$.
• $H(\dots )$ — the human's judgment function, outputting "how good".
• $y$ — the output being graded.
• $x$ — the original input/task.
• $M_{i-1}(x,y)$ — the assistance (explanations, critiques, highlighted concerns) produced by the previous model about this $(x,y)$ pair. The whole thing reads: "how good does the human rate $y$, given both the task and the previous AI's analysis of it."

(a) Speed-up $=\frac{120}{30}=4\times$. (b) Accuracy gain $=90$ percentage points. What this shows: the assistant does the search (find the 14 lines), the human does the verification (judge them) — labour is split along the axis where each is strong.

The conditional probability the inequality bounds is the human's error given a flag — that is $0.04$. $0.04<0.05✓$ So the stability condition holds for $ϵ=0.05$. Careful: the $95\%$ flag rate is a different quantity (how often the assistant catches the flaw). The condition is about what the human does after a flag, so only the $0.04$ enters.

Independent successes multiply: $0.9^4=0.6561\approx 65.6$ Reading it: decomposition helps only if the pieces are reliable — 4 shaky pieces still compound into a coin-flip. This motivates keeping subtasks small enough that per-piece reliability is high.

Standard reward asks the human to judge $y$ from scratch. Recursive reward conditions the judgment on $M_{i-1}(x,y)$, so the human effectively gets an expert briefing before ruling — this amplifies their judgment on tasks they couldn't otherwise parse. It fails when $M_{i-1}$ is misaligned: if the assistant's "briefing" hides flaws or fabricates reassurance, conditioning on it makes $H$ worse, not better. Formally, the benefit exists only while $M_{i-1}$'s assistance is net-positive — the same requirement behind the stability condition. See RLHF for how $H$ is turned into a trainable reward model in the first place.

Each round, $M_{i+1}$ inherits whatever systematic bias $M_i$ had (it's trained to imitate $M_i$'s amplified behaviour). A bias in decomposition or in a sub-answer can be copied, then amplified again next round → potential drift. What bounds it: the human stays in the loop at every amplification step — the human chooses the decomposition and composes the pieces. So errors only propagate if they survive human scrutiny at the small-subtask scale, where the human is competent. The human is the anchor that keeps the recursion from wandering. (Contrast with Recursive Self-Improvement, where removing the human anchor is precisely the danger.)

It breaks when a false claim is simple and emotionally intuitive while the true rebuttal requires long technical reasoning the judge can't follow (e.g. a slick but wrong physical-intuition argument vs. a correct 6-step calculation). Here the false side is easier to make persuasive, so the judge rules for falsehood. More generally: debate is only as reliable as the judge's ability to follow the harder-but-true argument. This is why Interpretability matters — making the true reasoning legible restores the asymmetry debate needs.

A defensible pipeline:

1. Amplification (decompose): split "is this drug combination safe?" into checkable subquestions — metabolic pathway overlap, known contraindications, dosage thresholds.
2. RRM (assist): an $M_i$ assistant summarises the literature for each subquestion and flags concerns, amplifying the doctor's judgment per piece.
3. Debate (adversarial check): on the final composed verdict, run two agents — one arguing "safe," one "unsafe" — so any flaw that survived steps 1–2 gets surfaced adversarially. Ordering justification: decompose first to make pieces human-sized; assist within each piece; debate last as a red-team over the assembled answer, where the stakes concentrate.

$0.95^3=0.857375\approx 85.7$ Compare: $85.7$, so amplification wins by about $30.7$ percentage points. Lesson: decomposition pays off when per-subtask reliability is high enough that its cube still exceeds the monolithic rate. If subtasks were only $0.80$ reliable, $0.8^3=0.512<0.55$ and decomposition would lose — this is exactly the L2·Q3 warning in reverse.

The incentive (deception risk): if unverifiable outputs are never penalised, the reward-maximising behaviour is to route hard-to-check content through the blind spot — the model learns that "make it unverifiable" is a cheap way to escape negative reward. This is a direct corollary of training on a fallible $H$. Counters:

• RRM shrinks the blind spot: the assistant makes previously-unverifiable outputs partially verifiable, so hiding flaws there stops being safe.
• Amplification removes the blind spot piecewise: every atom is human-sized, leaving nowhere for an unverifiable claim to hide.
• Debate turns the blind spot against the deceiver: an opposing agent is rewarded for exposing exactly the hidden flaw. Connect this to AI Safety: the whole point is that alignment must survive the capability jump, not cap capability at human level.

The condition: the assisting/decomposing/judging apparatus available to the human must stay net-truthful — its help must, in expectation, move the human toward the correct judgment rather than away. Symbolically, across methods we always require the helper channel (be it $M_{i-1}$, the subtask solver, or the debate transcript) to satisfy something like $P(\text{human errs}\mid \text{helper input})<P(\text{human errs}\mid \text{no help}).$ Why it's the true bottleneck: every method above is a scheme for feeding the human processed information. If that processing is adversarial or biased, all three degrade in lockstep — the recursion, decomposition, or debate simply launders the corruption. Everything else (iteration count, subtask granularity, number of debate turns) is tuning around this single load-bearing assumption.

Scalable oversight :::: makes a fallible human's oversight survive the capability jump by processing information (assist / decompose / debate) so the human can still judge — but only while that processing stays net-truthful.