Worked examples — Scalable oversight

This page is a drill. The parent note told you the three big ideas — reward modelling, amplification, and debate. Here we grind through every kind of situation those ideas can throw at you, one worked example per situation, so that when you meet one on an exam or in a real system, you have already seen its twin.

We use only one number-idea repeatedly: a judgement. A judgement is a person (or a helper AI) looking at an output and saying "good" or "bad". We will write "good" as $+1$ and "bad" as $-1$. That is the whole notation. Everything else is built from it.

---

The scenario matrix

Every problem in scalable oversight is really one of a small number of cases. Think of it like the quadrants of a graph: once you have seen all of them, nothing is new. Here is the full grid.

#	Case class	What makes it special	Worked in
A	Human alone, easy task	Baseline: no AI help, human can verify	Ex 1
B	Human + aligned helper	Helper is honest → judgement improves	Ex 2
C	Human + misleading helper	Helper lies → the danger case	Ex 3
D	Decompose then compose (amplification)	Hard task split into checkable pieces	Ex 4
E	Debate, true claim	Honest debater should win	Ex 5
F	Debate, false claim	Liar should lose — the sign flips	Ex 6
G	Zero / degenerate input	Empty task, tie, no evidence — limiting case	Ex 7
H	Real-world word problem	Numbers from a real review pipeline	Ex 8
I	Exam-style twist	Recursion depth: does accuracy survive?	Ex 9

Cases A–C are the reward-modelling row (signs of the helper). D is amplification. E–F are the two signs of debate (true vs false claim — like quadrants where the answer flips). G is the degenerate/zero row every topic must cover. H and I are the application and the twist.

We will measure quality with one running quantity: accuracy $p$, the probability the final judgement is correct. $p$ lives between $0$ and $1$ ($0\%$ to $100\%$). Watch how $p$ moves.

---

Row 1 — Reward modelling: the three signs of a helper

Example 1 — Case A: human alone, easy task

Forecast: Guess — is the human better at catching the bug, or at clearing a clean file? Write down a number for each before reading on.

1. Model "catch the bug". The human misses any given bug with probability $0.02$, so they catch it with probability $1-0.02=0.98$. Why this step? "Miss" and "catch" are the only two outcomes, so their probabilities add to $1$; subtracting is how we turn one into the other.

2. Model "clear a clean file". With no bug there is nothing to miss, so the human is correct with probability $1$ (they just have to not invent a bug; we assume they don't here). Why this step? A degenerate sub-case (no bug) must still be counted — see the matrix; it is a mini version of case G.

3. Report accuracy. $p_{\text{catch}}=0.98$, $p_{\text{clean}}=1.00$.

Verify: Probabilities sit in $[0,1]$ ✓. Catch rate $0.98<1$ (harder than clearing a clean file, matching intuition) ✓.

---

Example 2 — Case B: human + an aligned helper

Forecast: Will the helper push accuracy above $0.80$? Guess a number.

1. Split into the two paths $M_0$ can take. Either $M_0$ flags right (prob $0.85$) or wrong (prob $0.15$). Why this step? The helper's own two outcomes are the "quadrants" here; we must cover both (the contract: never leave a case unshown).

2. Accuracy on the good path. $0.85\times 0.95=0.8075$. Why this step? We multiply because we treat "helper flags right" and "human then catches" as independent events: the human's re-reading skill on the flagged lines does not depend on how the helper found them, so the joint probability is the product. Both must happen, so we multiply rather than add.

3. Accuracy on the fallback path. $0.15\times 0.40=0.06$. Why this step? Same independence assumption on the branch where the helper missed: the unaided catch rate $0.40$ is unaffected by the helper's wrong flag.

4. Add the disjoint paths. $p=0.8075+0.06=0.8675$. Why this step? The two paths ("flags right" / "flags wrong") cannot both happen, so they are mutually exclusive, and probabilities of mutually exclusive events add.

Verify: $0.8675>0.40$ (helper strictly beat the unaided catch rate) and $0.8675<0.9575$ (below the perfect-flag path $0.85\times 0.95+0.15\times 0.95=0.95$'s ceiling) ✓. Conditioning the human's judgement on an honest helper $M_0$ raised the catch rate from $0.40$ to $0.8675$ — exactly the point of recursive reward modelling in the parent note.

---

Example 3 — Case C: human + a misleading helper (the danger case)

Forecast: Pass or fail? Guess $\epsilon$ vs $0.97$.

1. Read off the acceptance probability. $P(\text{human accepts bad }\hat{y}\mid M\text{ hides flaw})=0.97$. Why this step? This is exactly the quantity the stability condition constrains — and $\hat{y}$ was defined above as "the output being reviewed", so no new undefined symbol appears.

2. Compare to the threshold. We need this $<\epsilon =0.05$. But $0.97\nless 0.05$. Why this step? The condition is a hard inequality; one comparison settles it.

3. Conclude: the helper fails the stability condition catastrophically.

Verify: $0.97-0.05=0.92>0$, so the gap is enormous ✓.

---

Row 2 — Amplification: decompose, solve, compose

Example 4 — Case D: a hard task cracked by decomposition

Forecast: Does splitting the task beat brute-forcing it? By how much?

Figure walk-through: the top strip shows four separate red boxes, one per sub-question, each labelled $0.9$, joined left-to-right by black arrows into a composed proof — this is the "split" strategy. The bottom strip shows a single long black box labelled "one 4-step chain past the 3-step limit, each step 0.6" — this is the "brute force" strategy. Under each strip the resulting accuracy is printed in the matching colour: $0.9^4=0.6561$ (top, red) versus $0.6^4=0.1296$ (bottom, black). The picture makes the point visually: four tall $0.9$ factors multiply to a much bigger number than four short $0.6$ factors.

1. Amplified path — 4 sub-answers. Each sub-question is answered correctly with probability $0.9$; the composed proof is right only if all $4$ are: $0.9^4=\mathrm{0.6561.}$ Why this step? Decomposition places each sub-task within $M_0$'s reliable $3$-step reach, and — because the sub-questions are about different parts of the proof — we treat their successes as independent, so the joint success probability is the product $0.9\times 0.9\times 0.9\times 0.9$. Look at the four red boxes in the figure: one $0.9$ factor each.

2. Brute-force path — one 4-step chain past the limit. Now each step is only $0.6$ reliable, and again the chain succeeds only if every step does: $0.6^4=\mathrm{0.1296.}$ Why this step? Exceeding the $3$-step reach degrades every step to $0.6$; under the same independence-of-steps assumption we again multiply, but now with the smaller factor.

3. Compare. Amplified $0.6561$ vs brute $0.1296$ — decomposition wins by a factor of about $5$. Why this step? Dividing the two accuracies gives the multiplicative advantage of splitting, the number that answers the forecast.

Verify: $0.6561/0.1296\approx 5.06>1$ ✓. Breaking the task keeps every piece inside the model's reliable zone, and the composed trajectory becomes the training signal for the next model — the amplification loop of the parent note.

---

Row 3 — Debate: the two signs of a claim

Debate has two "quadrants": the claim is true (Ex 5) or the claim is false (Ex 6). The whole point is that the honest side should win in both. We reuse the payoff $J$ from the top of the page: the judge returns $J=+1$ if the pro-side wins, $J=-1$ if the con-side wins. When the outcome is forced (one side has an unbeatable checkable step) $J$ is a single definite value.

Example 5 — Case E: debate on a true claim

Forecast: $J=+1$ or $J=-1$?

1. Identify who can be concrete. $\sqrt{17}\approx 4.12$, so candidate divisors are $2,3,4$. None divides $17$ ($17/2,17/3,17/4$ are non-integers), so the pro-side shows primality directly. Why this step? A true claim has a short verifiable witness; that is exactly the "true claims are more defensible" premise from the parent note.

2. The con-side cannot produce a divisor, so its argument is unbacked. The judge follows the checkable step, which only the pro-side has, and rules for it: $J=+1$. Why this step? The judge is defined to side with the concretely checkable argument, so once we know only one side has it, the ruling is forced.

Verify: $17$ has no divisor in $\{2,3,4\}$ ✓, so $J=+1$ is correct — the honest (pro) side wins on a true claim.

---

Example 6 — Case F: debate on a false claim (the sign flips)

Forecast: Does the sign of $J$ flip versus Example 5?

1. The con-side finds a witness. $21=3\times 7$, a single checkable fact. Why this step? A false primality claim always has an explicit divisor — the falsehood's Achilles heel, and again a short verifiable witness the judge can trust.

2. The pro-side cannot refute $3\times 7=21$, so it loses. The judge follows the concrete step, which now belongs to the con-side, and rules against $c$: $J=-1$. Why this step? Same judge rule as Ex 5 — side with the checkable argument — but this time the checkable argument is the con-side's, so the sign of the ruling flips from $+1$ to $-1$.

Verify: $3\times 7=21$ ✓ so $c$ is false and $J=-1$ ✓. Together Ex 5 ($J=+1$ on truth) and Ex 6 ($J=-1$ on falsehood) show both signs: whichever way the claim points, the truthful debater is the winner. That is the entire justification for debate as alignment.

---

Row 4 — Degenerate inputs, real world, and the exam twist

Example 7 — Case G: the zero / tie / no-evidence cases

Forecast: Which of these can a well-designed system answer, and which must it abstain on?

1. (a) Empty decomposition, $k=0$. With no subtasks, the composed answer is the empty answer; there is nothing to get wrong. Accuracy is defined as $p=1$ (vacuously correct), by the same convention as an empty product ${\prod }_{i=1}^0=1$. Why this step? Every topic must define its zero input; here the empty product from Ex 4's multiplication rule gives the value automatically.

2. (b) Tie. Both sides equally convincing → the judge has no distinguishing checkable step, so the honest side has no advantage and the ruling is a coin flip: the judge returns $J=+1$ and $J=-1$ with probability $\frac{1}{2}$ each. This is the one place the average payoff from the top of the page is used: $\overline{J}=\frac{1}{2}(+1)+\frac{1}{2}(-1)=0,$ which by $\overline{J}=2p-1$ corresponds to accuracy $p=0.5$ — pure chance. Why this step? A tie is the boundary between the $+1$ and $-1$ quadrants; averaging the two equally-likely signed rulings gives $\overline{J}=0$, and we convert that to the probability $p=0.5$ so we never confuse the signed payoff with the accuracy.

3. (c) No evidence. The judge cannot verify anything, so the system should abstain rather than guess — a forced guess would have accuracy $p=0.5$, no better than a coin. Why this step? Guessing under no signal adds no value; abstention is the safe limiting behaviour.

Verify: empty product $=1$ ✓; tie average payoff $\frac{1}{2}-\frac{1}{2}=0$, hence $p=0.5$ ✓; forced-guess accuracy $=0.5$ ✓.

---

Example 8 — Case H: real-world review pipeline

Forecast: Guess the extra correct decisions (out of 12) and the hours saved.

1. Correct decisions, unaided. $12\times 0.70=8.4$. Why this step? Expected count = number of trials × per-trial accuracy (each paper is an independent decision, so counts add and the expectation is the product).

2. Correct decisions, assisted. $12\times 0.92=11.04$. Why this step? Same formula with the raised accuracy.

3. Extra correct decisions. $11.04-8.4=2.64$. Why this step? Subtract the two expected counts to isolate the assistant's contribution.

4. Hours saved. Per paper $4-1.5=2.5$ hours; over $12$ papers $12\times 2.5=30$ hours. Why this step? Time saved is per-paper saving times number of papers.

Verify: $12\times 0.70=8.4$, $12\times 0.92=11.04$, difference $2.64$ ✓; $12\times 2.5=30$ hours ✓. Units: (papers)×(hours/paper) = hours ✓.

---

Example 9 — Case I: the exam twist — does accuracy survive recursion?

Forecast: Does the edge collapse to $0$, or level off above chance?

Figure walk-through: the horizontal axis is recursion depth $i$; the vertical axis is final accuracy $p$. The red curve with round markers is $p_i=0.5+e_0{r}^i$; it starts high at $0.9$ and bends downward, flattening as it approaches — but never touches — the dashed black line at $p=0.5$ (chance). A dotted vertical marker at $i=3$ is annotated "$i=3$: 0.84295", the value we compute below. The curve staying strictly above the dashed line is the whole message: oversight degrades gracefully, it never inverts.

1. Write the recursion. Each step multiplies the edge by $r$: $e_i=r\cdot e_{i-1}$, so the closed form is $e_i=e_0{r}^i$. Why this step? "Preserves a fraction $r$" means the loss is proportional to the current edge, not a fixed subtraction, so we use a geometric (repeated-multiplication) sequence rather than a linear one.

2. Plug in $i=3$. $e_3=0.4\times 0.95^3=0.4\times 0.857375=0.34295$, so accuracy $p_3=0.5+0.34295=0.84295$. Why this step? Direct substitution into the closed form gives the depth-$3$ answer the annotation in the figure marks.

3. Limiting behaviour. As $i\to \infty$, $r^i\to 0$ (since $0<r<1$), so $e_i\to 0$ and $p_i\to 0.5$ — accuracy approaches chance but stays strictly above it for every finite $i$. Why this step? This is the safety question itself: the edge shrinks but never goes negative, so oversight decays gracefully rather than flipping to worse-than-chance. (If instead $r>1$ the edge would grow — the ideal regime, but harder to guarantee.)

Verify: $0.4\times 0.95^3=0.34295$ ✓; final accuracy $0.84295$ ✓; and $e_i>0$ for all finite $i$ because a positive number times a positive power stays positive ✓.

---

Recall

Recall Why does an

aligned helper raise accuracy but a misaligned one lower it? Because the human's judgement is conditioned on the helper's. If that helper correlates with truth (flags real flaws) it lifts the catch rate, as in Ex 2 ($0.40\to 0.8675$); if it anti-correlates (hides flaws) it drives acceptance of bad outputs up to $0.97$, as in Ex 3 — same mechanism, opposite direction.

Recall What is the difference between the payoff

$J$ and the accuracy $p$? $J\in \{+1,-1\}$ is a signed ruling direction; $p\in [0,1]$ is a probability of being correct. They connect through the average payoff $\overline{J}=2p-1$, so $\overline{J}=0\iff p=0.5$.

Connections: Scalable oversight · RLHF · AI Safety · Interpretability · Recursive Self-Improvement · Hinglish version