Foundations — Scalable oversight

Before you can read the parent note, you need to be fluent in the little pieces of notation it throws around: $x$, $y$, $H$, $M_i$, $R$, $\arg \min$, $\mathbb{E}$, $\mathcal{L}$, $\sup$, $ϵ$, and a handful of words like "recursive", "amplify", "judge". This page builds each one from nothing, in an order where every symbol is earned before it is used.

1. The cast of characters: what a "model" and a "task" even are

Everything in this topic is a conversation between three kinds of things. Figure s01 (below) draws them: a blue question card on the left, a yellow model box in the middle, a pink answer card on the right — read it left-to-right as "question goes in, answer comes out."

Why the topic needs this. Scalable oversight is entirely about the gap between "producing $y$" (easy for a strong AI) and "checking $y$ is good" (hard for a weak human). You cannot talk about that gap without names for the question and the answer.

2. The human as a function: $H$

Here is the idea the whole field pivots on: we treat a human judgement as a machine too. Figure s02 (below) shows it: two inputs feed a yellow "H" box — the pink answer $y$ from the top, the blue question $x$ from the bottom — and out comes a score on the right.

Why the bar $\mid$ matters. An answer is only good relative to a question. "42" is a great answer to "6×7" and a terrible answer to "capital of France". The $\mid$ says: always judge the answer in the light of its question. You will meet this exact bar again in the recursive formula, where a second thing gets added after it.

Why the topic needs $H$. Scalable oversight's whole worry is: what happens when $H$ can no longer tell good from bad by itself? Naming the human as $H$ lets us write, precisely, the moment $H$ breaks — and the fix.

3. Reward $R$: turning a judgement into a training number

An AI cannot be trained on the word "good." It needs a number to chase. That number is the reward.

Why the topic needs $R$. This is the bridge to RLHF (Reinforcement Learning from Human Feedback), the standard method the parent note is upgrading. In RLHF, $R$ comes straight from $H$. Scalable oversight's central move is to change what feeds into $R$ — that is literally the whole "recursive reward" idea.

4. The word amplify, and the picture behind it

Before we can read the training formula, we need the word that appears inside it. It is really a picture, so we draw it first. Figure s04 (below) shows the whole move: one hard yellow task on the left is chopped into blue pieces, each piece is solved into a pink partial answer, and the pieces are glued back into one answer on the right.

5. The three shorthand operators: $\arg \min$, $\mathbb{E}$, $\mathcal{L}$

The training formula uses three more compact symbols. They look scary; each is one plain sentence.

Figure s03 (below) nails the $\min$-vs-$\arg \min$ distinction that trips everyone up: the pink dot marks the value at the bottom (that is $\min$), while the yellow arrow points along the ground at the model that sits there (that is $\arg \min$).

6. The conditioning bar, extended: the recursive reward

Now we can read the topic's key formula, because we already know every piece — including the two-argument reading $M(x,y)$ ("review this answer") from Section 1.

Here $P(\cdots )$ is a probability — a number between $0$ (never) and $1$ (always) measuring how often something happens.

7. Two last symbols for the debate and proof examples: $J$ and $\sup$

Prerequisite map

This feeds directly into Scalable oversight, and connects onward to RLHF, Interpretability, and Recursive Self-Improvement.

Equipment checklist

Reveal each line only after answering it in your head.