Scalable oversight: Visual walkthrough — AI-ML deep dive

We will use exactly three symbols throughout, and we earn each one before we lean on it. Nothing here assumes you have seen RLHF or AI Safety before — but if you want the wider "why", those are your neighbours.

Step 1 — What "solve" and "capability" even mean

WHAT. Before any math, picture a machine as a box. You slide a question into the box; a guess slides out. That's all a model is to us right now.

WHY. Every symbol we introduce must point at something in a picture, or it's cheating. So we anchor the very first symbol — the box — to a drawing.

PICTURE. Look at the figure. The blue box is our model. We give it a name, $M$ . The arrow going in is the question $x$ (think "x = the thing I'm asking"). The arrow coming out is the answer the box produces, written $M (x)$ — read it aloud as "M of x", meaning "the answer box $M$ gives to question $x$ ."

Now the crucial extra idea: a box has a capability ceiling. Small questions fit through; a question that is too hard comes out as garbage. In the figure, the green questions fit; the red question is above the ceiling — the box fails it. Hold onto that ceiling; the whole page is about raising it.

Step 2 — A hard task nobody can do in one shot

WHAT. Pick a task $T$ (a specific hard question) that sits above the box's ceiling. Our running example: "Prove the Intermediate Value Theorem" (the fact that a continuous line starting below zero and ending above zero must cross zero somewhere).

WHY. If the task were already inside the ceiling we'd just ask the box and go home. The entire method exists only because $T$ is out of reach. So we must draw exactly that: $T$ living above the line.

PICTURE. The red star $T$ floats above the ceiling. The box, alone, cannot reach it — the red arrow bounces off. A human, alone, also can't grind out the full proof in reasonable time. Two failing solvers. That's our starting predicament.

Step 3 — Decompose: cut $T$ into pieces below the ceiling

WHAT. A human takes the hard task $T$ and splits it into smaller sub-questions $x_{1}, x_{2}, \dots, x_{k}$ , each chosen so that it lands below the ceiling — small enough for the current box $M$ to answer.

WHY. This is the one thing humans are genuinely good at even when they can't solve the whole: knowing the shape of a solution. You may not be able to prove the IVT cold, but you know it needs these bricks: "define the set of points where $f$ is negative", "show that set has a least upper bound", "show that bound is the crossing point". Each brick is easy; the architecture is the human's contribution.

PICTURE. The red star cracks into $k$ small green dots — every dot below the ceiling. Notice each dot is a genuine question in its own right ( $x_{1}$ = "why is the set non-empty?", etc.).

T ⟶ each below the ceiling x_{1}, x_{2}, \dots, x_{k}

$T$ ::: the too-hard task (the red star).
$x_{1}, \dots, x_{k}$ ::: the sub-questions the human carves out — the green dots.
$k$ ::: how many pieces. It can be any whole number $\geq 1$ ; small tasks give small $k$ , big tasks give big $k$ .

The arrow $⟶$ here is doing the work of the word "decompose": it is a human act, not the box's.

Step 4 — Solve each piece with the weak box

WHAT. Feed every sub-question into the same weak box $M$ . Because each $x_{i}$ is below the ceiling, each comes back with a good answer $y_{i} = M (x_{i})$ .

WHY. We deliberately made the pieces small in Step 3 so that this step succeeds. This is where the box actually earns its keep — but only on problems it can genuinely handle, so its answers are trustworthy.

PICTURE. Each green dot passes through a copy of the box and turns into a filled answer $y_{i}$ . All $k$ answers now sit ready on the right.

y_{i} = M (x_{i}) for i = 1, 2, \dots, k

$y_{i}$ ::: the answer to the $i$ -th piece — e.g. $y_{2} =$ "the set is bounded above because it lives inside $[a, b]$ ."
$M (x_{i})$ ::: the box, applied to piece $i$ . Same box every time — we are not using a stronger model, just the one we have.
the subscript $i$ ::: a counter, running $1$ through $k$ , telling us which piece we mean.

Step 5 — Compose: glue the piece-answers into an answer to $T$

WHAT. The human stitches $y_{1}, \dots, y_{k}$ back together into one full answer $y$ to the original hard task $T$ .

WHY. Each brick was verified (each was small enough for both box and human to trust). Stacking verified bricks in the human's chosen architecture yields a whole the human also trusts — even though the human could never have produced the bricks unaided in the time available. We just manufactured a correct answer to $T$ without anyone solving $T$ in one shot. That is the payoff we needed in Step 2.

PICTURE. The $k$ answer-dots flow up into a single red-outlined answer $y$ sitting exactly on the red star $T$ . The star is reached — but by the team, not by the box alone.

We give this whole three-part dance (decompose → solve → compose) a single name, the Amplify operator:

Amplify (M, x) = human glues Compose (box solves pieces M (x_{1}), \dots, M (x_{k})), human splits x \to x_{1}, \dots, x_{k}

$Amplify (M, x)$ ::: the answer the human-plus-box team gives to task $x$ . It is stronger than $M$ alone — that's why it's called amplify.
Reading it right to left: human splits, box solves, human glues.

Step 6 — Distill: train a new box to do all of that in one step

WHAT. Collect many pairs (hard task $x$ , team-answer $Amplify (M_{i}, x)$ ) and train a fresh box $M_{i + 1}$ to reproduce the team's answer directly — no decomposition at run time.

WHY. The team's answers are correct answers to hard tasks. They are exactly the training data we complained we didn't have in Step 2 — and now we've fabricated it. Show enough examples and the new box internalises the decompose-solve-compose pattern, folding a whole tree of reasoning into a single feed-forward answer.

PICTURE. On the left, the slow team (human + old box $M_{i}$ ) grinds out answers to hard tasks. Those answers become a stack of flashcards. On the right, a new box $M_{i + 1}$ studies the flashcards until it can answer the hard task alone. Its ceiling has visibly risen past the old one.

Step 7 — Iterate: the ceiling climbs

WHAT. Now $M_{i + 1}$ has a higher ceiling. So take a task $T^{''}$ that was too hard even for the team last round, decompose it into pieces that $M_{i + 1}$ (stronger now!) can handle, and run Steps 3–6 again to build $M_{i + 2}$ .

WHY. One round raises the ceiling once. Repeating rounds raises it again and again, in principle without bound — this is the "iterated" in iterated amplification, and it's the same recursive spirit behind Recursive Self-Improvement, but kept under human-shaped decomposition so oversight survives the climb.

PICTURE. A staircase of boxes $M_{0}, M_{1}, M_{2}, \dots$ , each ceiling higher than the last. The dashed line traces how a task that was out of reach becomes routine two steps later.

M_{0} amplify M_{1} amplify M_{2} amplify \dots

Step 8 — The edge cases (never let the reader hit an untold scenario)

Real derivations must survive their degenerate inputs. Here are the corners.

The one-picture summary

One loop, read clockwise: a hard task is split by a human into ceiling-sized pieces, each solved by the weak box, glued back into a full answer, and that answer is distilled into a new box whose ceiling is higher — feeding the next, harder loop.

Recall Feynman retelling — say it like you'd tell a friend

Imagine you're a chess coach and your student can only calculate two moves ahead. You can't calculate twenty moves ahead either — but you know the plan: control the centre, then open a file, then swing the rook over. So you ask the student the little questions ("is this square safe?", "can the rook reach that file?") — all two-move questions the student can answer. You stitch their little answers into a twenty-move plan that neither of you could compute alone. Then you make the student play through that whole plan a thousand times until they can just see it in one glance. Now they calculate five moves ahead by themselves. Repeat: your questions can now be bigger, because the student is stronger. That loop — you supply the shape, the student supplies the steps, then the student swallows the whole thing — is iterated amplification. The catch: every little question you ask must be one you can genuinely check, because a wrong little answer sneaks into the plan, and the plan becomes the next lesson.

Recall Quick self-test

What does $Amplify (M, x)$ produce, in one sentence? ::: The answer a human-plus-box team gives to task $x$ , by splitting $x$ into small pieces, solving each with $M$ , and gluing the results — stronger than $M$ alone. Why do we train $M_{i + 1}$ to imitate the team instead of just always using the team? ::: The team needs a human in the loop and is slow; distilling it into one box lets the box solve the hard task alone, and raises its ceiling for the next round. What breaks if $k = 1$ ? ::: No real decomposition happens, so $Amplify (M_{i}, x) = M_{i} (x)$ and the team is no stronger than the box — zero amplification. Why must decomposition bottom out at verifiable pieces, not just answerable ones? ::: A wrong brick the human can't catch poisons the composed answer, which then becomes a training label and corrupts the next box.

Parent: Scalable oversight · Hinglish: 6.4.05 Scalable oversight (Hinglish)