Code-generation agents: Visual walkthrough — AI-ML deep dive

Everything below refers to a picture. Read the picture first, then the words.

Step 1 — What is a single attempt, and what does $p$ mean?

WHAT. An agent writes code once, runs it in the sandbox, and the run either works or doesn't work. That is one attempt. We give it a name: the chance it works is the number $p$ . We also name how many attempts we're willing to make: $k$ , a positive whole number ( $k \geq 1$ ) — you cannot make "half an attempt" or "zero-and-a-bit" attempts, so $k$ counts $1, 2, 3, \dots$ .

WHY a single number? Because before we can talk about "trying again," we must nail down what one try even is. Every symbol must be earned — so $p$ is earned here: it is a fraction between $0$ and $1$ measuring "out of many single tries, how many succeeded."

$p = a fraction, so it lives between 0 and 1 \frac{number of tries that produced working code}{total number of tries}$

If $p = 0$ : the code never works on the first go (a broken agent).
If $p = 1$ : it always works first try (a dream agent).
Real code agents sit around $p \approx 0.4$ — less than half. That is the whole reason this page exists.

PICTURE. A bar of 10 tries; the coloured slice is the fraction that worked.

Step 2 — The trick: it is easier to count FAILURE

WHAT. We want the chance of succeeding at least once in several tries. That phrase "at least once" is slippery — success could happen on try 1, or try 2, or 3... lots of cases to add up. So we flip the question: what is the chance of the opposite — failing every single time?

WHY flip it? Because "at least one success" and "all failures" are the only two possible worlds. There is no third. If we know the size of one world, the other is just the rest. This is the complement idea, and it turns a messy sum into one subtraction.

$what we want P (at least one success) = 1 - easier to count P (every try fails)$

The $1$ here is "the whole pie" — all possible outcomes together. Chop off the failure slice, and what remains must contain at least one success.

PICTURE. The full circle = 1. Two slices only: "all fail" vs "the rest."

Step 3 — The chance ONE try fails

WHAT. If a try succeeds with chance $p$ , then it fails with whatever is left over.

WHY? Success and failure of a single try are also a two-slice pie summing to $1$ . So failure fills the rest.

$P (one try fails) = whole pie 1 - success slice p = 1 - p$

$p = 0.4 \Rightarrow$ failure chance $= 0.6$ .
Edge check: $p = 1 \Rightarrow$ failure $= 0$ (never fails, good). $p = 0 \Rightarrow$ failure $= 1$ (always fails).

PICTURE. One try = one coin with a "works" face and a "breaks" face, sized $p$ and $1 - p$ .

Step 4 — Failing $k$ times in a row: why we MULTIPLY

WHAT. The agent tries $k$ times (recall from Step 1: $k$ is a positive whole number, $k \geq 1$ ). For every try to fail, try 1 fails and try 2 fails and ... and try $k$ fails.

WHY multiply, not add? Here is the key assumption, stated out loud: each retry is independent — the agent re-rolls fresh each time (fresh prompt, fresh generation), so one failure doesn't change the odds of the next. When independent things must all happen, their chances multiply. (Adding would give the chance of "one or another" — the wrong question.)

Think of a branching tree: each try splits into "fail" (weight $1 - p$ ) and "success." To reach the "all-failed" leaf you walk down the fail branch $k$ times, multiplying the weight each step:

$P (all k fail) = try 1 fails (1 - p) \times try 2 fails (1 - p) \times \dots \times try k fails (1 - p) = (1 - p)^{k}$

The exponent $k$ is literally how many times you walked down the fail branch.

PICTURE. A failure tree; the highlighted all-fail path multiplies $(1 - p)$ each level.

Step 5 — Snap it together: the parent's formula

WHAT. Plug Step 4 back into the complement flip of Step 2.

WHY? Step 2 said "what we want = $1 -$ (all fail)." Step 4 just told us "all fail $= (1 - p)^{k}$ ." Substitute:

$P (success in \leq k attempts) = 1 - (1 - p)^{k}$

Term by term:

$1$ — the whole probability pie.
$(1 - p)$ — one try's failure chance (Step 3).
$()^{k}$ — that failure repeated and multiplied across $k$ independent tries (Step 4).
The leading $1 -$ — the complement flip: subtract the all-fail world to get "at least one win" (Step 2).

Sanity on the edges:

$k = 1$ : $1 - (1 - p)^{1} = p$ . One try = the plain single-shot rate. ✓
$p = 0$ : $1 - 1^{k} = 0$ . A broken agent never succeeds no matter how many tries. ✓
$k \to \infty$ with $p > 0$ : $(1 - p)^{k} \to 0$ , so the whole thing $\to 1$ . Given enough tries and any positive skill, success becomes near-certain. ✓

PICTURE. The curve $1 - (1 - p)^{k}$ vs $k$ for $p = 0.4$ , with the parent's numbers marked.

Step 6 — Edge case: what if retries are NOT independent?

WHAT. Real agents see the error message and adjust — so retry 2 is smarter than retry 1. The tries are not independent, and $p$ is not constant.

WHY show this? Because Step 4's multiplication rule assumed independence. We must not let you walk into a scenario we hid. When feedback helps, let $p_{i}$ be the success chance on the $i$ -th try (rising as the agent learns from retrieved context and stack traces).

$P (all k fail) = \prod_{i = 1}^{k} (1 - p_{i})$

The symbol $\prod$ (capital Greek "pi") is a multiply-them-all sign: $\prod_{i = 1}^{k} (1 - p_{i})$ just means "take the factors $(1 - p_{1}), (1 - p_{2}), \dots, (1 - p_{k})$ and multiply the whole list together" — the multiply-cousin of the more familiar $\sum$ (add-them-all) sign. It runs over the same $k$ tries. If every $p_{i} = p$ , this collapses back to $(1 - p)^{k}$ — our simple formula is the special case.

If feedback helps ( $p_{i}$ rising): each factor $(1 - p_{i})$ shrinks faster → success climbs even faster than the independent curve.
If feedback hurts (agent loops on the same wrong idea, the Step-4 mistake box): $p_{i}$ can stall or drop, and the curve flattens. This is the real-world failure mode.

PICTURE. Two curves: constant- $p$ (dashed) vs improving- $p_{i}$ (solid, steeper) vs stuck- $p_{i}$ (flat).

Step 7 — The benchmark version: pass@k with a fixed pool

WHAT. Benchmarks like HumanEval don't retry live. Instead they generate $n$ samples once, count how many ( $c$ ) are correct, then ask: "if I randomly grab $k$ of those $n$ , chance at least one is correct?" That is pass@k. (Here $k$ is how many we draw, and it must satisfy $0 \leq k \leq n$ so that we never try to draw more samples than exist — otherwise $(k n)$ is not defined.)

WHY a different formula? Because now we are drawing without replacement from a fixed bag of $n$ samples — not re-rolling fresh coins. So instead of $(1 - p)^{k}$ we count how many ways to pick $k$ all-wrong samples out of the $n - c$ wrong ones.

$pass@ k = 1 - \frac{( k n - c )}{( k n )} (0 \leq k \leq n)$

$(k n)$ — ways to choose any $k$ samples from all $n$ (the whole pie of picks).
$(k n - c)$ — ways to choose $k$ samples using only the $n - c$ wrong ones (the all-fail picks).
Their ratio = chance a random draw is all-wrong; $1 -$ that = at least one right. Same complement flip as Step 2!

Edge cases:

$c = 0$ (no correct samples): $(k n) / (k n) = 1$ , so pass@k $= 0$ . Can't pick a winner that isn't there. ✓
$c \geq 1$ and $k$ large enough to force a correct pick: $(k n - c) = 0$ , pass@k $= 1$ . ✓

PICTURE. A bag of $n$ marbles, $c$ green (correct); we scoop $k$ and win if any is green.

The one-picture summary

The single figure ties it all: one coin ( $p$ ) → flip to failure ( $1 - p$ ) → repeat and multiply across $k$ tries ( $(1 - p)^{k}$ ) → subtract from the whole pie ( $1 - (1 - p)^{k}$ ) → the rising success curve. The benchmark box on the side shows the without-replacement cousin, pass@k.

Recall Feynman retelling — say it to a 12-year-old

Imagine your agent is a kid solving a puzzle who gets it right 4 times out of 10 on a single guess. That's $p = 0.4$ . Now let the kid guess again and again — call the number of guesses $k$ . The clever move is: don't ask "how likely to win eventually?" — ask "how likely to lose every single time?" Losing once has chance $0.6$ . Losing twice in a row is $0.6 \times 0.6$ , three times $0.6 \times 0.6 \times 0.6$ — the number shrinks fast, because to lose you have to be unlucky over and over. After 5 guesses, losing-every-time is only about $0.08$ , so winning at least once is about $0.92$ . Flip "all losses" over, and out pops $1 - (1 - p)^{k}$ . If the kid actually learns from each wrong answer, they win even faster. And when a test just makes $n$ guesses up front and grabs $k$ of them, the same "count the all-wrong picks and flip it" trick gives pass@k. Same idea, dressed three ways.

Recall Quick self-check

Why do we compute "all fail" instead of "at least one success" directly? ::: "At least one" spreads across many cases (win on try 1, or 2, or 3...); "all fail" is a single product. They're complements, so $P (win \geq 1) = 1 - P (all fail)$ . Why do the $k$ failure chances multiply rather than add? ::: The tries are independent and must all fail together; independent "and" events multiply. Adding would overcount and could exceed 1. What does $(1 - p)^{k}$ do as $k$ grows (with $p > 0$ )? ::: It shrinks toward 0, so success $1 - (1 - p)^{k}$ rises toward 1 — more tries, near-certain success. When does the simple formula $1 - (1 - p)^{k}$ break? ::: When retries aren't independent (the agent learns from feedback), so $p$ changes per try; use $1 - \prod_{i} (1 - p_{i})$ instead. What is the valid range of $k$ ? ::: For retries, $k$ is any positive integer $k \geq 1$ ; for pass@k it must also satisfy $k \leq n$ so $(k n)$ is defined.

Related: Program synthesis · Prompt engineering · AI safety and alignment · back to the parent topic.