Code-generation agents: Worked examples — AI-ML deep dive

Before any symbol appears, one reminder in plain words:

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The worked examples below are tagged with the cell they hit, and together they fill every row.

#	Case class	What's special about it	Covered by
A	Ordinary interior value	$0 < p < 1$ , small $k$	Ex 1
B	Large- $k$ / limiting behaviour	$k \to \infty$ , what does $P_{success}$ approach?	Ex 2
C	Degenerate low: $p = 0$	agent never succeeds — no amount of iteration helps	Ex 3
D	Degenerate high: $p = 1$	agent always succeeds — one try is enough	Ex 3
E	Inverse question	given a target $P_{success}$ , solve for required $k$	Ex 4
F	pass@k from raw counts	small $n, c$ — exact combinatorics, edge $c = 0$ and $c \geq k$	Ex 5
G	Real-world word problem	budget/cost trade-off, units of dollars & minutes	Ex 6
H	Exam twist: broken independence	why naive formula over-counts; the TDD/ $n$ mini-check	Ex 7

The figure above plots $P_{success} (k) = 1 - (1 - p)^{k}$ against $k$ for three values of $p$ . Notice the shape: every curve climbs fast then flattens toward $1$ (the dashed ceiling). The $p = 0$ curve is the flat red line stuck at $0$ ; the $p = 1$ curve jumps to $1$ immediately. Those two flats are cells C and D — the degenerate edges of the whole family.

Worked examples

Example 1 — the ordinary case (cell A)

Worked example Statement

An agent solves a coding task on the first try with probability $p = 0.4$ . It is allowed $k = 3$ independent attempts. What is the probability it succeeds at least once?

Forecast: guess before computing — will it be above or below $0.4$ ? (More tries can only help, so it must be above $0.4$ . Above $0.9$ feels too high for only 3 tries — pencil in "around 0.75".)

Write the failure of one attempt. Failure probability $= 1 - p = 1 - 0.4 = 0.6$ . Why this step? It is easier to compute "fails every time" than "succeeds at least once"; the complement turns an OR into a product.
Multiply failures across attempts. All three fail: $(0.6)^{3} = 0.216$ . Why this step? Independence means we multiply — the chance of three separate unlucky flips.
Take the complement. Success in at least one $= 1 - 0.216 = 0.784$ . Why this step? "At least one success" is exactly "not (all failures)".

$P_{success} (3) = 1 - (1 - 0.4)^{3} = 0.784$

Verify: $0.784$ sits above $0.4$ (more tries helped) and below $1$ (not guaranteed) — matches the forecast. Sanity units: it's a probability, dimensionless, in $[0, 1]$ . ✓

Example 2 — the limit as attempts grow (cell B)

Worked example Statement

Same agent ( $p = 0.4$ ). What does $P_{success} (k)$ approach as we let $k \to \infty$ ? And how many attempts to first cross $99%$ ?

Forecast: with unlimited tries, does it hit exactly $1$ , or stall below? (Any positive $p$ means the failure factor $0.6 < 1$ ; raising a number below one to higher powers shrinks it toward $0$ , so $P_{success} \to 1$ .)

Look at the failure term $(1 - p)^{k} = 0. 6^{k}$ . Why this step? All the $k$ -dependence lives here; the constant $1$ out front doesn't move.
Send $k \to \infty$ . Since $0 < 0.6 < 1$ , $0. 6^{k} \to 0$ . Why this step? Repeatedly multiplying by a fraction below one drives it to zero — this is the "flattening toward the ceiling" you saw in figure s01. $lim_{k \to \infty} [1 - 0. 6^{k}] = 1 - 0 = 1$
Find the crossing of $99%$ . Solve $1 - 0. 6^{k} \geq 0.99 \Rightarrow 0. 6^{k} \leq 0.01$ . Take natural logs (base $e$ ): $k \geq \frac{ln 0.01}{ln 0.6} = \frac{- 4.6052}{- 0.5108} \approx 9.015$ . Why $ln$ ? The unknown $k$ is stuck in an exponent; the logarithm is the tool that pulls an exponent down into a plain multiplier so we can isolate $k$ . (Any base works because it cancels in the ratio; we use $ln$ throughout.)
Round up. $k$ must be a whole number and satisfy the " $\geq$ ", so $k = 10$ .

Verify: $1 - 0. 6^{10} = 1 - 0.006047 = 0.99395 \geq 0.99$ ✓, while $1 - 0. 6^{9} = 0.98992 < 0.99$ — so $9$ is not enough and $10$ is the first that works. Limit $= 1$ matches the dashed ceiling in s01. ✓

Example 3 — both degenerate edges (cells C and D)

Worked example Statement

Two broken-record cases. (C) A hopeless agent with $p = 0$ given $k = 100$ tries. (D) A perfect agent with $p = 1$ given $k = 1$ try. Compute both.

Forecast: Will 100 tries rescue a $p = 0$ agent? (No — zero times anything is still zero.)

Case C, $p = 0$ : $P_{success} (100) = 1 - (1 - 0)^{100} = 1 - 1^{100} = 1 - 1 = 0$ . Why this step? $1$ raised to any power stays $1$ ; the failure never shrinks. Iteration cannot help an agent that cannot ever succeed — this is the flat red floor in s01.
Case D, $p = 1$ : $P_{success} (1) = 1 - (1 - 1)^{1} = 1 - 0^{1} = 1 - 0 = 1$ . Why this step? If it always works, one attempt already gives certainty; extra attempts change nothing.

Verify: C gives $0$ (agent stuck at the floor), D gives $1$ (agent at the ceiling from step one). Both are the extreme rows of the matrix and both stay inside $[0, 1]$ . ✓

Example 4 — inverse question: solve for $k$ (cell E)

Worked example Statement

Your agent has $p = 0.25$ . You need overall success probability $P_{success} (k) \geq 0.90$ . What is the smallest number of attempts $k$ ?

Forecast: $p$ is weak ( $1$ -in- $4$ ), so this will need several tries — guess "around 8".

Set up the inequality. $1 - (0.75)^{k} \geq 0.90 \Rightarrow (0.75)^{k} \leq 0.10$ . Why this step? We rearrange so the unknown exponent is isolated on one side.
Apply the natural logarithm. $k \cdot ln (0.75) \leq ln (0.10)$ . Why $ln$ again? Same reason as Ex 2 — it is the tool that frees $k$ from the exponent, and its base cancels in the ratio below.
Divide, and flip the inequality. $ln (0.75) = - 0.28768$ is negative, so dividing by it reverses " $\leq$ " into " $\geq$ ": $k \geq \frac{l n 0.10}{l n 0.75} = \frac{- 2.302585}{- 0.287682} \approx 8.004$ Why watch the sign? Dividing an inequality by a negative number flips its direction — forgetting this is the #1 exam error here.
Round up to the next integer. $k = 9$ .

Verify: $1 - 0.7 5^{9} = 1 - 0.075085 = 0.92492 \geq 0.90$ ✓; and $1 - 0.7 5^{8} = 0.89989 < 0.90$ , just short — confirming $9$ is the smallest that clears the bar. ✓

Example 5 — pass@k from raw counts, including edges (cell F)

Worked example Statement

On one benchmark problem you sampled $n = 5$ candidate solutions; $c = 2$ were correct. Compute the exact pass@2 using the parent's estimator, then handle two edges: $c = 0$ correct, and $c = 5$ correct.

The estimator: $pass@ k = 1 - \frac{( k n - c )}{( k n )}$ , where $(k n)$ (" $n$ choose $k$ ") counts the ways to pick $k$ items from $n$ ignoring order.

Forecast: with $2$ of $5$ good and $2$ picks, you'll usually catch at least one good — guess "around 0.7".

Count all ways to pick 2 of 5. $(2 5) = \frac{5 \cdot 4}{2 \cdot 1} = 10$ . Why this step? This is the denominator — the total, equally-likely draws of $k = 2$ samples.
Count all-wrong draws. Only $n - c = 3$ wrong solutions exist; ways to pick 2 wrong: $(2 3) = 3$ . Why this step? The numerator counts draws that miss every correct one — the "you got unlucky" outcomes.
Form the fraction and complement. $pass@2 = 1 - \frac{3}{10} = 0.7$ . Why complement? Same trick as Ex 1: "at least one correct" $=$ "not (all wrong)".
Edge $c = 0$ : numerator $(2 5) = 10$ equals denominator $\Rightarrow 1 - \frac{10}{10} = 0$ . (No correct samples $\Rightarrow$ can never pass.)
Edge $c = 5$ : $n - c = 0$ , and $(2 0) = 0$ $\Rightarrow 1 - \frac{0}{10} = 1$ . (All correct $\Rightarrow$ always pass.) Why these edges matter? They match cells C and D but for the pass@k formula — the two degenerate ends must give $0$ and $1$ , and they do.

Verify: main answer $0.7$ lands on the forecast; edges give $0$ and $1$ ; and note the convention $(k m) = 0$ whenever $m < k$ , which is exactly why $c = 5$ produced $0$ in the numerator. ✓

Example 6 — real-world cost trade-off (cell G)

Worked example Statement

Each agent attempt costs $0.03 in API tokens and takes 8 seconds. With $p = 0.4$ , your policy is "retry until success, capped at $k$ attempts". You want $P_{success} (k) \geq 0.95$ . (a) What $k$ ? (b) What is the expected cost per solved task if you retry only until first success?

Forecast: From Ex 2 we saw $\sim 10$ tries reaches $99%$ ; $95%$ should need fewer — guess $k \approx 6$ .

Find $k$ for $95%$ . $0. 6^{k} \leq 0.05 \Rightarrow k \geq \frac{ln 0.05}{ln 0.6} = \frac{- 2.99573}{- 0.510826} \approx 5.865 \Rightarrow k = 6$ . Why this step? Same inverse- $ln$ method as Ex 4.
Check the cap's success. $1 - 0. 6^{6} = 1 - 0.046656 = 0.953344 \geq 0.95$ ✓.
Expected attempts until first success (unbounded retry, geometric mean) $= \frac{1}{p} = \frac{1}{0.4} = 2.5$ attempts. Why $1/ p$ ? For an independent "keep flipping until heads" process, the average number of flips is one over the success probability — a standard result of the geometric distribution.
Convert to money and time. Cost $= 2.5 \times \$ 0.03 = $0.075 $p er so l v e d t a s k . T im e$ = 2.5 \times 8 = 20 $seco n d s p er so l v e d t a s k . * W h y t hi ss t e p ? * M u l t i pl y t h e a v er a g e a tt e m pt co u n t b y t h e p er - a tt e m ptp r i ce / d u r a t i o n — t h e u ni t sc an ce l cor r ec tl y (a tt e m pt s$ \times $\$ /attempt $=$ $).

Verify: $k = 6$ (near the forecast of 6). Units: $2.5\,\text{attempts}\times 0.03\,\tfrac{\$ }{\text{attempt}} = 0.075,$ $;$ 2.5 \times 8,\tfrac{\text{s}}{\text{attempt}} = 20,\text{s}$ — dollars and seconds, as required. ✓

Example 7 — exam twist: independence breaks (cell H)

Worked example Statement

A student claims: "My agent has $p = 0.3$ per try and $k = 4$ tries, so overall success is $1 - 0. 7^{4} = 0.7599$ ." Their agent actually reads its own error messages and improves each round. Is the true success probability higher or lower than $0.7599$ , and why? Also verify the $n$ divisor-check bound the parent used for is_prime.

Forecast: does learning from errors help or hurt? (Helps — later tries are informed, so the real number should be higher than the independence estimate.)

Name the assumption. The formula $P_{success} (k) = 1 - (1 - p)^{k}$ assumes independence: every attempt has the same fixed $p$ , ignoring what happened before. Why this step? You can only judge a formula once you know its assumption.
Compare to reality. A real agent uses the error trace, so attempt 2's success chance is conditioned on attempt 1's failure and is usually higher than the raw $p$ . Successive attempts are positively informative, not independent. Why this direction? Learning removes already-ruled-out mistakes, so the effective per-round $p$ rises. Therefore true $P_{success} \geq 0.7599$ — the independence formula is a conservative lower bound for a debugging agent.
The $n$ check (why the prime loop stops at $n$ ): if $n = a \times b$ with $a \leq b$ , then $a \leq n$ . So any composite number must have a factor at or below $n$ ; checking past it is wasted work. For $n = 37$ , $37 \approx 6.08$ , so we test divisors $2..6$ ; none divide $37$ , so it is prime. Why include this? It's the classic exam "explain the bound" that pairs with the agent's TDD example — a reasoning check, not just a plug-in.

Verify: The independence number $1 - 0. 7^{4} = 0.7599$ is correct as a bound; the true value is $\geq 0.7599$ for a learning agent. And $37$ has no divisor in ${2, 3, 4, 5, 6}$ , confirming primality with only $⌊ 37 ⌋ = 6$ tests instead of $36$ . ✓

Recall Self-test (reveal after trying)

One-try success $p = 0.5$ , three attempts — $P_{success} (3)$ ? ::: $1 - 0. 5^{3} = 0.875$ With $p = 0$ , does raising $k$ to a million ever help? ::: No — $P_{success}$ stays $0$ ; iteration multiplies an existing chance, it can't create one. Given $p = 0.25$ , smallest $k$ for $P_{success} (k) \geq 90%$ ? ::: $k = 9$ (since $0.7 5^{9} \approx 0.075 \leq 0.10$ ). pass@2 with $n = 5$ , $c = 2$ ? ::: $1 - (2 3) / (2 5) = 1 - 3/10 = 0.7$ . Why does a real debugging agent beat the independence formula? ::: It conditions on past errors, raising the effective per-round success, so the true $P_{success}$ is at least the independence estimate.

Prerequisites & neighbours: Large Language Models · Tool use in AI systems · Automated testing · Program synthesis · Retrieval-Augmented Generation · Prompt engineering · AI safety and alignment · back to Code-generation agents.